How Many Packets or Bytes Are in the Socket Receive Queue

How many packets or bytes are in the socket receive queue?

On Windows, what you are looking for is available via ioctlsocket(FIONREAD) and WSAIoCtl(FIONREAD), which both return the full size of the complete buffered data, even when multiple datagram messages are buffered. However, there is no equivalent on Linux. There is ioctl(FIONREAD), which only returns the size of the next buffered message.

Python socket receive - incoming packets always have a different size

Note: As people have pointed out in the comments, calling recv() with no parameters is not allowed in Python, and so this answer should be disregarded.

Original answer:

---

The network is always unpredictable. TCP makes a lot of this random behavior go away for you. One wonderful thing TCP does: it guarantees that the bytes will arrive in the same order. But! It does not guarantee that they will arrive chopped up in the same way. You simply cannot assume that every send() from one end of the connection will result in exactly one recv() on the far end with exactly the same number of bytes.

When you say socket.recv(x), you're saying 'don't return until you've read x bytes from the socket'. This is called "blocking I/O": you will block (wait) until your request has been filled. If every message in your protocol was exactly 1024 bytes, calling socket.recv(1024) would work great. But it sounds like that's not true. If your messages are a fixed number of bytes, just pass that number in to socket.recv() and you're done.

But what if your messages can be of different lengths? The first thing you need to do: stop calling socket.recv() with an explicit number. Changing this:

data = self.request.recv(1024)

to this:

data = self.request.recv()

means recv() will always return whenever it gets new data.

But now you have a new problem: how do you know when the sender has sent you a complete message? The answer is: you don't. You're going to have to make the length of the message an explicit part of your protocol. Here's the best way: prefix every message with a length, either as a fixed-size integer (converted to network byte order using socket.ntohs() or socket.ntohl() please!) or as a string followed by some delimiter (like '123:'). This second approach often less efficient, but it's easier in Python.

Once you've added that to your protocol, you need to change your code to handle recv() returning arbitrary amounts of data at any time. Here's an example of how to do this. I tried writing it as pseudo-code, or with comments to tell you what to do, but it wasn't very clear. So I've written it explicitly using the length prefix as a string of digits terminated by a colon. Here you go:

length = None
buffer = ""
while True:
  data += self.request.recv()
  if not data:
    break
  buffer += data
  while True:
    if length is None:
      if ':' not in buffer:
        break
      # remove the length bytes from the front of buffer
      # leave any remaining bytes in the buffer!
      length_str, ignored, buffer = buffer.partition(':')
      length = int(length_str)

    if len(buffer) < length:
      break
    # split off the full message from the remaining bytes
    # leave any remaining bytes in the buffer!
    message = buffer[:length]
    buffer = buffer[length:]
    length = None
    # PROCESS MESSAGE HERE

Linux Sockets, how to get number of bytes/packets in sending buffer?

Here is how to obtain,

1. The length of data in Receive Buffer which is not read yet:

   ioctl( socket_descriptor, FIONREAD, &size );  // alternative 1
   ioctl( socket_descriptor, SIOCINQ, &size );   // alternative 2
   

2. The length of data in Send Buffer which is not drained yet (either not sent yet or send but not acknowledged by receiver):

   ioctl( socket_descriptor, TIOCOUTQ, &size );  // alternative 1
   ioctl( socket_descriptor, SIOCOUTQ, &size );  // alternative 2
   

Reference: http://linux.die.net/man/7/tcp

Raw Socket Receive Buffer

Thanks for your inputs. However, we have been able to "solve" this problem.
Earlier, I described how we read messages.
Once select returns, we run a loop (to the tune of number of Raw Messages to read which was >1 in our case).
1) we call ioctl(FIONREAD) to find the number of bytes to read;
2) read that many bytes by calling recvfrom
3) send the bytes upto the user
4) go into loop again and call ioctl(FIONREAD) and then repeat the steps

However, at point 4, ioctl(FIONREAD) use to return 0. Our code had a defensive check. It was expecting, a 0 bytes from ioctl(FIONREAD) means that the sender has send an IP header with 0 payload. Therefore, it use to call recvfrom(bytes to read=0) to flush out the IP header lest the select will set again on this.

At time t0, ioctl(FIONREAD) returns 0 as number of bytes to read
At time t1, recvfrom(bytes to read=0) is called.
Sometimes, between t0 and t1, actual data use to get queued in the socket receive queue and use to get discarded as we were calling recvFrom(bytes=0).

Setting, the number of rawMsgsToRead=1 has "solved" this problem. However, my guess is it will impact our performance. Is their any ioctl call which can differentiate between octets in the queue as 0 and IP header with payload 0

How large should my recv buffer be when calling recv in the socket library

The answers to these questions vary depending on whether you are using a stream socket (SOCK_STREAM) or a datagram socket (SOCK_DGRAM) - within TCP/IP, the former corresponds to TCP and the latter to UDP.

How do you know how big to make the buffer passed to recv()?

• SOCK_STREAM: It doesn't really matter too much. If your protocol is a transactional / interactive one just pick a size that can hold the largest individual message / command you would reasonably expect (3000 is likely fine). If your protocol is transferring bulk data, then larger buffers can be more efficient - a good rule of thumb is around the same as the kernel receive buffer size of the socket (often something around 256kB).

• SOCK_DGRAM: Use a buffer large enough to hold the biggest packet that your application-level protocol ever sends. If you're using UDP, then in general your application-level protocol shouldn't be sending packets larger than about 1400 bytes, because they'll certainly need to be fragmented and reassembled.

What happens if recv gets a packet larger than the buffer?

• SOCK_STREAM: The question doesn't really make sense as put, because stream sockets don't have a concept of packets - they're just a continuous stream of bytes. If there's more bytes available to read than your buffer has room for, then they'll be queued by the OS and available for your next call to recv.

• SOCK_DGRAM: The excess bytes are discarded.

How can I know if I have received the entire message?

• SOCK_STREAM: You need to build some way of determining the end-of-message into your application-level protocol. Commonly this is either a length prefix (starting each message with the length of the message) or an end-of-message delimiter (which might just be a newline in a text-based protocol, for example). A third, lesser-used, option is to mandate a fixed size for each message. Combinations of these options are also possible - for example, a fixed-size header that includes a length value.

• SOCK_DGRAM: An single recv call always returns a single datagram.

Is there a way I can make a buffer not have a fixed amount of space, so that I can keep adding to it without fear of running out of space?

No. However, you can try to resize the buffer using realloc() (if it was originally allocated with malloc() or calloc(), that is).

Get the number of bytes available in socket by 'recv' with 'MSG_PEEK' in C++

You're looking for is ioctl(fd,FIONREAD,&bytes_available) , and under windows ioctlsocket(socket,FIONREAD,&bytes_available).

Be warned though, the OS doesn't necessarily guarantee how much data it will buffer for you, so if you are waiting for very much data you are going to be better off reading in data as it comes in and storing it in your own buffer until you have everything you need to process something.

To do this, what is normally done is you simply read chunks at a time, such as

char buf[4096];
ssize_t bytes_read;
do {
     bytes_read = recv(socket, buf, sizeof(buf), 0);
     if (bytes_read > 0) {
         /* do something with buf, such as append it to a larger buffer or
          * process it */
     }
} while (bytes_read > 0);

And if you don't want to sit there waiting for data, you should look into select or epoll to determine when data is ready to be read or not, and the O_NONBLOCK flag for sockets is very handy if you want to ensure you never block on a recv.

Unix domain socket queue details

1. You can use the ioctl to find out.

To check a write buffer if it empty (assuming you have already put data there and want to check if they were consumed):

ioctl(fd, SIOCOUTQ, &pending);

Where fd is the socket’s file descriptor and pending the variable were the remaining size of data will be returned.

To check a read buffer if it empty (assuming someone has already put data there and you want to check if they there is any without consuming them):

ioctl(fd, SIOCINQ, &pending);
/*note the difference on the second parameter, where we change the flag from SIOCOUTQ to SIOCINQ*/

2. Based on the error message returned by the send() function , we can identify the buffer full event.By checking the error==ENOBUFS you can identify the buffer full.

How do I get amount of queued data for UDP socket?

As ldx mentioned, it is not supported through ioctl or getsockopt.
It seems to me that the current implementation of SIOCINQ was aimed to determine how much buffer is needed to read the entire waiting buffer (but I guess it is not so useful for that, as it can change between the read of it to the actual buffer read).

There are many other telemetries which are not supported though such system calls, I guess there is no real need in normal production usage.

You can check the drops/errors through "netstat -su" , or better using SNMP (udpInErrors) if you just want to monitor the machine state.

BTW: You always have the option to hack in the Kernel code and add this value (or others).

Can I determine how much data is queued up on an incoming TCP client stream (in C#)?

You can use the TcpClient.Available property for that.

From MSDN:

Type: System.Int32

The number of bytes of data received from the
network and available to be read.

---

Related Topics