Handling Partial Return from Recv() Tcp in C

Handling partial return from recv() TCP in C

Yes, you will need multiple recv() calls, until you have all data.

To know when that is, using the return status from recv() is no good - it only tells you how many bytes you have received, not how many bytes are available, as some may still be in transit.

It is better if the data you receive somehow encodes the length of the total data. Read as many data until you know what the length is, then read until you have received length data. To do that, various approaches are possible; the common one is to make a buffer large enough to hold all data once you know what the length is.

Another approach is to use fixed-size buffers, and always try to receive min(missing, bufsize), decreasing missing after each recv().

Socket programming issue with recv() receiving partial messages

You're right, you need a loop. recv only retrieves the data that's currently available; once any data has been read, it doesn't wait for more to appear before it returns.

The manual page says "The receive calls normally return any data available, up to the requested amount, rather than waiting for receipt of the full amount requested."

Handling multiple recv() calls and all possible scenarios

The method that Beej is alluding to, and AlastairG mentions, works something like this:

For each concurrent connection, you maintain a buffer of read-but-not-yet-processed data. (This is the buffer that Beej suggests sizing to twice the maximum packet length). Obviously, the buffer starts off empty:

unsigned char recv_buffer[BUF_SIZE];
size_t recv_len = 0;

Whenever your socket is readable, read into the remaining space in the buffer, then immediately try and process what you have:

result = recv(sock, recv_buffer + recv_len, BUF_SIZE - recv_len, 0);

if (result > 0) {
    recv_len += result;
    process_buffer(recv_buffer, &recv_len);
}

The process_buffer() will try and process the data in the buffer as a packet. If the buffer doesn't contain a full packet yet, it just returns - otherwise, it processes the data and removes it from the buffer. So for you example protocol, it would look something like:

void process_buffer(unsigned char *buffer, size_t *len)
{
    while (*len >= 3) {
        /* We have at least 3 bytes, so we have the payload length */

        unsigned payload_len = buffer[2];

        if (*len < 3 + payload_len) {
            /* Too short - haven't recieved whole payload yet */
            break;
        }

        /* OK - execute command */
        do_command(buffer[0], buffer[1], payload_len, &buffer[3]);

        /* Now shuffle the remaining data in the buffer back to the start */
        *len -= 3 + payload_len;
        if (*len > 0)
            memmove(buffer, buffer + 3 + payload_len, *len);
    }
}

(The do_command() function would check for a valid header and command byte).

This kind of technique ends up being necessary, because any recv() can return a short length - with your proposed method, what happens if your payload length is 500, but the next recv() only returns you 400 bytes? You'll have to save those 400 bytes until the next time the socket becomes readable anyway.

When you handle multiple concurrent clients, you simply have one recv_buffer and recv_len per client, and stuff them into a per-client struct (which likely contains other things too - like the client's socket, perhaps their source address, current state etc.).

How to receive CHUNK's of data from recv() in BSD Sockets in C programming?

I have used MSG_DONTWAIT flag in recv and run it in a loop and append new data to the buffer. After that the recv throws -1 with errno EAGAIN then I break the loop and parse the websocket data. And thanks @user253751 for the explanation, it helps to clear some doubt which arose in my mind about the implementation. The below is the code snippet I used in my library.

ws_recv_totalSize = 0;
ws_recv_ret = 0;
while(1){
  ws_recv_totalSize += ws_recv_ret;
  ws_recv_ret = recv(*ws_sock, ws_recv + ws_recv_totalSize, MAX_BUFFER- ws_recv_totalSize, MSG_DONTWAIT);
      
  if(ws_recv_ret <= 0)
    break;
}

How C++ `recv` function acts at data receving? Could it receive a partial packet?

Is it possible that recv first got 500 bytes, and the remain 12 will receive from second attempt?

Yes, definitely.

You have no guarantee that, when the sending end sends a burst of X bytes, that the receiving end will pick them all up in a single call to recv. recv doesn't even know how many bytes are in your application-layer "packet".

The whole problem is - do I need to make some possibility for client to combine received bytes into one packet or split over-received bytes to different packets?

Your application will definitely have to accumulate data from possibly sequential reads, fill up a buffer, and implement parsing to look for a full packet.

TCP/IP doesn't know your application's protocol; as David said, if you want to split the incoming data stream into "packets" then you must do that yourself.

Reading more than one message from recv()

If you have a fixed size you want to receive you could do something like this:

ssize_t recv_all(int socket, char *buffer_ptr, size_t bytes_to_recv)
{
    size_t original_bytes_to_recv = bytes_to_recv;

    // Continue looping while there are still bytes to receive
    while (bytes_to_recv > 0)
    {
        ssize_t ret = recv(socket, buffer_ptr, bytes_to_recv, 0);
        if (ret <= 0)
        {
            // Error or connection closed
            return ret;
        }

        // We have received ret bytes
        bytes_to_recv -= ret;  // Decrease size to receive for next iteration
        buffer_ptr += ret;     // Increase pointer to point to the next part of the buffer
    }

    return original_bytes_to_recv;  // Now all data have been received
}

Simply use as

// Somewhere above we have received the size of the data to receive...

// Our data buffer
char buffer[the_full_size_of_data];

// Receive all data
recv_all(socket, buffer, sizeof buffer);  // TODO: Add error checking

_{[Note that I use POSIX types like ssize_t and int for the sockets. Modify to fit your system (e.g. SOCKET for the socket on Windows).]}

Client and Server send() and recv() in C

You need some way to indicate the server that you have finished sending a file, and now you want to send another thing.

While the socket abstraction seems to show you that the recv and send calls are somehow synchronized, this means that the data you send from the client in one call to send is recv'd in the server with exactly one recv, that is not true, due fundamentally to how the sockets are implemented (the client tcp can decide to split your transfer in several packets, and the unattending of the server can make all those packets to buffer n the receiver before the server receives part of them in one call to recve and others in the text call.

The only thing sure is that a byte that has been sent before, is receive before, and no repeated or missing bytes in between. But the number of bytes received at some recve call is dictated only by the amount of buffered data that one side of the connection has.

This means that, for telling the server that you are finished with your file and some other thing is to be sent, you must do something that allows the server to recognize that the data has ended and more data on a different thing is about to come.

There are several approaches here... you can send an inband sequence (some control sequence) that, wen read in the other side, will be recognized as the end of a block of data and the beginning of the next. Some systems use a packet encoding that simply prepends each block with a number of bytes to follow, and an empty packet (e.g. a single number 0, meaning 0 bytes to follow) can represent this sequence. Another way, can be to use a specific sequence you know is improbable to occur in the data (e.g. \n.\m, two newlines with one dot interspersed ---this has been used many times in unix's ed editor, for example, or the post office protocol uses it.) and if it is the case that such a sequence happens to be in the file, just double the dot, to indicate that it is not the separator sequence. And both ends must agree on this (so called) protocol.

Other mechanisms are valid, you can close the connection and use another socket for a new data exchange.... for example, FTP protocol uses this approach by using one control connection for FTP commands, while using other connections for data transfers.

I have not included code because there are plenty of examples in the literature. Try to get access to the book "Unix Network Programming" of Richard W. Stevens. That's a very good reference to get initiated on socket protocols and how to deal with all these things.

---

Related Topics