What's the Practical Limit on the Size of Single Packet Transmitted Over Domain Socket

What's the practical limit on the size of single packet transmitted over domain socket?

There are a number of factors which will determine the maximum of size of a packet that can be sent on a Unix socket:

1. The wmem_max socket send buffer maximum size kernel setting, which determines the maximum size of the send buffer that can be set using setsockopt (SO_SNDBUF). The current setting can be read from /proc/sys/net/core/wmem_max and can be set using sysctl net.core.wmem_max=VALUE (add the setting to /etc/sysctl.conf to make the change persistent across reboots). Note this setting applies to all sockets and socket protocols, not just to Unix sockets.

2. If multiple packets are sent to a Unix socket (using SOCK_DATAGRAM), then the maximum amount of data which can be sent without blocking depends on both the size of the socket send buffer (see above) and the maximum number of unread packets on the Unix socket (kernel parameter net.unix.max_dgram_qlen).

3. Finally, a packet (SOCK_DATAGRAM) requires contiguous memory (as per What is the max size of AF_UNIX datagram message that can be sent in linux?). How much contiguous memory is available in the kernel will depend on many factors (e.g. the I/O load on the system, etc...).

So to maximize the performance on your application, you need a large socket buffer size (to minimize the user/kernel space context switches due to socket write system calls) and a large Unix socket queue (to decouple the producer and consumer as much as possible). However, the product of the socket send buffer size and queue length must not be so large as to cause the kernel to run out of contiguous memory areas (causing write failures).

The actual figures will depend on your system configuration and usage. You will need to determine the limits by testing... start say with wmem_max at 256Kb and max_dgram_qlen at 32 and keep doubling wmem_max until you notice things start breaking. You will need to adjust max_dgram_qlen to balance the activity of the producer and consumer to a certain extent (although if the producer is much faster or much slower than the consumer, the queue size won't have much affect).

Note your producer will have to specifically setup the socket send buffer size to wmem_max bytes with a call to setsockopt (SO_SNDBUF) and will have to split data into wmem_max byte chunks (and the consumer will have to reassemble them).

Best guess: the practical limits will be around wmem_max ~8Mb and unix_dgram_qlen ~32.

Maximum limit on size of data in IPC using sockets in unix

Q1) Is there a limit on the maximum limit on the size of data that I can send over the AF_UNIX,SOCK_STREAM socket ( from what I have read , I don't think there is )

You are correct. There is no real limit for SOCK_STREAM.

Q2 a) Does the socket break up the data into smaller blocks when transferring

You are correct. The stream is broken up into a manageable sized packets, negotiated by both the transmitter and the receiver.

Q2 b) and if thats the case do I need to receive the smaller blocks individually or as single block like I am doing right now.

You needn't do anything. The stream is reassembled on the other end, just as it was transmitted.

Q3) Will it be better to use AF_UNIX,SOCK_DGRAM socket here.

No. Unless you need to learn all about packet-size negotiation, checking for missing packets and having them resent, making sure that packets received out-of-order are managed properly, etc.

What is the max size of AF_UNIX datagram message in Linux?

AF_UNIX SOCK_DATAGRAM/SOCK_SEQPACKET datagrams need contiguous memory. Contiguous physical memory is hard to find, and the allocation fails, logging something similar to this on the kernel log:

udgc: page allocation failure. order:7, mode:0x44d0
[...snip...]
DMA: 185*4kB 69*8kB 34*16kB 27*32kB 11*64kB 1*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3788kB
Normal: 13*4kB 6*8kB 100*16kB 62*32kB 24*64kB 10*128kB 0*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 7012kB
[...snip...]

unix_dgram_sendmsg() calls sock_alloc_send_skb() lxr1, which calls sock_alloc_send_pskb() with data_len = 0 and header_len = size of datagram lxr2. sock_alloc_send_pskb() allocates header_len from "normal" skbuff buffer space, and data_len from scatter/gather pages lxr3. So, it looks like AF_UNIX sockets don't support scatter/gather on current Linux.

Is there a guideline for the maximum buffer size I send over a socket?

It depends on the kind of sockets. With TCP a connection is a byte stream and the OS will take care of how best to split the bytes into packets and concatenate these together at the other side.

With UDP instead each send will result in a single UDP message (datagram) and there is an upper limit of 64k for the size of a datagram. But even while UDP supports 64k datagrams in theory it is not a good idea to use that large messages. Since the maximum transfer unit of the underlying layer is much smaller (like around 1500 bytes for Ethernet) the message needs to be fragmented and it is easy to loose a single fragment - in which case the whole message is considered lost.

Reading the maximum size of an expected packet from a C socket

As mentioned in some other comments, reading from a socket can return any number of bytes, up to the maximum requested.

A better loop, although still not without problems, would be something along these lines:

/* 4 byte message header that contains the length */
#define MSG_HEADER_SIZE 4
#define MAX_MESSAGE_LENGTH 128
struct message {
    uint32_t length;
    char body[MAX_MESSAGE_LENGTH];
};

#define BUFFER_SIZE 1024
char buffer[BUFFER_SIZE];
uint32_t buf_used = 0;

/* main loop */
while (1) {
    n = recv(socket, buffer + buf_used, sizeof(buffer) - buf_used, 0);

    if (n == -1) {
        /* handle error */
        exit(1);
    }

    if (n == 0) {
        /* connection closed, do something. */
        exit(1);
    }

    buf_used += n;

    /* check for partial/completed message(s) */
    while (buf_used >= MSG_HEADER_SIZE) {
        struct message *cur_msg = (struct message *) buffer;
        uint32_t total_msg_length;

        total_msg_length = cur_msg->length + MSG_HEADER_SIZE;

        /* is this message completed yet? */
        if (buf_used >= total_msg_length) {
            process_message(cur_msg);

            /* remove message since it has been processed */
            buf_used -= total_msg_length;

            /* this could potentially be optimized */
            memmove(buffer, buffer + total_msg_length, buf_used);
        } else {
            /* have incomplete message */
            break;
        }
    }
}

For an introduction to socket programming, I would recommend checking out Beej's Guide to Network Programming.

When will a TCP network packet be fragmented at the application layer?

It will be split when it hits a network device with a lower MTU than the packet's size. Most ethernet devices are 1500, but it can often be smaller, like 1492 if that ethernet is going over PPPoE (DSL) because of the extra routing information, even lower if a second layer is added like Windows Internet Connection Sharing. And dialup is normally 576!

In general though you should remember that TCP is not a packet protocol. It uses packets at the lowest level to transmit over IP, but as far as the interface for any TCP stack is concerned, it is a stream protocol and has no requirement to provide you with a 1:1 relationship to the physical packets sent or received (for example most stacks will hold messages until a certain period of time has expired, or there are enough messages to maximize the size of the IP packet for the given MTU)

As an example if you sent two "packets" (call your send function twice), the receiving program might only receive 1 "packet" (the receiving TCP stack might combine them together). If you are implimenting a message type protocol over TCP, you should include a header at the beginning of each message (or some other header/footer mechansim) so that the receiving side can split the TCP stream back into individual messages, either when a message is received in two parts, or when several messages are received as a chunk.

---

Related Topics