Linux Select() and Fifo Ordering of Multiple Sockets

FIFO pipe is always readable in select()

You opened that FIFO as read only (O_RDONLY), whenever there is no writer to the FIFO, the read end will receive an EOF.

Select system call will return on EOF and for every EOF you handle there will be a new EOF. This is the reason for the observed behavior.

To avoid this open that FIFO for both reading and writing (O_RDWR). This ensures that you have at least one writer on the FIFO thus there wont be an EOF and as a result select won't return unless someone writes to that FIFO.

How to get timestamp at which a file descriptor monitored by select() becomes ready?

The timestamp at which a file descriptor becomes ready is the modification time of the file descriptor.
Or in other words, the time at which the file represented by the file descriptor was last modified.

The modification time of a file (represented by a file descriptor) can be obtained using the fstat() method.
Read http://pubs.opengroup.org/onlinepubs/009695399/functions/fstat.html for details.

Data accumulates (FIFO?) in broadcast UDP socket

I must misunderstand something in socket ... can you help me ?

There are receive and send socket buffers in the kernel. The kernel receives datagrams for your listening socket and stores them in the kernel socket receive buffer. recvfrom call copies the oldest datagram from the kernel socket buffer into your user-space buffer. The fact that you do not call recvfrom doesn't mean the kernel stops receiving datagrams (you would need to close the socket for the kernel to stop receiving data).

epoll order of events from epoll_wait

The observation by @jxh about the behavior is correct, and the behavior is long established (and was originally intended, if I correctly recall my email conversations with the implementer, Davide Libenzi, many years ago). It's unfortunate that it has not been documented so far. But, I've fixed that for the upcoming manual pages release, where epoll_wait(2) will carry the text:

If more than maxevents file descriptors are ready when
epoll_wait() is called, then successive epoll_wait() calls will
round robin through the set of ready file descriptors. This
behavior helps avoid starvation scenarios, where a process fails
to notice that additional file descriptors are ready because it
focuses on a set of file descriptors that are already known to be
ready.

receive multicast from specific network interface on Linux

First, check if any of your calls fail, socket,bind,setsockopt in this case. Printing an error message with the perror() function will help you diagnose problems.

However, for receiving multicast datagrams you might need to specify the ip address of the interface when you join a multicast group using the IP_ADD_MEMBERSHIP socket option
Something like

  setsockopt (sd, IPPROTO_IP, IP_ADD_MEMBERSHIP, &mreq, sizeof(mreq));

where the mreq struct is

struct ip_mreq
{
        struct in_addr imr_multiaddr;   /* IP multicast address of group */
        struct in_addr imr_interface;   /* local IP address of interface */
};

More info here.

multicast basics - lost data

Windows XP has a 8KB default socket buffer, Vista and later has a 64KB default socket buffer. You can override this with SO_RCVBUF, it is recommended to never set to 0 unlike SO_SNDBUF for zero-copy when using IOCP.

Usually with socket buffers the queue fills up and starts dropping new packets, so with a large buffer you can end up with stale out-of-date messages waiting in the buffer taking precedence over newer fresh data.

The design you need to follow is a high priority thread in a tight loop reading packets as fast as possible from the network and inserting into a larger application queue for later processing if your business logic is slower. When using IOCP you can post several blocking recv calls to minimize data loss but as the MSDN documentation states there is no guarantee on order of completions on the completion port.

Pipes and sockets - message passing or shared memory?

message passing, as they require participation on both sides, sender and receiver in case of sockets for example. they can be implemented using shared memory, but communication pattern is message passing

Multiple UDP sockets and multiple clients

What I want to do is move all communication with a specific client to
the new socket. Can I do that?

You could create a new UDP socket, bind() it to a different port, and then instruct the client to start sending traffic to the new socket's port instead of the usual one; but note that there isn't really any advantage to be gained by doing that.

If not, won't having all the clients write to socket 0 clog it if I
won't do recvfrom often enough?

Yes, but the usual solution that is to call recvfrom() often enough to keep up with the flow of incoming traffic, and/or to increase its SO_RECVBUF buffer size to make its incoming-data buffer big enough that it is inlikely to become full.
One way to ensure recvfrom() is called often enough is to create a separate thread that does nothing but call recvfrom() in a loop and then hand off the data to another thread for more intensive processing. Run this network thread at higher priority if possible to make sure it doesn't get held off of the CPU by other threads.

If it's possible to recv data from a specific client on a separate
socket, will that data come to BOTH socket 0 and the specific socket?

If you have two threads both calling recvfrom() on the same socket, then any given incoming UDP packet will be delivered to either one thread or the other, and it's impossible to predict which thread will receive which packet -- it will just be luck of the draw depending on what is the next packet in the socket's incoming buffer at the instant a particular thread calls recvfrom() on it. In general having multiple threads accessing a single socket is not a recommended design.

I know that TCP would work better for this but I have to use UDP. How
should I do this? Is there an accepted "standard" of handling such
situations?

I don't know about what is "standard", but I usually have a dedicated I/O thread that does nothing but read from (and if necessary, write to) the UDP socket. It sets the UDP socket to non-blocking mode, then loops around select() to recvfrom() any incoming UDP packets and append (in a thread-safe manner) both the incoming packet's data and its source-address/port info to a FIFO queue for other (less time-sensitive) threads to take out and work on later. That way even if a packet (or series or packets) takes a relatively long time to process, the result won't be that packets get dropped (although it might temporarily increase RAM usage as the FIFO grows larger)

Is there a way to have multiple threads listen to the same socket in python?

Having multiple threads simultaneously use the same TCP socket is possible in that most operating systems support it (i.e. you won't crash), but it's not useful because it's unpredictable which thread will end up receiving which bytes of data from the socket (or conversely, which thread will end up sending which bytes of data to the socket when) -- in either case your TCP data will end up getting randomly send to your various threads, which is a big problem if your messages are ever longer than one byte long.

What you can do, OTOH, is have one thread that is dedicated to only reading from the TCP socket, and when it has read in a full message, it hands that message over to one of the other threads (using a mutex-protected FIFO queue and a condition-variable, or similar) for processing. If the processing-thread later wants to send data back to the TCP socket, it would hand the data back to the network-thread via another FIFO queue, and then the network thread would take care of pulling the data out of the FIFO and writing it out to the TCP socket. In that design, because only one thread has access to the TCP socket, you can guarantee that messages boundaries are respected and messages do not get mixed up with each other.

Note that in order to have the network thread handle both reading and writing (and responding to incoming messages on its FIFO queue) in a timely manner, you may want to have it use non-blocking I/O and event-loop that blocks only inside select().

---

Related Topics