What Conditions Result in an Opened, Nonblocking Named Pipe (Fifo) Being "Unavailable" for Reads

What conditions result in an opened, nonblocking named pipe (fifo) being unavailable for reads?

From the POSIX specification of the read system call (emphasis mine):

When attempting to read from an empty pipe or FIFO:

• If no process has the pipe open for writing, read() shall return 0 to
  indicate end-of-file.

• If some process has the pipe open for writing and O_NONBLOCK is set,
  read() shall return -1 and set errno to [EAGAIN].

So basically your second assumption is wrong:

If the writer has the pipe opened, but no data is in the fifo, empty str ('') is also returned

This would be against the specification and I can't reproduce that behaviour on my machine (it raises EAGAIN for me). This is not a big problem however, you can just catch the exception and retry:

import errno

def safe_read(fd, size=1024):
   ''' reads data from a pipe and returns `None` on EAGAIN '''
   try:
      return os.read(fd, size)
   except OSError, exc:
      if exc.errno == errno.EAGAIN:
         return None
      raise

How to read named FIFO non-blockingly?

According to the manpage of read(2):

   EAGAIN or EWOULDBLOCK
          The  file  descriptor  fd refers to a socket and has been marked
          nonblocking   (O_NONBLOCK),   and   the   read   would    block.
          POSIX.1-2001  allows  either error to be returned for this case,
          and does not require these constants to have the same value,  so
          a portable application should check for both possibilities.

So what you're getting is that there is no data available for reading. It is safe to handle the error like this:

try:
    buffer = os.read(io, BUFFER_SIZE)
except OSError as err:
    if err.errno == errno.EAGAIN or err.errno == errno.EWOULDBLOCK:
        buffer = None
    else:
        raise  # something else has happened -- better reraise

if buffer is None: 
    # nothing was received -- do something else
else:
    # buffer contains some received data -- do something with it

Make sure you have the errno module imported: import errno.

Pipe/FIFO read attempt after writer closed

The data still in the pipe is read first before an EOF is signaled.

Read (os.read) FIFO non-blockingly without fixed buffer size

Assuming that your delimiter is a you can read multiple variable length strings in a non-blocking manner, as shown in this program which counts while receiving output from a named pipe.

import os
import time
import errno
import sys

io = os.open(expanduser("~/named_pipes/cob_input"), os.O_RDONLY | os.O_NONBLOCK)

# For implementing non-blocking IO
def read_pipe_non_blocking(input_pipe, size):
    try:
        in_buffer = os.read(input_pipe, size)
    except OSError as err:
        if err.errno == errno.EAGAIN or err.errno == errno.EWOULDBLOCK:
            in_buffer = None
        else:
            raise  # something else has happened -- better reraise
    return in_buffer

def get_azimuth(input_pipe):
    in_buffer = read_pipe_non_blocking(input_pipe, 1)
    print(in_buffer)
    if(in_buffer is None):
        sys.stderr.write("n")
        return ""
    else:
        tmp_buffer = None
        while(tmp_buffer != "a"):
            sys.stderr.write("m")
            time.sleep(0.1)
            tmp_buffer = read_pipe_non_blocking(input_pipe, 1)
            if(tmp_buffer != None and tmp_buffer != "a"):
                in_buffer += tmp_buffer
        read_pipe_non_blocking(input_pipe, 1) #Read in the newline character and the toss it
        sys.stderr.write("\nReturning \{%s\}" %in_buffer)
        return in_buffer

i = 0
while 1:
    print i
    time.sleep(1)
    i += 1
    get_azimuth(io)

This code has been directly copy pasted from my code and isn't really that clear. If anyone needs clarification, leave a comment.

O_NONBLOCK does not raise exception in Python

This solution is incorrect.

while True:
    buffer = os.read(ph, BUFFER_SIZE)
    if len(buffer) < BUFFER_SIZE:
        break

This will not actually read everything, it will only read until it gets a partial read. Remember: You are only guaranteed to fill the buffer with regular files, in all other cases it is possible to get a partial buffer before EOF. The correct way to do this is to loop until the actual end of file is reached, which will give a read of length 0. The end of file indicates that there are no writers (they have all exited or closed the fifo).

while True:
    buffer = os.read(ph, BUFFER_SIZE)
    if not buffer:
        break

However, this will not work correctly in the face of non-blocking IO. It turns out non-blocking IO is completely unnecessary here.

import os
import fcntl

h = os.open("pipe.fifo", os.O_RDONLY | os.O_NONBLOCK)
# Now that we have successfully opened it without blocking,
# we no longer want the handle to be non-blocking
flags = fcntl.fcntl(h, fcntl.F_GETFL)
flags &= ~os.O_NONBLOCK
fcntl.fcntl(h, fcntl.F_SETFL, flags)
try:
    while True:
        # Only blocks if there is a writer
        buf = os.read(h, 65536)
        if not buf:
            # This happens when there are no writers
            break
finally:
    os.close(h)

The only scenario which will cause this code to block is if there is an active writer which has opened the fifo but is not writing to it. From what you've described, it doesn't sound like this is the case.

Non-blocking IO doesn't do that

Your program wants to do two things, depending on circumstance:

1. If there are no writers, return immediately.

2. If there are writers, read data from the FIFO until the writers are done.

Non-blocking read() has no effect whatsoever on task #1. Whether you use O_NONBLOCK or not, read() will return immediately in situation #1. So the only difference is in situation #2.

In situation #2, your program's goal is to read the entire block of data from the writers. That is exactly how blocking IO works: it waits for the writers to finish, and then read() returns. The whole point of non-blocking IO is to return early if the operation can't complete immediately, which is the opposite of your program's goal—which is to wait until the operation is complete.

If you use non-blocking read(), in situation #2, your program will sometimes return early, before the writers have finished their jobs. Or maybe your program will return after reading half of a command from the FIFO, leaving the other (now corrupted) half there. This concern is expressed in your question:

If the non blocking read does not raise exception, how should I know when to stop reading?

You know when to stop reading because read() returns zero bytes when all writers have closed the pipe. (Conveniently, this is also what happens if there were no writers in the first place.) This is unfortunately not what happens if the writers do not close their end of the pipe when they are done. It is far simpler and more straightforward if the writers close the pipe when done, so this is the recommended solution, even if you need to modify the writers a little bit. If the writers cannot close the pipe for whatever reason, the solution is more complicated.

The main use case for non-blocking read() is if your program has some other task to complete while IO goes on in the background.

Are there repercussions to having many processes write to a single reader on a named pipe in posix?

The FIFO write should be atomic, as long as it's under the page size. So there shouldn't be an issue with 100 bytes messages. On linux the max size used to be 4K, I believe it is larger now. I've used this technique on a few systems for message passing, since the writes end up atomic.

You can end up with an issue, if you are using a series of writes, since output buffering could cause a sync issue. So make sure the whole message is written at one time. eg. build a string, then print, don't print multiple pieces at once.

s="This is a message"
echo $s

NOT

echo "This "
echo "is "
echo " a message"

---

Related Topics