Python Read Named Pipe

Python read named PIPE

In typical UNIX fashion, read(2) returns 0 bytes to indicate end-of-file which can mean:

• There are no more bytes in a file
• The other end of a socket has shutdown the connection
• The writer has closed a pipe

In your case, fifo.read() is returning an empty string, because the writer has closed its file descriptor.

You should detect that case and break out of your loop:

reader.py:

import os
import errno

FIFO = 'mypipe'

try:
    os.mkfifo(FIFO)
except OSError as oe: 
    if oe.errno != errno.EEXIST:
        raise

print("Opening FIFO...")
with open(FIFO) as fifo:
    print("FIFO opened")
    while True:
        data = fifo.read()
        if len(data) == 0:
            print("Writer closed")
            break
        print('Read: "{0}"'.format(data))

Example session

Terminal 1:

$ python reader.py 
Opening FIFO...
<blocks>

Terminal 2:

$ echo -n 'hello' > mypipe 

Terminal 1:

FIFO opened
Read: "hello"
Writer closed
$ 

---

Update 1 - Continuously re-open

You indicate that you want to keep listening for writes on the pipe, presumably even after a writer has closed.

To do this efficiently, you can (and should) take advantage of the fact that

Normally, opening the FIFO blocks until the other end is opened also.

Here, I add another loop around open and the read loop. This way, once the pipe is closed, the code will attempt to re-open it, which will block until another writer opens the pipe:

import os
import errno

FIFO = 'mypipe'

try:
    os.mkfifo(FIFO)
except OSError as oe:
    if oe.errno != errno.EEXIST:
        raise

while True:
    print("Opening FIFO...")
    with open(FIFO) as fifo:
        print("FIFO opened")
        while True:
            data = fifo.read()
            if len(data) == 0:
                print("Writer closed")
                break
            print('Read: "{0}"'.format(data))

---

Terminal 1:

$ python reader.py 
Opening FIFO...
<blocks>

Terminal 2:

$ echo -n 'hello' > mypipe 

Terminal 1:

FIFO opened
Read: "hello"
Writer closed
Opening FIFO...
<blocks>

Terminal 2:

$ echo -n 'hello' > mypipe 

Terminal 1:

FIFO opened
Read: "hello"
Writer closed
Opening FIFO...
<blocks>

... and so on.

---

You can learn more by reading the man page for pipes:

• PIPE(7) - Linux Programmer's Manual
• FIFO(7) - Linux Programmer's Manual

Wait for Named Pipe to be read before Disconnecting

I think there is a problem with Python f.read() and Windows pipes. Maybe I missed something but it looks like there is no EOF when you read a pipe like you do, and python read() will read past the end and error, even though it read everything correctly before.

To work around this, you can use a buffered pipe (open(r"pipe", 'r')) and then either read characters one by one (f.read(1)) until you have an error, or use os.read(f.fileno(), 1024), which is lower level and works in this case.

Python get data from named pipe

os.mkfifo is used to create fifo. Use open to open/read fifo already exist:

with open('/tmp/shairport-sync-metadata') as f:   # add `rb` for binary mode
    # line-by-line read
    for line in f:
        print(line)

    # f.read(1024)  # to read 1024 characters

Named pipe contents are discarded when read only a single line

Update #1

The problem is that consumer between each select closes and reopens pipe.
It consumed just one line and upon close all other data waiting in pipe to be read are gone.

Therefore you should permute while True with fdopen. Also you should handle EOF properly in consumer (this is where first, additional while True comes from). Here is fixed code:

Consumer

import os, sys
import select

while True:

    print("Opening pipe")
    queue: int = os.open('pipe', flags=os.O_RDONLY)
    with os.fdopen(queue, 'rb') as stream:

        while True:
            readers, _, _ = select.select([stream], [], [])

            if readers:
                reader = readers.pop()

                contents: bytes = reader.readline()
                if contents is b'':
                    print("EOF detected")
                    break

                contents: bytes = contents.strip()
                if b'quit' == contents:
                    sys.exit(0)

                print(contents)

Producer

Producer lacked of flush call:

import os

fd = os.open('pipe', os.O_WRONLY)
ps = open(fd, 'wb')
for i in range(10):
    ps.write(str(i).encode())
    ps.write(os.linesep.encode())
    ps.flush()

ps.close()

Great help for analyzing your code was running consumer under strace which revealed all syscalls and I could notice closes between selects.

---

Old outdated answer

In the line:

reader = readers.pop()

You get from pipe more content than just first byte.
But then you print out just first byte from what you got.

See that when you add at the end of consumer code another read from reader:

contents2: bytes = reader.readline().strip()
print(contents2)

You will get:

b'0'
b'1'

Of course complete code should test how much it got from pipe. Print it and then wait for more data to appear in pipe.

Here is my updated consumer code:

import os, sys
import select

while True:
    queue: int = os.open('pipe', flags=os.O_RDONLY | os.O_NONBLOCK)
    with os.fdopen(queue, 'rb') as stream:
        readers, _, _ = select.select([stream], [], [])
        if readers:
            reader = readers.pop()
            while True:
                contents: bytes = reader.readline().strip()
                if contents is b'':
                    break
                if b'quit' == contents:
                    sys.exit(0)

                print(contents)

---

Related Topics