How to handle a broken pipe (SIGPIPE) in python?
Read up on the try: statement.
try:
# do something
except socket.error, e:
# A socket error
except IOError, e:
if e.errno == errno.EPIPE:
# EPIPE error
else:
# Other error
Python - How to catch a broken pipe
Since you are writing such a small amount of data, it is all buffered and nothing is actually written to the pipe until the file is closed. During the close, an attempt is made to write data to the pipe, which fails, but your try/except clause is already done. If you flush stdout during your try/except, you should catch the error. (Although, since you are writing to the pipe in the except clause, you won't see it!)
How to prevent errno 32 broken pipe?
Your server process has received a SIGPIPE writing to a socket. This usually happens when you write to a socket fully closed on the other (client) side. This might be happening when a client program doesn't wait till all the data from the server is received and simply closes a socket (using close function).
In a C program you would normally try setting to ignore SIGPIPE signal or setting a dummy signal handler for it. In this case a simple error will be returned when writing to a closed socket. In your case a python seems to throw an exception that can be handled as a premature disconnect of the client.
Catch Broken Pipe in Python 2 AND Python 3
If all you care about are broken pipe errors, then you might want to catch socket.error and simply check whether it's indeed a broken pipe error.
You can do so using the exception's errno attribute, which is present in both Python 2 and Python 3, which means, you don't need different Python 2 vs. 3 logic (I'd argue the intent is a little clearer this way):
import socket
import errno
try:
do_something()
except socket.error as e:
if e.errno != errno.EPIPE:
# Not a broken pipe
raise
do_something_about_the_broken_pipe()
If you do care about more than broken pipes, thefourtheye's answer is appropriate and idiomatic.
IOError: [Errno 32] Broken pipe when piping: `prog.py | othercmd`
I haven't reproduced the issue, but perhaps this method would solve it: (writing line by line to stdout rather than using print)
import sys
with open('a.txt', 'r') as f1:
for line in f1:
sys.stdout.write(line)
You could catch the broken pipe? This writes the file to stdout line by line until the pipe is closed.
import sys, errno
try:
with open('a.txt', 'r') as f1:
for line in f1:
sys.stdout.write(line)
except IOError as e:
if e.errno == errno.EPIPE:
# Handle error
You also need to make sure that othercommand is reading from the pipe before it gets too big - https://unix.stackexchange.com/questions/11946/how-big-is-the-pipe-buffer
How to prevent Brokenpipe error in my server script
There was a bunch of things I had to do to make this work.
First, the Broken pipe can occur at two different places in the server code. First is when a new client connects and the server tries to send message New Client Connected to all the clients and second is when an existing client sends a message. So, we need to handle the exception at both the places.
So, we put try/except on both the blocks. where it says if c != conn.
Now, about how to handle the exception.
As I first thought, simply removing the client c from the list on clients will work, but the for loop, for c in clients will throw out runtime error as we try to modify the set clients during iteration.
I tried different methods to get over this problem, but this is the most efficient working method I got.
I changed clients from a set() to an empty list [ ]
Then I changed clients.add to clients.append
Then I changed the for loop to for c in range(len(clients)) and used clients[c] to access clients.
But when I tried this, I saw that the if statement if clients[c] != conn may throw out list index out of bounds error if the program tries to go over a non existant client after removal. So I put it too in a try/except block and let the program continue on exception.
for c in range(len(clients)):
try:
if clients[c] != conn:
try:
message = connection_message.encode(FORMAT)
msg_length = len(message)
send_length = str(msg_length).encode(FORMAT)
send_length += b' ' * (HEADER - len(send_length))
clients[c].sendall(send_length)
clients[c].sendall(message)
except:
clients.remove(clients[c])
except:
continue
The last problem was that even after removal of the client, the thread is still alive, so the active thread count returns more than the number of clients connected. So, instead of printing out the number of active connection as number of alive threads - 1, I print len(clients) + 1, + 1 because on connection of a new client, it prints this line before appending the client to list.
print(f"[ACTIVE CONNECTIONS] {len(clients) + 1}")
So, the entire program now is:
import socket
import threading
HEADER = 64
PORT = 5050
SERVER = socket.gethostbyname(socket.gethostname())
ADDR = (SERVER, PORT)
FORMAT = 'utf-8'
DISCONNECT_MESSAGE = "!DISCONNECT"
server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server.bind(ADDR)
clients = []
clients_lock = threading.Lock()
def handle_client(conn, addr):
name = conn.recv(HEADER).decode(FORMAT)
if name:
name = int(name)
msg_name = conn.recv(name).decode(FORMAT)
print(f"[NEW CONNECTION] {msg_name} connected.")
connection_message = f"{msg_name} connected."
with clients_lock:
for c in range(len(clients)):
try:
if clients[c] != conn:
try:
message = connection_message.encode(FORMAT)
msg_length = len(message)
send_length = str(msg_length).encode(FORMAT)
send_length += b' ' * (HEADER - len(send_length))
clients[c].sendall(send_length)
clients[c].sendall(message)
except:
clients.remove(clients[c])
except:
continue
with clients_lock:
clients.append(conn)
connected = True
try:
while connected:
msg_length = conn.recv(HEADER).decode(FORMAT)
if msg_length:
msg_length = int(msg_length)
msg1 = conn.recv(msg_length).decode(FORMAT)
msg = f"{msg_name}: {msg1}"
if msg1 == DISCONNECT_MESSAGE:
connected = False
print(f"{msg}")
with clients_lock:
for c in range(len(clients)):
try:
if clients[c] != conn:
try:
message = msg.encode(FORMAT)
msg_length = len(message)
send_length = str(msg_length).encode(FORMAT)
send_length += b' ' * (HEADER - len(send_length))
clients[c].sendall(send_length)
clients[c].sendall(message)
except:
clients.remove(clients[c])
except:
continue
msg = f"You: {msg1}"
message = msg.encode(FORMAT)
msg_length = len(message)
send_length = str(msg_length).encode(FORMAT)
send_length += b' ' * (HEADER - len(send_length))
conn.send(send_length)
conn.send(message)
finally:
with clients_lock:
clients.remove(conn)
conn.close()
def start():
server.listen()
print(f"[LISTENING] Server is listening on {SERVER}")
while True:
conn, addr = server.accept()
thread = threading.Thread(target=handle_client, args=(conn, addr))
thread.daemon = True
thread.start()
print(f"[ACTIVE CONNECTIONS] {len(clients) + 1}")
print("[STARTING] server is starting...")
start()
Broken pipe error when trying to send data from server to client in Python sockets
After trying out the code, I think the biggest problem you have is that the server is trying to send on the wrong socket. i.e. this line:
s.send("hey".encode())
should be rewritten like this:
conn.send("hey".encode())
As it is, you are trying to send() on the TCP accepting-socket rather than on the TCP connection to the client, which doesn't make sense. On my (MacOS/X) system, the server process prints this error output:
Jeremys-Mac-mini-2:~ jaf$ python server.py
('address: ', '127.0.0.1')
Traceback (most recent call last):
File "server.py", line 18, in <module>
server()
File "server.py", line 14, in server
s.send("hey".encode())
socket.error: [Errno 57] Socket is not connected
Why does python generate sigpipe exception on closing a fifo file?
It is because Python won't clear the write buffer when fifo.write fails. So the buffer will be wrote to the broken pipe again when executing fifo.close, which causes the second SIGPIPE.
I found the reason with the help of strace. Here are some details.
First, modify a small part of that Python code, as following,
#!/usr/bin/env python3
import os
import signal
import subprocess
# The FIFO file.
os.mkfifo('tmp.fifo')
# A subprocess to simply discard any input from the FIFO.
FNULL = open(os.devnull, 'w')
proc = subprocess.Popen(['/bin/cat', 'tmp.fifo'], stdout=FNULL, stderr=FNULL)
print('pid = %d' % proc.pid)
# Open the FIFO, and MUST BE BINARY MODE.
fifo = open('tmp.fifo', 'wb')
i = 0
# Endlessly write to the FIFO.
while True:
# Try to write to the FIFO, restart the subprocess on demand, until succeeded.
while True:
try:
# Optimistically write to the FIFO.
fifo.write(f'hello{i}'.encode())
fifo.flush()
except IOError as e:
# The subprocess died. Close the FIFO and reap the subprocess.
print('IOError is occured.')
fifo.close()
os.kill(proc.pid, signal.SIGKILL)
proc.wait()
# Start the subprocess again.
proc = subprocess.Popen(['/bin/cat', 'tmp.fifo'], stdout=FNULL, stderr=FNULL)
print('pid = %d' % proc.pid)
fifo = open('tmp.fifo', 'wb')
else:
# The write goes on well.
break
os.kill(proc.pid, signal.SIGKILL)
i += 1
and save it as test.py.
Then run strace -o strace.out python3 test.py in the shell. Check the strace.out and we can find something like
openat(AT_FDCWD, "tmp.fifo", O_WRONLY|O_CREAT|O_TRUNC|O_CLOEXEC, 0666) = 4
fstat(4, {st_mode=S_IFIFO|0644, st_size=0, ...}) = 0
ioctl(4, TCGETS, 0x7ffcba5cd290) = -1 ENOTTY (Inappropriate ioctl for device)
lseek(4, 0, SEEK_CUR) = -1 ESPIPE (Illegal seek)
write(4, "hello0", 6) = 6
kill(35626, SIGKILL) = 0
write(4, "hello1", 6) = 6
kill(35626, SIGKILL) = 0
write(4, "hello2", 6) = -1 EPIPE (Broken pipe)
--- SIGPIPE {si_signo=SIGPIPE, si_code=SI_USER, si_pid=35625, si_uid=1000} ---
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_KILLED, si_pid=35626, si_uid=1000, si_status=SIGKILL, si_utime=0, si_stime=0} ---
write(1, "IOError is occured.\n", 20) = 20
write(4, "hello2", 6) = -1 EPIPE (Broken pipe)
--- SIGPIPE {si_signo=SIGPIPE, si_code=SI_USER, si_pid=35625, si_uid=1000} ---
close(4) = 0
write(2, "Traceback (most recent call last"..., 35) = 35
write(2, " File \"test.py\", line 26, in <m"..., 39) = 39
Note that Python tried to write hello2 twice, during fifo.flush and fifo.close respectively. The output explains why two SIGPIPE exceptions are generated well.
In order to solve the problem, we can use open('tmp.fifo', 'wb', buffering=0) to disable the write buffer. Then only one SIGPIPE exception will be generated.
BrokenPipeError in Python but not in Perl
What's going on here is that in both cases you have a process writing to a pipe whose read end was closed (by head exiting after a certain number of bytes).
This causes a SIGPIPE signal to be sent to the writing process. By default this kills the process. The process can ignore the signal if it wants to, which just makes the write call fail with an EPIPE error.
Starting with version 3.3, Python raises a BrokenPipeError exception in this case, so it looks like Python 1) ignores SIGPIPE by default and 2) translates EPIPE to a BrokenPipeError exception.
Perl does not ignore or handle signals by default. That means it gets killed by SIGPIPE in your example, but because it is not the last command in a pipeline (that would be head here), the shell just ignores it. You can make it more visible by not using a pipeline:
perl pipe.pl > >(head -n3000 >/dev/null)
This piece of bash trickery makes perl write to a pipe, but not as part of a shell pipeline. I can't test it now, but at minimum this will set $? (the command exit status) to 141 in the shell (128 + signal number, which for SIGPIPE is 13), and it may also report a Broken pipe.
You can deal with it manually in the Perl code, though:
Variant 1: Throw an error from the signal handler
$SIG{PIPE} = sub { die "BrokenPipeError" };Variant 2: Ignore the signal, handle write errors
$SIG{PIPE} = 'IGNORE';
...
print $i, "\n" or die "Can't print: $!";Note that in this case you have to think about buffering, however. If you don't enable autoflush (as in
STDOUT->autoflush(1)) and output is going to a pipe or file, Perl will collect the text in an internal buffer first (and theprintcall will succeed). Only when the buffer gets full (or when the filehandle is closed, whichever happens first) is the text actually written out and the buffer emptied. This is whyclosecan also report write errors.
Related Topics
Concatenate Numpy Arrays Without Copying
Import a Python Module Without the .Py Extension
Syntax Error When Using Command Line in Python
Difference Between Except: and Except Exception as E:
Pandas Index Column Title or Name
Adding a Legend to Pyplot in Matplotlib in the Simplest Manner Possible
Set Up Python on Windows to Not Type "Python" in Cmd
Remove Non-Numeric Rows in One Column with Pandas
How to Compute Derivative Using Numpy
How to Implement a Binary Tree
Remove Adjacent Duplicate Elements from a List
Vectorised Haversine Formula with a Pandas Dataframe
Does Python Have a Stack/Heap and How Is Memory Managed