Yes' Reporting Error With Subprocess Communicate()

yes' reporting error with subprocess communicate()

The issue is that subprocess module before Python 3.2+ doesn't restore SIGPIPE signal handler to default action. That is why you get EPIPE write error instead.

In Python 3.2+

>>> from subprocess import check_output
>>> check_output("yes | head -3", shell=True)
b'y\ny\ny\n'

yes is killed by SIGPIPE when head exits.

In Python 2:

>>> from subprocess import check_output
>>> check_output("yes | head -3", shell=True)
yes: standard output: Broken pipe
yes: write error
'y\ny\ny\n'

yes got EPIPE write error. It is safe to ignore the error. It communicates the same information as SIGPIPE.

To workaround the problem, you could emulate restore_signals in Python 2 using preexec_fn parameter :

>>> from subprocess import check_output
>>> import signal
>>> def restore_signals(): # from http://hg.python.org/cpython/rev/768722b2ae0a/
...     signals = ('SIGPIPE', 'SIGXFZ', 'SIGXFSZ')
...     for sig in signals:
...         if hasattr(signal, sig):
...            signal.signal(getattr(signal, sig), signal.SIG_DFL)
... 
>>> check_output("yes | head -3", shell=True, preexec_fn=restore_signals)
'y\ny\ny\n'

newbie python subprocess: write error: Broken pipe

From the recipes on subprocess docs:

# To replace shell pipeline like output=`dmesg | grep hda`
p1 = Popen(["dmesg"], stdout=PIPE)
p2 = Popen(["grep", "hda"], stdin=p1.stdout, stdout=PIPE)
output = p2.communicate()[0]

Python subprocess.Popen blocks with shell and pipe

So it turns out that this is caused due to a known issue with python not resetting signal handlers before exec in the subprocess module.

This is the same problem that causes 'yes' reporting error with subprocess communicate(), except with GNU yes, the EPIPE returned by a write causes the program to abort, whereas with BSD yes (which is used on OS X as far as I can tell), the return code of the write is not checked, so without a SIGPIPE, yes will not terminate.

It can be fixed by backporting the reset_signals code, as in the above linked question:

def restore_signals():
    signals = ('SIGPIPE', 'SIGXFZ', 'SIGXFSZ')
    for sig in signals:
        if hasattr(signal, sig):
            signal.signal(getattr(signal, sig), signal.SIG_DFL)

then setting preexec_fn=restore_signals in the Popen call.

Communicate with subprocess by sending int data

The argument to Popen.communicate() must be a bytes or a str object. This should work in both Python 2.x and 3.x:

proc.communicate(b'1')

or using a variable:

proc.communicate(str(myint).encode())

For the sake of completeness, to send an actual integer instead of it's ASCIIfication, you could do:

proc.communicate(bytes(bytearray([myint])))

Broken-pipe Error Python subprocess

Okay so this is a little bit obscure, but it just so happens that I ran across a similar issue while researching a question on the python-tutor mailing list some time ago.

The reason you're seeing different behavior when running your script via the subprocess module (in python) vs. bash directly, is that python overrides the disposition of SIGPIPEs to SIG_IGN (ignore) for all child processes (globally).

When the following pipeline is executed ...

awk '{print $1}' testFile2.txt | sort -g | head -1

... head will exit after it prints the first line of stdout from the sort command, due to the -1 flag. When the sort command attempts to write more lines to its stdout, a SIGPIPE is raised.

The default action of a SIGPIPE; when the pipeline is executed in a shell like bash, for example; is to terminate the sort command.

As stated earlier, python overrides the default action with SIG_IGN (ignore), so we end up with this bizarre, and somewhat inexplicable, behavior.

That's all well and good, but you might be wondering what to do now? It's dependant on the version of python you're using ...

For Python 3.2 and greater, you're already set. subprocess.Popen in 3.2 added the restore_signals parameter, which defaults to True, and effectively solves the issue without further action.

For previous versions, you can supply a callable to the preexec_fn argument of subprocess.Popen, as in ...

import signal
def default_sigpipe():
    signal.signal(signal.SIGPIPE, signal.SIG_DFL)

# ...

with open('log.txt','w') as outfile:
    CLEAN=subprocess.Popen("./bashScriptTest.sh", 
                           stdout=outfile, stderr=outfile
                           preexec_fn=default_sigpipe)

I hope that helps!

EDIT: It should probably be noted that your program is actually functioning properly, AFAICT, as is. You're just seeing additional error messages that you wouldn't normally see when executing the script in a shell directly (for the reasons stated above).

Using subprocess to get output of grep piped through head -1

The docs show you how to replace shell piping using Popen:

from subprocess import PIPE, Popen


p1 = Popen(['grep', '-n', 'some phrase', '{some file path}'],stdout=PIPE)
p2 = Popen(['head', '-1'], stdin=p1.stdout, stdout=PIPE)
p1.stdout.close()  # Allow p1 to receive a SIGPIPE if p2 exits.
out,err = output = p2.communicate()

How to check if Popen from subprocess throws an error

Both check_call and check_output should be passed a list of commands to be run by the command system (the same list of commands that would be sent to Popen). Both of these are blocking calls, meaning that Python will wait until they finish to run more code.

Using Popen sends the list of commands to the command system, but does not block further execution of Python code. You can check in on the process using the .poll method of the Popen object, or you can make a blocking call using .communicate which will return a tuple of the standard out and standard error streams.

Assuming you want the results of the executed command, and the command will properly report an error to the error stream, you can use:

process = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
out, err = process.communicate()

if err:
     print('The process raised an error:', err.decode())

Here are some examples:

Using `Popen`:

import subprocess

# first a command that works correctly
proc = subprocess.Popen(['ls', '-a'], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
out, err = proc.communicate()
if not err:
    print('--No errors--\n', out.decode())
else:
    print('--Error--\n', err.decode())

# prints:
--No errors--
anaconda3
Desktop
Documents
Downloads

# next a command, which generates an error. The `-w` switch is invalid
proc = subprocess.Popen(['ls', '-w'], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
out, err = proc.communicate()
if not err:
    print('--No errors--\n', out.decode())
else:
    print('--Error--\n', err.decode())

# prints:
--Error--
 ls: option requires an argument -- 'w'
Try 'ls --help' for more information.

Using `check_call`

check_call will raise a Python exception if the return code from the command system is not 0.

# first with a working command:
ret_code = subprocess.check_call(['ls', '-a'], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
ret_code
# returns:
0

# and now with the command that generates an error:
ret_code = subprocess.check_call(['ls', '-w'], stdout=subprocess.PIPE, stderr=subprocess.PIPE)

# raises an exception:
---------------------------------------------------------------------------
CalledProcessError                        Traceback (most recent call last)
<ipython-input-25-3cd5107991a2> in <module>()
----> 1 ret_code = subprocess.check_call(['ls', '-w'], stdout=subprocess.PIPE, 
        stderr=subprocess.PIPE)

~/anaconda3/lib/python3.6/subprocess.py in check_call(*popenargs, **kwargs)
    289         if cmd is None:
    290             cmd = popenargs[0]
--> 291         raise CalledProcessError(retcode, cmd)
    292     return 0
    293

CalledProcessError: Command '['ls', '-w']' returned non-zero exit status 2.

To handle the exception, use a try/except block.

try:
    ret_code = subprocess.check_call(['ls', '-w'], stdout=subprocess.PIPE, 
        stderr=subprocess.PIPE)
except subprocess.CalledProcessError as e:
    ret_code = e.returncode
    print('An error occurred.  Error code:', ret_code)

# prints:
An error occurred.  Error code: 2

Using `check_output`

check_output is very similar to check_call in that it will raise a Python exception if the return code from the command system is not 0. However, if the return code is 0, it will return the output in the standard output stream.

# just with the error process this time
try:
    ret_code = subprocess.check_output(['ls', '-w'], stdout=subprocess.PIPE, 
        stderr=subprocess.PIPE)
except subprocess.CalledProcessError as e:
    ret_code = e.returncode
    print('An error occurred.  Error code:', ret_code)

# prints:
An error occurred.  Error code: 2

Subproces.popen - slave write failed: Broken pipe; aborting

The bug is related to 'yes' reporting error with subprocess communicate() which provides the fix: reenable SIGPIPE signal in the child process using preexec_fn (or use Python 3).

Unrelated: your code can deadlock if catch_stderr is true and p.stderr and p.stdout are not perfectly in sync.

Otherwise catch_stderr has no effect (ignoring buffering): your code captures stderr regardless. You could simplify it:

#!/usr/bin/env python
from shutil import copyfileobj
from subprocess import Popen, PIPE, STDOUT

def run_and_print(command, cwd=None):
    p = Popen(command, stdout=PIPE, stderr=STDOUT, bufsize=-1, cwd=cwd,
              preexec_fn=restore_signals)
    with p.stdout:
        tee = Tee()
        copyfileobj(p.stdout, tee)        
    return p.wait(), tee.getvalue()

where Tee() is a file-like object that writes to two places: to stdout and to StringIO():

import sys
from io import BytesIO

class Tee:
   def __init__(self):
       self.file = BytesIO()
   def write(self, data):
       getattr(sys.stdout, 'buffer', sys.stdout).write(data)
       self.file.write(data)
   def getvalue(self):
       return self.file.getvalue()

where restore_signals() is defined here.

If you want to see the output on the screen as soon as command prints them; you could inline Tee, copyfileobj() and use os.read(), to avoid reading the complete length before writing it to stdout:

chunks = []
with p.stdout:
    for chunk in iter(lambda: os.read(p.stdout.fileno(), 1 << 13), b''):
        getattr(sys.stdout, 'buffer', sys.stdout).write(chunk)
        sys.stdout.flush()
        chunks.append(chunk)
return p.wait(), b''.join(chunks)

To disable the internal block-buffering in the child process, you might try to run it using stdbuf or pass pseudo-tty instead of the pipe.

Hudson : yes: standard output: Broken pipe

But how would you explain that I dont get this error while running the script locally, but I get the error when running it remotely from a Hudson job?

When you are running it in a terminal (locally); yes is killed by SIGPIPE signal that is generated when it tries to write to the pipe when MyScript.sh has already exited.

Whatever runs the command (remotely) in Hudson traps that signal (set its handler to SIG_IGN, you can test it by running trap command and searching for SIGPIPE in the output) and it doesn't restore the signal for new child processes (yes and whatever runs MyScript.sh e.g., sh in your case). It leads to the write error (EPIPE) instead of the signal. yes detects the write error and reports it.

You can simply ignore the error message:

yes 2>/dev/null | ./MyScript.sh

You could also report the bug against the component that runs the pipeline. The bug is in not restoring SIGPIPE to the default handler after the child is forked. It is what programs expect when they are run in a terminal on POSIX systems. Though I don't know whether there is a standard way to do it for a java-based program. jvm probably raises an exception for every write error so not-dying on SIGPIPE is not a problem for a java program.

It is common for daemons such as hudson process to ignore SIGPIPE signal. You don't want your daemon to die only because the process you are communicating with dies and you would check for write errors anyway.

Ordinary programs that are written to be run in a terminal do not check status of every printf() for errors but you want them to die if programs down the pipeline die e.g., if you run source | sink pipeline; usually you want source process to exit as soon as possible if sink exits.

EPIPE write error is returned if SIGPIPE signal is disabled (as it looks like in hudson's case) or if a program does not die on receiving it (yes program does not defined any handlers for SIGPIPE so it should die on receiving the signal).

I don't want to ignore the error, I want to do the right command or fix to get rid of the error.

the only way yes process stops if it is killed or encountered a write error. If SIGPIPE signal is set to be ignored (by the parent) and no other signal kills the process then yes receives write error on ./MyScript.sh exit. There are no other options if you use yes program.

SIGPIPE signal and EPIPE error communicate the exact same information -- pipe is broken. If SIGPIPE were enabled for yes process then you wouldn't see the error. And only because you see it; nothing new happens. It just means that ./MyScript.sh exited (successfully or unsuccessfully -- doesn't matter).

Yes' Reporting Error With Subprocess Communicate()