File Not Found Error When Launching a Subprocess Containing Piped Commands

File not found error when launching a subprocess containing piped commands

You have to add shell=True to execute a shell command. check_output is trying to find an executable called: date | grep -o -w '"+tz+"'' | wc -w and he cannot find it. (no idea why you removed the essential information from the error message).

See the difference between:

>>> subprocess.check_output('date | grep 1')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.4/subprocess.py", line 603, in check_output
    with Popen(*popenargs, stdout=PIPE, **kwargs) as process:
  File "/usr/lib/python3.4/subprocess.py", line 848, in __init__
    restore_signals, start_new_session)
  File "/usr/lib/python3.4/subprocess.py", line 1446, in _execute_child
    raise child_exception_type(errno_num, err_msg)
FileNotFoundError: [Errno 2] No such file or directory: 'date | grep 1'

And:

>>> subprocess.check_output('date | grep 1', shell=True)
b'gio 19 giu 2014, 14.15.35, CEST\n'

Read the documentation about the Frequently Used Arguments for more information about the shell argument and how it changes the interpretation of the other arguments.

---

Note that you should try to avoid using shell=True since spawning a shell can be a security hazard (even if you do not execute untrusted input attacks like Shellshock can still be performed!).

The documentation for the subprocess module has a little section about replacing the shell pipeline.
You can do so by spawning the two processes in python and use subprocess.PIPE:

date_proc = subprocess.Popen(['date'], stdout=subprocess.PIPE)
grep_proc = subprocess.check_output(['grep', '1'], stdin=date_proc.stdout, stdout=subprocess.PIPE)
date_proc.stdout.close()
output = grep_proc.communicate()[0]

You can write some simple wrapper function to easily define pipelines:

import subprocess
from shlex import split
from collections import namedtuple
from functools import reduce

proc_output = namedtuple('proc_output', 'stdout stderr')

def pipeline(starter_command, *commands):
    if not commands:
        try:
            starter_command, *commands = starter_command.split('|')
        except AttributeError:
            pass
    starter_command = _parse(starter_command)
    starter = subprocess.Popen(starter_command, stdout=subprocess.PIPE)
    last_proc = reduce(_create_pipe, map(_parse, commands), starter)
    return proc_output(*last_proc.communicate())

def _create_pipe(previous, command):
    proc = subprocess.Popen(command, stdin=previous.stdout, stdout=subprocess.PIPE)
    previous.stdout.close()
    return proc

def _parse(cmd):
    try:
        return split(cmd)
    except Exception:
        return cmd

With this in place you can write pipeline('date | grep 1') or pipeline('date', 'grep 1') or pipeline(['date'], ['grep', '1'])

FileNotFound error when executing subprocess.run()

Like Jean-François Fabre said, the solution is to add "shell=True" to the call

import subprocess
result = subprocess.run("echo \"blah\"", cwd=directory, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True)

shell=True seems to tell subprocess to use the string as a command.

subprocess: FileNotFound

FileNotFoundError: [Errno 2] No such file or directory: 'command'

command is a shell builtin. subprocess.Popen does NOT run the shell by default.

To run the shell, pass shell=True:

>>> import subprocess
>>> subprocess.check_output('command -v python', shell=True)
b'/usr/bin/python\n'

To find the full path to an executable, you could use shutil.which() instead:

>>> import shutil
>>> shutil.which('python')
'/usr/bin/python'

Subprocess command shows FileNotFoundError: [Errno 2] No such file or directory

Your commands are still wrong. If you just want to run these commands like the shell does, the absolutely easiest way to do that is to ... use the shell.

result = subprocess.run('''
# useless cat, but bear with
cat output3.txt |
sed 's/"/ /g' |
grep "shipOption" |
grep -v "getShipMethod" |
cut -d ',' -f2 |
sed 's/"//g' |
sort |
uniq -c |
sort -nr |
head -10
    ''',
    # Probably add these too
    check=True,
    capture_output=True,
    # We are using the shell for piping etc
    shell=True)

If you want to remove the shell=True and manually run all these processes, you have to understand how the shell works. In particular, you need to fix the quoting so that the commands you run have the quotes which remain after the shell has processed the syntactic quotes.

p1 = subprocess.Popen(['cat', 'output3.txt'], stdout=subprocess.PIPE)  # still useless
p2 = subprocess.Popen(['sed','s/"/ /g'], stdin=p1.stdout, stdout=subprocess.PIPE)
p3 = subprocess.Popen(['grep', "shipOption"], stdin=p2.stdout,stdout=subprocess.PIPE)
p4 = subprocess.Popen(['grep', '-v', "getShipMethod"], stdin=p3.stdout, stdout=subprocess.PIPE)
p5 = subprocess.Popen(['cut', '-d', ',', '-f2'], stdin=p4.stdout, stdout=subprocess.PIPE)
p6 = subprocess.Popen(['sed', 's/"//g'],stdin=p5.stdout, stdout=subprocess.PIPE)
p7 = subprocess.Popen(['sort'], stdin=p6.stdout, stdout=subprocess.PIPE)
p8 = subprocess.Popen(['uniq', '-c'], stdin=p7.stdout, stdout=subprocess.PIPE)
p9 = subprocess.Popen(['sort', '-nr'], stdin=p8.stdout, stdout=subprocess.PIPE)
p0 = subprocess.Popen(['head', '-10'], stdin=p9.stdout, stdout=subprocess.PIPE)

Notice in particular how the arguments to sed and grep have their outer quotes removed, and how we removed shell=True everywhere. As a rule of thumb, if the first argument to Popen (or other subprocess methods) is a list, you should not use shell=True, and vice versa. (There are situations where you can pass a list to shell=True but ... let's not even begin to go there.)

All of this seems rather moot, though, since Python can eminently well do all of these things.

from collections import Counter

counts = Counter()
with open('output3.txt', 'r', encoding='utf-8') as lines:
    for line in lines:
        line = line.rstrip('\n').replace('"', ' ')
        if "shipOption" in line and "getShipMethod" not in line:
            field = line.split(',')[1].replace('"', '')
            counts[field] += 1
print(counts.most_common(10))

Probably you would want to put the rstrip and replace inside the if to avoid unnecessary work. The same refactoring could be done to the shell pipeline, of course.

FileNotFoundError: subprocess.Popen(['dir'] (Windows 7)

As dir is simply a command understood by cmd.exe (or powershell.exe) then you could:

subprocess.Popen(["cmd", "/c", "dir", "c:\\Users"], stdout=subprocess.PIPE)

which corresponds to doing the following in a shell

C:\>cmd /c dir c:\Users

You may find you have to fully path cmd, as c:\\Windows\\System32\\cmd.exe

FileNotFoundError from subprocess.run when executing command from /usr/bin/* - Python

After troubleshooting I found that problem was not related to Python or its subprocess module, but actually it was due to Firefox being installed as snap package in Ubuntu. I don't know much about snap packages but they provide confined environment for installed application.

So, native app (Python script) is launched by extension or Firefox snap in that confined environment and it is unable to find system programs. Also, the Python traceback mention of Python3.5 was due to because script was interpreted by snap packaged version of Python in /snap/core/current/usr/bin/python3 instead of system global Python. I guess equivalent directory for snap packages are snap/core/current/bin/ and /snap/core/current/usr/bin/. Also, you're not allowed to modify snap directories, so dig can't be copied over.

There may exist alternate solution(s) but I decided to remove Firefox snap and reinstall Firefox as regular program from official Mozilla website. Now, everything works fine as expected.

---

Related Topics