Run Linux Grep Command from Python Subprocess

Python Subprocess Grep

I think you're running up against two problems:

1. This call:

   p = subprocess.Popen(['grep', "%s *.log"%userid]...
   

   will not work as expected without shell=True because the list of arguments are being passed directly to os.execvp, which requires each item to be a single string representing an argument. You've squished two separate arguments together into a single string (in other words, grep is interpreting "foo12 *.log" as the pattern to search, and not pattern+file list).

   You can fix this by saying:

   p = subprocess.Popen(['grep', userid, '*.log']...)
   

2. The second issue is that, again without shell=True, execvp doesn't know what you mean by *.log and passes it directly along to grep, without going through the shell's wildcard expansion mechanism. If you don't want to use shell=True, you can instead do something like:

   import glob
   args = ['grep', userid]
   args.extend(glob.glob('*.log')
   p = subprocess.Popen(args, ...)
   

run linux grep command from python subprocess

This has been gone over many, many times before; but here is a simple pure Python replacement for the inefficient postprocessing.

from subprocess import Popen, PIPE
eth1 = subprocess.Popen(['/sbin/ifconfig', 'eth1'], stdout=PIPE)
out, err = eth1.communicate()
for line in out.split('\n'):
    line = line.lstrip()
    if line.startswith('inet addr:'):
        ip = line.split()[1][5:]

Cant use grep in subprocess command

When you split the text, the list will look like

['mosquitto_sub', ..., 'ITT/#', '-v', '|', 'grep', '"Online"']

When you pass this list to subprocess.Popen, a literal '|' will be one of the arguments to mosquitto_sub.

If you use shell=True, you must escape any special characters like # in the command, for instance with double quotes:

import subprocess
command = 'echo -e "ITT/#\\ni am Online\\nbar Online\\nbaz" | grep "Online" '
p = subprocess.Popen(
    command, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
for line in iter(p.stdout.readline, b''):
    print(line)

Alternatively, connect the pipes as you wrote, but make sure to iterate until b'', not u'':

import subprocess

def run_command(command, command2):
    p1 = subprocess.Popen(command,stdout=subprocess.PIPE,stderr=subprocess.STDOUT)
    p2 = subprocess.Popen(command2,stdin=p1.stdout,stdout=subprocess.PIPE)
    return iter(p2.stdout.readline, b'')

command = ['echo', '-e', 'ITT/#\\ni am Online\\nbar Online\\nbaz']
command2 = 'grep Online'.split()

for line in run_command(command,command2):
    print(line)

Using the subprocess module to grep with the | character

The result of command.split() contains quotes which should no longer be there. That's why Python provides shlex.split, but it's also not hard to understand how to split the command manually, though obviously you need to understand the role of the quotes in the shell, and how basically you need to remove them when there is no shell.

command = 'grep "one\|three" ./file.txt'
results1 = subprocess.check_output(['grep', r'one\|three', './file.txt'])
results2 = subprocess.check_output(shlex.split(command))
results3 = subprocess.check_output(command, shell=True) # better avoid

Quotes tell the shell to not perform whitespace tokenization and/or wildcard expansion on a value, but when there is no shell, you should simply provide a string instead where the shell allowed or even required you to use a quoted string.

Python subprocess library: Running grep command from Python

On UNIX, specifying shell=True will cause the first argument to be treated as the command to execute, with all subsequent arguments treated as arguments to the shell itself. Thus, the > won't have any effect (since with /bin/sh -c, all arguments after the command are ignored).

Therefore, you should actually use

catdoc_cmd = ['catdoc -w "%s" > testing.txt' % name]

A better solution, though, would probably be to just read the text out of the subprocess' stdout, and process it using re or Python string operations:

catdoc_cmd = ['catdoc', '-w' , name]
catdoc_process = subprocess.Popen(catdoc_cmd, stdout=subprocess.PIPE,stderr=subprocess.PIPE)
for line in catdoc_process.stdout:
    if keyword in line:
        print line.strip()

python shell command pipes grep no results

Your script fails because of the quotes around DNS=.

You can use shell=True to make the script work:

dnslineprocess = subprocess.Popen(getdnsline, stdin=userdataprocess.stdout, stdout=subprocess.PIPE, shell=True)

Source: Passing double quote shell commands in python to subprocess.Popen()?

Using subprocess to get output of grep piped through head -1

The docs show you how to replace shell piping using Popen:

from subprocess import PIPE, Popen

p1 = Popen(['grep', '-n', 'some phrase', '{some file path}'],stdout=PIPE)
p2 = Popen(['head', '-1'], stdin=p1.stdout, stdout=PIPE)
p1.stdout.close()  # Allow p1 to receive a SIGPIPE if p2 exits.
out,err = output = p2.communicate()

How to run ps cax | grep something in Python?

See Replacing shell pipeline:

import subprocess

proc1 = subprocess.Popen(['ps', 'cax'], stdout=subprocess.PIPE)
proc2 = subprocess.Popen(['grep', 'python'], stdin=proc1.stdout,
                         stdout=subprocess.PIPE, stderr=subprocess.PIPE)

proc1.stdout.close() # Allow proc1 to receive a SIGPIPE if proc2 exits.
out, err = proc2.communicate()
print('out: {0}'.format(out))
print('err: {0}'.format(err))

PS. Using shell=True can be dangerous. See for example the warning in the docs.

---

There is also the sh module which can make subprocess scripting in Python a lot more pleasant:

import sh
print(sh.grep(sh.ps("cax"), 'something'))

---

Related Topics