Calling the "Source" Command from Subprocess.Popen

Calling the source command from subprocess.Popen

source is not an executable command, it's a shell builtin.

The most usual case for using source is to run a shell script that changes the environment and to retain that environment in the current shell. That's exactly how virtualenv works to modify the default python environment.

Creating a sub-process and using source in the subprocess probably won't do anything useful, it won't modify the environment of the parent process, none of the side-effects of using the sourced script will take place.

Python has an analogous command, execfile, which runs the specified file using the current python global namespace (or another, if you supply one), that you could use in a similar way as the bash command source.

Python's subprocess.Popen() and source

source uses $PATH to find what you pass to it, if you don't specify a directory. Try source ./houdini_setup.

Using python subprocess to use the source linux command with a bashrc file

def shell_source(script):
"""Sometime you want to emulate the action of "source" in bash,
settings some environment variables. Here is a way to do it."""
import subprocess, os
pipe = subprocess.Popen(". %s; env" % script, stdout=subprocess.PIPE, shell=True)
output = pipe.communicate()[0]
env = dict((line.split("=", 1) for line in output.splitlines()))
os.environ.update(env)

shell_source(adastralrc.sh)

Looks like a duplicate

How to source script via python

There's absolutely nothing wrong with this in and of itself:

# What you're already doing -- this is actually fine!
sourcevars = "cd /etc/openvpn/easy-rsa && . ./vars"
runSourcevars = subprocess.Popen(sourcevars, shell=True)

# ...*however*, it won't have any effect at all on this:
runOther = subprocess.Popen('./easy-rsa build-key yadda yadda', shell=True)

However, if you subsequently try to run a second subprocess.Popen(..., shell=True) command, you'll see that it doesn't have any of the variables set by sourcing that configuration.

This is entirely normal and expected behavior: The entire point of using source is to modify the state of the active shell; each time you create a new Popen object with shell=True, it's starting a new shell -- their state isn't carried over.

Thus, combine into a single call:

prefix = "cd /etc/openvpn/easy-rsa && . ./vars && "
cmd = "/etc/openvpn/easy-rsa/clean-all"
runCmd = subprocess.Popen(prefix + cmd, shell=True)

...such that you're using the results of sourcing the script in the same shell invocation as that in which you actually source the script.


Alternately (and this is what I'd do), require your Python script to be invoked by a shell which already has the necessary variables in its environment. Thus:

# ask your users to do this
set -a; . ./vars; ./yourPythonScript

...and you can error out if people don't do so very easy:

import os, sys
if not 'EASY_RSA' in os.environ:
print >>sys.stderr, "ERROR: Source vars before running this script"
sys.exit(1)

Python's sh module - Running 'source' command

To invoke sh -c "source hello" via the Python sh module (ignoring that sh is not guaranteed to support the source command at all -- the POSIX-compliant equivalent is . hello):

sh.sh('-c', 'source test.sh')

That said, consider using subprocess.Popen() instead, which contains far less magic and thus behaves more predictably:

# ...if hardcoding the script to source in your Python code
# ...because shell=True uses /bin/sh instead of bash, use . instead of source
subprocess.Popen('. ./test.sh', shell=True)

# ...otherwise:
# ...because shell=True uses /bin/sh instead of bash, use . instead of source
subprocess.Popen(['. "$@"', './test.sh'], shell=True)

When an array is passed to the first argument of subprocess.Popen, the first argument is treated as the source to run, and subsequent arguments became $1, $2, etc. during operation of that script, allowing the combination of an array of string literals with /bin/sh invoked via the shell=True.


However: The intent and purpose behind sourcing content into a shell is generally to modify that shell's state. With either sh.sh or subprocess.Popen(), the shell only lasts as long as that single Python function call's invocation, so no state persists to future sh or subprocess calls, making it unlikely that any of these uses will actually accomplish your goals at hand.

What you really want is probably more like this:

sh.sh('-c', '. ./test.sh; do-something-else-here')

...where your do-something-else-here depends on changes to the shell and its environment made by source ./test.sh.

Python subprocess.call raise OSError with linux source command

source is a shell builtin command. Python's subprocess module is provided to spawn new process, not running a proper Bourne shell (or zsh or ksh or etc.). You cannot access shel builtins from subprocess.call.

To determine if you can or not run a specific command with subprocess module, you may want to use which to get information about the command you need to use:

user@machine: ~
$ which source [7:41:38]
source: shell built-in command

user@machine: ~
$ which cat [7:41:42]
/bin/cat

user@machine: ~
$ which ls [7:41:47]
ls: aliased to ls --color=tty

Using Source Command From A Python Script

subprocess is there for executing things. It doesn't give you a bridge between shell functions and Python. You can't "load" a shell file and then treat it as a Python object you can call. You can just execute the contents of the script.

how to make subprocess called with call/Popen inherit environment variables

Regarding

If I were doing this directly at the command line, I'd "source" a script called mySetUpFreeSurfer.sh that does nothing but set three environment variables, and then "source" another script, FreeSurferEnv.sh.

I think you would be better off using Python to automate the process of writing
a shell script newscript.sh, and then calling this script with one call
subprocess.check_output (instead of many calls to Popen, check_output,
call, etc.):

newscript.sh:

#!/bin/bash
source ~/scripts/mySetUpFreeSurfer.sh
source /usr/local/freesurfer/FreeSurferEnv.sh
recon-all -i /media/foo/bar -subjid s1001
...

and then calling

subprocess.check_output(['newscript.sh'])

import subprocess
import tempfile
import os
import stat

with tempfile.NamedTemporaryFile(mode='w', delete=False) as f:
f.write('''\
#!/bin/bash
source ~/scripts/mySetUpFreeSurfer.sh
source /usr/local/freesurfer/FreeSurferEnv.sh
''')
root = "/media/foo/"
for sub_dir in os.listdir(root):
sub = "s" + sub_dir[0:4]
anat_dir = os.path.join(root, sub_dir, "anatomical")
for directory in os.listdir(anat_dir):
time_dir = os.path.join(anat_dir, directory)
for d in os.listdir(time_dir):
dicoms_dir = os.path.join(time_dir, d, 'dicoms')
dicom_list = os.listdir(dicoms_dir)
dicom = dicom_list[0]
path = os.path.join(dicoms_dir, dicom)
cmd1 = "recon-all -i {} -subjid {}\n".format(path, sub)
f.write(cmd1)
cmd2 = "recon-all -all -subjid {}\n".format(sub)
f.write(cmd2)

filename = f.name
os.chmod(filename, stat.S_IRUSR | stat.S_IXUSR)
subprocess.call([filename])
os.unlink(filename)

By the way,

def source(script, update=1):
pipe = Popen(". %s; env" % script, stdout=PIPE, shell=True)
data = pipe.communicate()[0]
env = dict((line.split("=", 1) for line in data.splitlines()))
if update:
os.environ.update(env)
return env

is broken. For example, if script contains something like

VAR=`ls -1`
export VAR

then

. script; env

may return output like

VAR=file1
file2
file3

which will result in source(script) raising a ValueError:

env = dict((line.split("=", 1) for line in data.splitlines()))
ValueError: dictionary update sequence element #21 has length 1; 2 is required

There is a way to fix source: have env separate environment variables with a zero byte instead of the ambiguous newline:

def source(script, update=True):
"""
http://pythonwise.blogspot.fr/2010/04/sourcing-shell-script.html (Miki Tebeka)
http://stackoverflow.com/questions/3503719/#comment28061110_3505826 (ahal)
"""
import subprocess
import os
proc = subprocess.Popen(
['bash', '-c', 'set -a && source {} && env -0'.format(script)],
stdout=subprocess.PIPE, shell=False)
output, err = proc.communicate()
output = output.decode('utf8')
env = dict((line.split("=", 1) for line in output.split('\x00') if line))
if update:
os.environ.update(env)
return env

Fixable or not, however, you are still probably better off constructing a
conglomerate shell script (as shown above) than you would be parsing env and
passing env dicts to subprocess calls.



Related Topics



Leave a reply



Submit