Calling the source command from subprocess.Popen
source
is not an executable command, it's a shell builtin.
The most usual case for using source
is to run a shell script that changes the environment and to retain that environment in the current shell. That's exactly how virtualenv works to modify the default python environment.
Creating a sub-process and using source
in the subprocess probably won't do anything useful, it won't modify the environment of the parent process, none of the side-effects of using the sourced script will take place.
Python has an analogous command, execfile
, which runs the specified file using the current python global namespace (or another, if you supply one), that you could use in a similar way as the bash command source
.
Python's subprocess.Popen() and source
source
uses $PATH
to find what you pass to it, if you don't specify a directory. Try source ./houdini_setup
.
Using python subprocess to use the source linux command with a bashrc file
def shell_source(script):
"""Sometime you want to emulate the action of "source" in bash,
settings some environment variables. Here is a way to do it."""
import subprocess, os
pipe = subprocess.Popen(". %s; env" % script, stdout=subprocess.PIPE, shell=True)
output = pipe.communicate()[0]
env = dict((line.split("=", 1) for line in output.splitlines()))
os.environ.update(env)
shell_source(adastralrc.sh)
Looks like a duplicate
How to source script via python
There's absolutely nothing wrong with this in and of itself:
# What you're already doing -- this is actually fine!
sourcevars = "cd /etc/openvpn/easy-rsa && . ./vars"
runSourcevars = subprocess.Popen(sourcevars, shell=True)
# ...*however*, it won't have any effect at all on this:
runOther = subprocess.Popen('./easy-rsa build-key yadda yadda', shell=True)
However, if you subsequently try to run a second subprocess.Popen(..., shell=True)
command, you'll see that it doesn't have any of the variables set by sourcing that configuration.
This is entirely normal and expected behavior: The entire point of using source
is to modify the state of the active shell; each time you create a new Popen
object with shell=True
, it's starting a new shell -- their state isn't carried over.
Thus, combine into a single call:
prefix = "cd /etc/openvpn/easy-rsa && . ./vars && "
cmd = "/etc/openvpn/easy-rsa/clean-all"
runCmd = subprocess.Popen(prefix + cmd, shell=True)
...such that you're using the results of sourcing the script in the same shell invocation as that in which you actually source the script.
Alternately (and this is what I'd do), require your Python script to be invoked by a shell which already has the necessary variables in its environment. Thus:
# ask your users to do this
set -a; . ./vars; ./yourPythonScript
...and you can error out if people don't do so very easy:
import os, sys
if not 'EASY_RSA' in os.environ:
print >>sys.stderr, "ERROR: Source vars before running this script"
sys.exit(1)
Python's sh module - Running 'source' command
To invoke sh -c "source hello"
via the Python sh
module (ignoring that sh
is not guaranteed to support the source
command at all -- the POSIX-compliant equivalent is . hello
):
sh.sh('-c', 'source test.sh')
That said, consider using subprocess.Popen()
instead, which contains far less magic and thus behaves more predictably:
# ...if hardcoding the script to source in your Python code
# ...because shell=True uses /bin/sh instead of bash, use . instead of source
subprocess.Popen('. ./test.sh', shell=True)
# ...otherwise:
# ...because shell=True uses /bin/sh instead of bash, use . instead of source
subprocess.Popen(['. "$@"', './test.sh'], shell=True)
When an array is passed to the first argument of subprocess.Popen
, the first argument is treated as the source to run, and subsequent arguments became $1
, $2
, etc. during operation of that script, allowing the combination of an array of string literals with /bin/sh
invoked via the shell=True
.
However: The intent and purpose behind sourcing content into a shell is generally to modify that shell's state. With either sh.sh
or subprocess.Popen()
, the shell only lasts as long as that single Python function call's invocation, so no state persists to future sh
or subprocess
calls, making it unlikely that any of these uses will actually accomplish your goals at hand.
What you really want is probably more like this:
sh.sh('-c', '. ./test.sh; do-something-else-here')
...where your do-something-else-here
depends on changes to the shell and its environment made by source ./test.sh
.
Python subprocess.call raise OSError with linux source command
source
is a shell builtin command. Python's subprocess module is provided to spawn new process, not running a proper Bourne shell (or zsh or ksh or etc.). You cannot access shel builtins from subprocess.call
.
To determine if you can or not run a specific command with subprocess
module, you may want to use which
to get information about the command you need to use:
user@machine: ~
$ which source [7:41:38]
source: shell built-in command
user@machine: ~
$ which cat [7:41:42]
/bin/cat
user@machine: ~
$ which ls [7:41:47]
ls: aliased to ls --color=tty
Using Source Command From A Python Script
subprocess
is there for executing things. It doesn't give you a bridge between shell functions and Python. You can't "load" a shell file and then treat it as a Python object you can call. You can just execute the contents of the script.
how to make subprocess called with call/Popen inherit environment variables
Regarding
If I were doing this directly at the command line, I'd "source" a script called mySetUpFreeSurfer.sh that does nothing but set three environment variables, and then "source" another script, FreeSurferEnv.sh.
I think you would be better off using Python to automate the process of writing
a shell script newscript.sh
, and then calling this script with one callsubprocess.check_output
(instead of many calls to Popen
, check_output
,call
, etc.):
newscript.sh:
#!/bin/bash
source ~/scripts/mySetUpFreeSurfer.sh
source /usr/local/freesurfer/FreeSurferEnv.sh
recon-all -i /media/foo/bar -subjid s1001
...
and then calling
subprocess.check_output(['newscript.sh'])
import subprocess
import tempfile
import os
import stat
with tempfile.NamedTemporaryFile(mode='w', delete=False) as f:
f.write('''\
#!/bin/bash
source ~/scripts/mySetUpFreeSurfer.sh
source /usr/local/freesurfer/FreeSurferEnv.sh
''')
root = "/media/foo/"
for sub_dir in os.listdir(root):
sub = "s" + sub_dir[0:4]
anat_dir = os.path.join(root, sub_dir, "anatomical")
for directory in os.listdir(anat_dir):
time_dir = os.path.join(anat_dir, directory)
for d in os.listdir(time_dir):
dicoms_dir = os.path.join(time_dir, d, 'dicoms')
dicom_list = os.listdir(dicoms_dir)
dicom = dicom_list[0]
path = os.path.join(dicoms_dir, dicom)
cmd1 = "recon-all -i {} -subjid {}\n".format(path, sub)
f.write(cmd1)
cmd2 = "recon-all -all -subjid {}\n".format(sub)
f.write(cmd2)
filename = f.name
os.chmod(filename, stat.S_IRUSR | stat.S_IXUSR)
subprocess.call([filename])
os.unlink(filename)
By the way,
def source(script, update=1):
pipe = Popen(". %s; env" % script, stdout=PIPE, shell=True)
data = pipe.communicate()[0]
env = dict((line.split("=", 1) for line in data.splitlines()))
if update:
os.environ.update(env)
return env
is broken. For example, if script
contains something like
VAR=`ls -1`
export VAR
then
. script; env
may return output like
VAR=file1
file2
file3
which will result in source(script)
raising a ValueError
:
env = dict((line.split("=", 1) for line in data.splitlines()))
ValueError: dictionary update sequence element #21 has length 1; 2 is required
There is a way to fix source
: have env
separate environment variables with a zero byte instead of the ambiguous newline:
def source(script, update=True):
"""
http://pythonwise.blogspot.fr/2010/04/sourcing-shell-script.html (Miki Tebeka)
http://stackoverflow.com/questions/3503719/#comment28061110_3505826 (ahal)
"""
import subprocess
import os
proc = subprocess.Popen(
['bash', '-c', 'set -a && source {} && env -0'.format(script)],
stdout=subprocess.PIPE, shell=False)
output, err = proc.communicate()
output = output.decode('utf8')
env = dict((line.split("=", 1) for line in output.split('\x00') if line))
if update:
os.environ.update(env)
return env
Fixable or not, however, you are still probably better off constructing a
conglomerate shell script (as shown above) than you would be parsing env
and
passing env
dicts to subprocess
calls.
Related Topics
Purpose of Calling Function Without Brackets Python
Finding What Methods a Python Object Has
Control the Size Textarea Widget Look in Django Admin
Equivalent of a Python Dict in R
Ruby Hash Equivalent to Python Dict Setdefault
Dead Simple Example of Using Multiprocessing Queue, Pool and Locking
MAC Osx Python Ssl.Sslerror: [Ssl: Certificate_Verify_Failed] Certificate Verify Failed (_Ssl.C:749)
Reversing 'One-Hot' Encoding in Pandas
Split by Comma and Strip Whitespace in Python
Is the List of Python Reserved Words and Builtins Available in a Library
Pandas: Peculiar Performance Drop for Inplace Rename After Dropna
Logger Configuration to Log to File and Print to Stdout
Matplotlib Table Formatting Column Width
Dealing with the Class Imbalance in Binary Classification
Aes Python Encryption and Ruby Encryption - Different Behaviour
Log All Requests from the Python-Requests Module
How to Change the Default MySQL Connection Timeout When Connecting Through Python