Python Subprocess/Popen with a Modified Environment

Python subprocess/Popen with a modified environment

I think os.environ.copy() is better if you don't intend to modify the os.environ for the current process:

import subprocess, os
my_env = os.environ.copy()
my_env["PATH"] = "/usr/sbin:/sbin:" + my_env["PATH"]
subprocess.Popen(my_command, env=my_env)

Python subprocess.Popen pass modified environment to use new environment varibels in postinst script

Well I just learned that when running a command with sudo the local user environment variables are not the same as the root user.

In order to preserve the environment variables when running a command with sudo, you should add the flag -E as described in this answer how-to-keep-environment-variables-when-using-sudo.

I copied from how-to-keep-environment-variables-when-using-sudo the quote from the man page:

-E, --preserve-env

Indicates to the security policy that the user wishes to preserve their
existing environment variables. The security policy may return an error
if the user does not have permission to preserve the environment.

Once I changed the command string to sudo -E dpkg -i path-to-package, the postinst script was able to print out the variables as I expected:

CUSTOM_PORT_1: 7755
CUSTOM_PORT_2: 7766

APP_PORT_1: 7755
APP_PORT_2: 7766

Start subprocess with environment variables

Use: subprocess.Popen('env', env={'A':'5','B':'5'})

Sample run:

>>> import subprocess
>>> subprocess.Popen('env', env={'A':'5','B':'5'})
<subprocess.Popen object at 0x102288950>
>>> A=5
B=5

From the docs:

If env is not None, it must be a mapping that defines the environment
variables for the new process; these are used instead of inheriting
the current process’ environment, which is the default behavior.

Python3 subprocess.run() is using the same Popen kwarg env:

The full function signature is largely the same as that of the Popen
constructor

Python: Exporting environment variables in subprocess.Popen(..)

The substitution of environment variables on the command line is done by the shell, not by /bin/echo. So you need to run the command in a shell to get the substitution:

In [22]: subprocess.Popen('/bin/echo $TEST_VARIABLE', shell=True, env=d).wait()
1234
Out[22]: 0

That doesn't mean the environment variable is not set when shell=False, however. Even without shell=True, the executable does see the enviroment variables set by the env parameter. For example, date is affected by the TZ environment variable:

In [23]: subprocess.Popen(["date"], env={'TZ': 'America/New_York'}).wait()
Wed Oct 29 22:05:52 EDT 2014
Out[23]: 0

In [24]: subprocess.Popen(["date"], env={'TZ': 'Asia/Taipei'}).wait()
Thu Oct 30 10:06:05 CST 2014
Out[24]: 0

Python - subprocess and the env argument

Pass the env parameter to the subprocess function (Popen in your case). To modify the environment of the calling process, make a copy of os.environ. For example:

sub_env = os.environ.copy()
for key in vars(args):
    if key.startswith('var_'):
        sub_env[key] = args[key]
for machine in vars(args)['foovar']:
    exit_code = subprocess.Popen("…", env=sub_env, …)

How to use an existing Environment variable in subprocess.Popen()

You could do:

cmdlist = ['echo','param',os.environ["PARAM"]]

Or:

cmdlist = ['echo','param1','$PARAM']
proc = subprocess.Popen(cmdlist,stdout=subprocess.PIPE, env={'PARAM':os.environ['PARAM'])

Why are PATH variables not effecting subprocess.Popen in Windows

Thanks to the comment by VGR, I took a more studied approach to the big red warning box on in the subprocess documentation here which states.

For Windows, see the documentation of the lpApplicationName and
lpCommandLine parameters of WinAPI CreateProcess, and note that when
resolving or searching for the executable path with shell=False, cwd
does not override the current working directory and env cannot
override the PATH environment variable. Using a full path avoids all
of these variations.

I guess I did not fully appreciate the last sentence, which says in Windows the PATH does not get overridden and the full path to the executable should be used to avoid confusion between executable names. This behavior explains what I am seeing.

python subprocess Popen environment PATH?

Relative paths (paths containing slashes) never get checked in any PATH, no matter what you do. They are relative to the current working directory only. If you need to resolve relative paths, you will have to search through the PATH manually.

If you want to run a program relative to the location of the Python script, use __file__ and go from there to find the absolute path of the program, and then use the absolute path in Popen.

Searching in the current process' environment variable PATH

There is an issue in the Python bug tracker about how Python deals with bare commands (no slashes). Basically, on Unix/Mac Popen behaves like os.execvp when the argument env=None (some unexpected behavior has been observed and noted at the end):

On POSIX, the class uses os.execvp()-like behavior to execute the child program.

This is actually true for both shell=False and shell=True, provided env=None. What this behavior means is explained in the documentation of the function os.execvp:

The variants which include a “p” near the end (execlp(), execlpe(), execvp(), and execvpe()) will use the PATH environment variable to locate the program file. When the environment is being replaced (using one of the exec*e variants, discussed in the next paragraph), the new environment is used as the source of the PATH variable.

For execle(), execlpe(), execve(), and execvpe() (note that these all end in “e”), the env parameter must be a mapping which is used to define the environment variables for the new process (these are used instead of the current process’ environment); the functions execl(), execlp(), execv(), and execvp() all cause the new process to inherit the environment of the current process.

The second quoted paragraph implies that execvp will use the current process' environment variables. Combined with the first quoted paragraph, we deduce that execvp will use the value of the environment variable PATH from the environment of the current process. This means that Popen looks at the value of PATH as it was when Python launched (the Python that runs the Popen instantiation) and no amount of changing os.environ will help you fix that.

Also, on Windows with shell=False, Popen pays no attention to PATH at all, and will only look in relative to the current working directory.

What shell=True does

What happens if we pass shell=True to Popen? In that case, Popen simply calls the shell:

The shell argument (which defaults to False) specifies whether to use the shell as the program to execute.

That is to say, Popen does the equivalent of:

Popen(['/bin/sh', '-c', args[0], args[1], ...])

In other words, with shell=True Python will directly execute /bin/sh, without any searching (passing the argument executable to Popen can change this, and it seems that if it is a string without slashes, then it will be interpreted by Python as the shell program's name to search for in the value of PATH from the environment of the current process, i.e., as it searches for programs in the case shell=False described above).

In turn, /bin/sh (or our shell executable) will look for the program we want to run in its own environment's PATH, which is the same as the PATH of the Python (current process), as deduced from the code after the phrase "That is to say..." above (because that call has shell=False, so it is the case already discussed earlier). Therefore, the execvp-like behavior is what we get with both shell=True and shell=False, as long as env=None.

Passing env to Popen

So what happens if we pass env=dict(PATH=...) to Popen (thus defining an environment variable PATH in the environment of the program that will be run by Popen)?

In this case, the new environment is used to search for the program to execute. Quoting the documentation of Popen:

If env is not None, it must be a mapping that defines the environment variables for the new process; these are used instead of the default behavior of inheriting the current process’ environment.

Combined with the above observations, and from experiments using Popen, this means that Popen in this case behaves like the function os.execvpe. If shell=False, Python searches for the given program in the newly defined PATH. As already discussed above for shell=True, in that case the program is either /bin/sh, or, if a program name is given with the argument executable, then this alternative (shell) program is searched for in the newly defined PATH.

In addition, if shell=True, then inside the shell the search path that the shell will use to find the program given in args is the value of PATH passed to Popen via env.

So with env != None, Popen searches in the value of the key PATH of env (if a key PATH is present in env).

Propagating environment variables other than PATH as arguments

There is a caveat about environment variables other than PATH: if the values of those variables are needed in the command (e.g., as command-line arguments to the program being run), then even if these are present in the env given to Popen, they will not get interpreted without shell=True.
This is easily avoided without changing shell=True: insert those value directly in the list argument args that is given to Popen. (Also, if these values come from Python's own environment, the method os.environ.get can be used to get their values).

Using /usr/bin/env

If you JUST need path evaluation and don't really want to run your command line through a shell, and are on UNIX, I advise using env instead of shell=True, as in

path = '/dir1:/dir2'
subprocess.Popen(['/usr/bin/env', '-P', path, 'progtorun', other, args], ...)

This lets you pass a different PATH to the env process (using the option -P), which will use it to find the program. It also avoids issues with shell metacharacters and potential security issues with passing arguments through the shell. Obviously, on Windows (pretty much the only platform without a /usr/bin/env) you will need to do something different.

About shell=True

Quoting the Popen documentation:

If shell is True, it is recommended to pass args as a string rather than as a sequence.

Note: Read the Security Considerations section before using shell=True.

Unexpected observations

The following behavior was observed:

• This call raises FileNotFoundError, as expected:

  subprocess.call(['sh'], shell=False, env=dict(PATH=''))
  

• This call finds sh, which is unexpected:

  subprocess.call(['sh'], shell=False, env=dict(FOO=''))
  

  Typing echo $PATH inside the shell that this opens reveals that the PATH value is not empty, and also different from the value of PATH in the environment of Python. So it seems that PATH was indeed not inherited from Python (as expected in the presence of env != None), but still, it the PATH is nonempty. Unknown why this is the case.

• This call raises FileNotFoundError, as expected:

  subprocess.call(['tree'], shell=False, env=dict(FOO=''))
  

• This finds tree, as expected:

  subprocess.call(['tree'], shell=False, env=None)
  

---

Related Topics