Popen with Conflicting Executable/Path

Popen with conflicting executable/path

Search for a program is not trivial. I'd specify the full path to the convert.exe executable explicitly instead.

subprocess uses CreateProcess on Windows that looks in system32 directory even before any other directory in %PATH%:

... If the file name does not contain an extension, .exe is appended.
Therefore, if the file name extension is .com, this parameter must
include the .com extension. If the file name ends in a period (.) with
no extension, or if the file name contains a path, .exe is not
appended. If the file name does not contain a directory path, the
system searches for the executable file in the following sequence:

1. The directory from which the application loaded.
2. The current directory for the parent process.
3. The 32-bit Windows system directory. Use the GetSystemDirectory function to get the path of this directory.
4. The 16-bit Windows system directory. There is no function that obtains the path of this directory, but it is searched. The name of this directory is System.
5. The Windows directory. Use the GetWindowsDirectory function to get the path of this directory.
6. The directories that are listed in the PATH environment variable. Note that this function does not search the per-application path specified by the App Paths registry key. To include this per-application path in the search sequence, use the ShellExecute function.

Therefore convert is equivalent to convert.exe in this case. It first looks in a directory that contains sys.executable e.g., C:\Python27. Then in the current directory: where you started the Python script from. Then in system32 where it finds convert.exe (filesystem utility, not imagemagick).

You could try to remove system32 directory from os.environ['PATH'] it may(?) suppress checking it: Popen(cmd, env=no_system32_environ) but it is fragile (worse than the explicit path).

There is a related issue on Python bug tracker: "Subprocess picks the wrong executable on Windows."

---

cmd.exe (the shell) uses different algorithm. See How does Windows locate files input in the shell?

If you set shell=True then the search sequence for convert program:

1. convert is not an internal shell command
2. there is no explicit path, so the search continues
3. search the current directory
4. search each directory specified by the PATH environment variable, in the order listed

%PATHEXT% defines which file extensions are checked and in what order e.g., convert.com, convert.exe, convert.bat, convert.cmd if %PATHEXT% is .com;.exe;.bat;.cmd.

Subprocess.call or Subprocess.Popen cannot use executables that are in PATH (Linux/Windows)

Ok here is how I got it to work.

env = os.environ
proc = subprocess.Popen(args, env=env)

Python-Subprocess-Popen inconsistent behavior in a multi-threaded environment

-11 as a return code might mean that C program is not fine e.g., you are starting too many subprocesses and it causes SIGSERV in the C executable. You can limit number of concurrent subprocesses using multiprocessing.ThreadPool, concurrent.futures.ThreadPoolExecutor, threading + Queue -based solutions:

#!/usr/bin/env python
from multiprocessing.dummy import Pool # uses threads
from subprocess import Popen, PIPE

def get_url(url):
    p = Popen(["executable", url], stdout=PIPE, stderr=PIPE, close_fds=True)
    output, error = p.communicate()
    return url, output, error, p.returncode

pool = Pool(20) # limit number of concurrent subprocesses
for url, output, error, returncode in pool.imap_unordered(get_url, urls):
    print("%s %r %r %d" % (url, output, error, returncode))

Make sure the executable can be run in parallel e.g., it doesn't use some shared resource. To test, you could run in a shell:

$ executable url1 & executable url2

---

Could you please explain more about "you are starting too many subprocesses and it causes SIGSERV in the C executable" and possibly solution to avoid that..

Possible problem:

• "too many processes"
• -> "not enough memory in the system or some other resource"
• -> "trigger the bug in the C code that otherwise is hidden or rare"
• -> "illegal memory access"
• -> SIGSERV

The suggested above solution is:

• "limit number of concurrent processes"
• -> "enough memory or other resources in the system"
• -> "bug is hidden or rare"
• -> no SIGSERV

Understand what is SIGSEGV run time error in c++? In short, your program is killed with that signal if it tries to access a memory that it is not supposed to. Here's an example of such program:

/* try to fail with SIGSERV sometimes */
#include <stdio.h>
#include <stdlib.h>
#include <time.h>

int main(void) {
  char *null_pointer = NULL;

  srand((unsigned)time(NULL));

  if (rand() < RAND_MAX/2) /* simulate some concurrent condition 
                              e.g., memory pressure */
    fprintf(stderr, "%c\n", *null_pointer); /* dereference null pointer */

  return 0;
}

If you run it with the above Python script then it would return -11 occasionally.

Also p.returncode is not sufficient for debugging purpose..Is there any other option to get more DEBUG info to get to the root cause?

I won't exclude the Python side completely but It is most likely that the problem is the C program. You could use gdb to get a backtrace to see where in a callstack the error comes from.

Subprocess.Popen behaves differently in interpreter, executable scripts

I wrote a little test script to test the subprocess module with.

#!/bin/bash

echo echo to stderr 1>&2
echo echo to stdout

Then I wrote a small Python script that calls it:

#!/usr/bin/python

import subprocess

command = ('./joe.sh',)
task = subprocess.Popen(command, stdout=subprocess.PIPE,
                                 stderr=subprocess.PIPE)
stdout, stderr = task.communicate()
print 'stdout == %r\nstderr == %r' % (stdout, stderr)

The output of running it looks just like this:

$ python joe.py 
stdout == 'echo to stdout\n'
stderr == 'echo to stderr\n'

The output of running that same sequence in ipython is the same.

So the subprocess module is behaving in the manner you expect, and not how it's behaving for you in your question. I think something other than the subprocess module must be at fault here because what you're doing works for me.

I'm running Python 2.7, so another possibility is that maybe there is some kind of weird bug in older versions of the subprocess module.

Capture output of Popen when shell=True

Add stderr=subprocess.PIPE to Popen. Otherwise, the standard error will continue to go to whatever file the subprocess inherits from your script.

Python subprocess communication

Try:

check_call('gdal2tiles -p raster -z 0-1 new.jpg abc', shell=True)

shell=True changes how the executable is searched on Windows.

Or if gdal2tiles works only in the environment created by OSGeo4W.bat:

shell = Popen(r'C:\OSGeo4W64\OSGeo4W.bat', stdin=subprocess.PIPE)
shell.communicate('gdal2tiles -p raster -z 0-1 new.jpg abc')
# you don't need shell.wait() here

Notice: r"" literal. It is necessary to avoid escaping the backslashes in the path.

---

Related Topics