Why Is the Subprocess.Popen Argument Length Limit Smaller Than What the Os Reports

Why is the subprocess.Popen argument length limit smaller than what the OS reports?

The maximum size for a single string argument is limited to 131072. It has nothing to do with python:

~$ /bin/echo "$(printf "%*s" 131071 "a")">/dev/null
~$ /bin/echo "$(printf "%*s" 131072 "a")">/dev/null
bash: /bin/echo: Argument list too long

It is actually the MAX_ARG_STRLEN that decides the max size for a single string:

And as additional limit since 2.6.23, one argument must not be longer than MAX_ARG_STRLEN (131072).
This might become relevant if you generate a long call like "sh -c 'generated with long arguments'".
(pointed out by Xan Lopez and Ralf Wildenhues)

See this discussion of ARG_MAX, under "Number of arguments and maximum length of one argument", and this question on unix.stackexchange.

You can see it in binfmts.h:

/*
 * These are the maximum length and maximum number of strings passed to the
 * execve() system call.  MAX_ARG_STRLEN is essentially random but serves to
 * prevent the kernel from being unduly impacted by misaddressed pointers.
 * MAX_ARG_STRINGS is chosen to fit in a signed 32-bit integer.
 */
#define MAX_ARG_STRLEN (PAGE_SIZE * 32)
#define MAX_ARG_STRINGS 0x7FFFFFFF

~$ echo $(( $(getconf PAGE_SIZE)*32 )) 
131072

You can pass multiple strings of length 131071:

subprocess.check_call(['echo', "a"*131071,"b"*131071], executable='/bin/bash',stdout=open("/dev/null","w"))

But a single string arg cannot be longer than 131071 bytes.

What is the subprocess.Popen max length of the args parameter?

If you're passing shell=False, then Cmd.exe does not come into play.

On windows, subprocess will use the CreateProcess function from Win32 API to create the new process. The documentation for this function states that the second argument (which is build by subprocess.list2cmdline) has a max length of 32,768 characters, including the Unicode terminating null character. If lpApplicationName is NULL, the module name portion of lpCommandLine is limited to MAX_PATH characters.

Given your example, I suggest providing a value for executable (args[0]) and using args for the first parameter. If my reading of the CreateProcess documentation and of the subprocess module source code is correct, this should solve your problem.

[edit: removed the args[1:] bit after getting my hands on a windows machine and testing]

Subprocess argument list to long

I solved this by using a temporary file on windows. For Linux the command could be executed as is.

Method to build the full command for the different plattforms:

import tempfile

temporary_file = 0
def make_full_command(base_command, files):
    command = list(base_command)

    if platform.system() == "Windows":
        global temporary_file
        temporary_file = tempfile.NamedTemporaryFile()
        posix_files = map((lambda f: f.replace(os.sep, '/')),files)
        temporary_file.write(str.encode(" ".join(posix_files)))
        temporary_file.flush()
        command.append("@" + temporary_file.name)
    else:
        command.extend(files)
    return command

Usage of the file as a global variable ensures it is cleaned up after the execution.

This way I didn't have to find the max command length for different OSes

Python subprocess with arguments

Try this:

subprocess.call(['python3', 'test.py', 'John', 'Henry'])

and change test.py to look something like this:

import sys

print(sys.argv[1])
print(sys.argv[2])

Is a HTTP Get request of size 269KB allowed?

Typical limit is 8KB, but it can vary (like, be even less).

Python Subprocess: Too Many Open Files

I guess the problem was due to the fact that I was processing an open file with subprocess:

cmd = "enerCHARMM.pl -par param=x,xtop=topology_modified.rtf,xpar=lipid27_modified.par,nobuildall -out vdwaals {0}".format(cmtup[1])
p = subprocess.Popen(cmd, stdout=subprocess.PIPE, shell=True)

Here the cmd variable contain the name of a file that has just been created but not closed. Then the subprocess.Popen calls a system command on that file. After doing this for many times, the program crashed with that error message.

So the message I learned from this is

Close the file you have created, then process it

live output from subprocess command

TLDR for Python 3:

import subprocess
import sys

with open("test.log", "wb") as f:
    process = subprocess.Popen(your_command, stdout=subprocess.PIPE)
    for c in iter(lambda: process.stdout.read(1), b""):
        sys.stdout.buffer.write(c)
        f.buffer.write(c)

---

You have two ways of doing this, either by creating an iterator from the read or readline functions and do:

import subprocess
import sys

# replace "w" with "wb" for Python 3
with open("test.log", "w") as f:
    process = subprocess.Popen(your_command, stdout=subprocess.PIPE)
    # replace "" with b'' for Python 3
    for c in iter(lambda: process.stdout.read(1), ""):
        sys.stdout.write(c)
        f.write(c)

or

import subprocess
import sys

# replace "w" with "wb" for Python 3
with open("test.log", "w") as f:
    process = subprocess.Popen(your_command, stdout=subprocess.PIPE)
    # replace "" with b"" for Python 3
    for line in iter(process.stdout.readline, ""):
        sys.stdout.write(line)
        f.write(line)

Or you can create a reader and a writer file. Pass the writer to the Popen and read from the reader

import io
import time
import subprocess
import sys

filename = "test.log"
with io.open(filename, "wb") as writer, io.open(filename, "rb", 1) as reader:
    process = subprocess.Popen(command, stdout=writer)
    while process.poll() is None:
        sys.stdout.write(reader.read())
        time.sleep(0.5)
    # Read the remaining
    sys.stdout.write(reader.read())

This way you will have the data written in the test.log as well as on the standard output.

The only advantage of the file approach is that your code doesn't block. So you can do whatever you want in the meantime and read whenever you want from the reader in a non-blocking way. When you use PIPE, read and readline functions will block until either one character is written to the pipe or a line is written to the pipe respectively.

POSIX limits the number of character acceptable as user input to 4096, how to increase it?

As Mali said: stty -icanon in terminal before launching the program... solves the problem..

POSIX systems support two basic modes of input: canonical and noncanonical (or raw mode).

In canonical input processing mode, terminal input is processed in lines terminated by newline ('\n'), EOF, or EOL characters. The operating system provides input editing facilities: the ERASE and KILL characters are interpreted specially to perform editing operations within the current line of text.

In noncanonical input processing mode (raw mode), characters are not grouped into lines, and ERASE and KILL processing is not performed. The granularity with which bytes are read in raw mode is controlled by the MIN and TIME settings (see man termios the VMIN and VTIME value. VMIN specifies the minimum number of bytes before a read returns. VTIME specifies the number of tenths of a second to wait for data to arrive.

More bascally, in canonical mode, input are accumulated in a buffer (4096) until a \n occured. In raw mode, input is flushed when VMIN or VTIME occured

Thanks Mali..

---

Related Topics