How to Get Around the Linux "Too Many Arguments" Limit

How to get around the Linux Too Many Arguments limit

edit:

I was finally able to pass <= 256 KB as a single command line argument (see edit (4) in the bottom). However, please read carefully how I did it and decide for yourself if this is a way you want to go. At least you should be able to understand why you are 'stuck' otherwise from what I found out.

With the coupling of ARG_MAX to ulim -s / 4 came the introduction of MAX_ARG_STRLEN as max. length of an argument:

/*
 *  linux/fs/exec.c
 *
 *  Copyright (C) 1991, 1992  Linus Torvalds
 */

#ifdef CONFIG_MMU
/*
 * The nascent bprm->mm is not visible until exec_mmap() but it can
 * use a lot of memory, account these pages in current->mm temporary
 * for oom_badness()->get_mm_rss(). Once exec succeeds or fails, we
 * change the counter back via acct_arg_size(0).
 */

static bool valid_arg_len(struct linux_binprm *bprm, long len)
{
 return len <= MAX_ARG_STRLEN;
}

#else

static bool valid_arg_len(struct linux_binprm *bprm, long len)
{
  return len <= bprm->p;
}

#endif /* CONFIG_MMU */

static int copy_strings(int argc, struct user_arg_ptr argv,
      struct linux_binprm *bprm)
{

    str = get_user_arg_ptr(argv, argc);

    len = strnlen_user(str, MAX_ARG_STRLEN);
    if (!len)
      goto out;

    ret = -E2BIG;
    if (!valid_arg_len(bprm, len))
      goto out;

MAX_ARG_STRLEN is defined as 32 times the page size in linux/include/uapi/linux/binfmts.h:

/*
 * These are the maximum length and maximum number of strings passed to the
 * execve() system call.  MAX_ARG_STRLEN is essentially random but serves to
 * prevent the kernel from being unduly impacted by misaddressed pointers.
 * MAX_ARG_STRINGS is chosen to fit in a signed 32-bit integer.
 */
#define MAX_ARG_STRLEN (PAGE_SIZE * 32)
#define MAX_ARG_STRINGS 0x7FFFFFFF

The default page size is 4 KB so you cannot pass arguments longer than 128 KB.

I can't try it now but maybe switching to huge page mode (page size 4 MB) if possible on your system solves this problem.

For more detailed information and references see this answer to a similar question on Unix & Linux SE.

edits:

(1)
According to this answer one can change the page size of x86_64 Linux to 1 MB by enabling CONFIG_TRANSPARENT_HUGEPAGE and setting CONFIG_TRANSPARENT_HUGEPAGE_MADVISE to n in the kernel config.

(2)
After recompiling my kernel with the above configuration changes getconf PAGESIZE still returns 4096.
According to this answer CONFIG_HUGETLB_PAGE is also needed which I could pull in via CONFIG_HUGETLBFS. I am recompiling now and will test again.

(3)
I recompiled my kernel with CONFIG_HUGETLBFS enabled and now /proc/meminfo contains the corresponding HugePages_* entries mentioned in the corresponding section of the kernel documentation.
However, the page size according to getconf PAGESIZE is still unchanged. So while I should be able now to request huge pages via mmap calls, the kernel's default page size determining MAX_ARG_STRLEN is still fixed at 4 KB.

(4)
I modified linux/include/uapi/linux/binfmts.h to #define MAX_ARG_STRLEN (PAGE_SIZE * 64), recompiled my kernel and now your code produces:

227982
227982
239981
239981
252611
252611
265906
./testCL: line 11: ./foo: Argument list too long
279901
./testCL: line 11: ./foo: Argument list too long
294632
./testCL: line 11: ./foo: Argument list too long

So now the limit moved from 128 KB to 256 KB as expected.
I don't know about potential side effects though.
As far as I can tell, my system seems to run just fine.

Argument list too long error for rm, cp, mv commands

The reason this occurs is because bash actually expands the asterisk to every matching file, producing a very long command line.

Try this:

find . -name "*.pdf" -print0 | xargs -0 rm

Warning: this is a recursive search and will find (and delete) files in subdirectories as well. Tack on -f to the rm command only if you are sure you don't want confirmation.

You can do the following to make the command non-recursive:

find . -maxdepth 1 -name "*.pdf" -print0 | xargs -0 rm

Another option is to use find's -delete flag:

find . -name "*.pdf" -delete

Circumvent Argument list too long in script (for loop)

Argument list too long workaroud

Argument list length is something limited by your config.

getconf ARG_MAX
2097152

But after discuss around differences between bash specifics and system (os) limitations (see comments from that other guy), this question seem wrong:

Regarding discuss on comments, OP tried something like:

ls "/simple path"/image*.{jpg,png} | wc -l
bash: /bin/ls: Argument list too long

This happen because of OS limitation, not bash!!

But tested with OP code, this work finely

for file in ./"simple path"/image*.{jpg,png} ;do echo -n a;done | wc -c
70980

Like:

 printf "%c" ./"simple path"/image*.{jpg,png} | wc -c

Reduce line length by reducing fixed part:

First step: you could reduce argument length by:

cd "/drive1/"
ls images*.{jpg,png} | wc -l

But when number of file will grow, you'll be buggy again...

More general workaround:

find "/drive1/" -type f \( -name '*.jpg' -o -name '*.png' \) -exec myscript {} +

If you want this to NOT be recursive, you may add -maxdepth as 1st option:

find "/drive1/" -maxdepth 1 -type f \( -name '*.jpg' -o -name '*.png' \) \
    -exec myscript {} +

There, myscript will by run with filenames as arguments. The command line for myscript is built up until it reaches a system-defined limit.

myscript /drive1/file1.jpg '/drive1/File Name2.png' /drive1/...

From man find:

   -exec command {} +
         This  variant  of the -exec action runs the specified command on
         the selected files, but the command line is built  by  appending
         each  selected file name at the end; the total number of invoca‐
         tions of the command will  be  much  less  than  the  number  of
         matched  files.   The command line is built in much the same way
         that xargs builds its command lines.  Only one instance of  `{}'

Inscript sample

You could create your script like

#!/bin/bash

target=( "/drive1" "/Drive 2/Pictures" )

[ "$1" = "--run" ] && exec find "${target[@]}" -type f \( -name '*.jpg' -o \
                         -name '*.png' \) -exec $0 {} +

for file ;do
    echo Process "$file"
done

Then you have to run this with --run as argument.

work with any number of files! (Recursively! See maxdepth option)
permit many target
permit spaces and special characters in file and directrories names

you could run same script directly on files, without --run:

 ./myscript hello world 'hello world'
 Process hello
 Process world
 Process hello world

Using pure bash

Using arrays, you could do things like:

allfiles=( "/drive 1"/images*.{jpg,png} )
[ -f "$allfiles" ] || { echo No file found.; exit ;}

echo Number of files: ${#allfiles[@]}

for file in "${allfiles[@]}";do
    echo Process "$file"
done

Does argument list too long restriction apply to shell builtins?

In bash, the OS-enforced limitation on command-line length which causes the error argument list too long is not applied to shell builtins.

This error is triggered when the execve() syscall returns the error code E2BIG. There is no execve() call involved when invoking a builtin, so the error cannot take place.

Thus, both of your proposed operations are safe: cmd <<< "$string" writes $string to a temporary file, which does not require that it be passed as an argv element (or an environment variable, which is stored in the same pool of reserved space); and printf '%s\n' "$cmd" takes place internal to the shell unless the shell's configuration has been modified, as with enable -n printf, to use an external printf implementation.

While loop in a shell script gives error : [: too many arguments. How to resolve this issue?

Here you go, this works for me:

#!/bin/bash

function read_num(){
    echo "Enter a lower limit"
    read lower_limit
    echo "Enter a upper limit"
    read uper_limit
    while [[ $lower_limit -lt 1 ]] || [[ $lower_limit -gt $upper_limit ]]
    do
    echo "Please enter again."
    read_num
    done
}

read_num

Reference: Bash scripting, multiple conditions in while loop

Perl command not executing because of too many arguments, how can I fix it to allow more arguments?

GNU:

find {folder_name} -maxdepth 1 -name '*.txt' -exec perl -i -pe's/\r//' {} +

More portable:

find {folder_name} -maxdepth 1 -name '*.txt' -print | xargs perl -i -pe's/\r//'

echo {folder_name}/*.txt | xargs perl -i -pe's/\r//'

find {folder_name} -maxdepth 1 -name '*.txt' -exec perl -i -pe's/\r//' {} \;

Notes

None of the solutions support a {folder_name} that start with -.
The first two portable solutions don't support file names with some special characters in them. (But it's unlikely that you'll find those characters in file names in the first place.)
The GNU solution and the first two portable solutions pass as many file names to perl as possible (minimizing the number of times perl is launched), while the third portable solution launches a perl process for every parameter.
echo is a shell builtin, so it's not subject to the system's limit on the number of parameters passed to utilities.

How can I grep while avoiding 'Too many arguments'

Run several instances of grep. Instead of

grep -i user@domain.com 1US* | awk '{...}' | xargs rm

(for i in 1US*; do grep -li user@domain "$i"; done) | xargs rm

Note the -l flag, since we only want the file name of the match. This will both speed up grep (terminate on first match) and makes your awk script unrequired. This could be improved by checking the return status of grep and calling rm, not using xargs (xargs is very fragile, IMO). I'll give you the better version if you ask.

Hope it helps.

How to handle more than 10 parameters in shell

Use curly braces to set them off:

echo "${10}"

Any positional parameter can be saved in a variable to document its use and make later statements more readable:

city_name=${10}

If fewer parameters are passed then the value at the later positions will be unset.

You can also iterate over the positional parameters like this:

for arg

for arg in "$@"

while (( $# > 0 ))    # or [ $# -gt 0 ]
do
    echo "$1"
    shift
done

How do you handle the Too many files problem when working in Bash?

In newer versions of findutils, find can do the work of xargs (including the glomming behavior, such that only as many grep processes as needed are used):

find ../path -exec grep foo '{}' +

The use of + rather than ; as the last argument triggers this behavior.

How to Get Around the Linux "Too Many Arguments" Limit