Does "Argument List Too Long" Restriction Apply to Shell Builtins

Does argument list too long restriction apply to shell builtins?

In bash, the OS-enforced limitation on command-line length which causes the error argument list too long is not applied to shell builtins.

This error is triggered when the execve() syscall returns the error code E2BIG. There is no execve() call involved when invoking a builtin, so the error cannot take place.

Thus, both of your proposed operations are safe: cmd <<< "$string" writes $string to a temporary file, which does not require that it be passed as an argv element (or an environment variable, which is stored in the same pool of reserved space); and printf '%s\n' "$cmd" takes place internal to the shell unless the shell's configuration has been modified, as with enable -n printf, to use an external printf implementation.

Why do I get /bin/sh: Argument list too long when passing quoted arguments?

TL;DR

A single argument must be shorter than MAX_ARG_STRLEN.

Analysis

According to this link:

This is exactly the "problem" identified by the OP. While the number of arguments allowed may be quite large (see getconf ARG_MAX), when you pass a quoted command to /bin/sh the shell interprets the quoted command as a single string. In the OP's example, it is this single string that exceeds the MAX_ARG_STRLEN limit, not the length of the expanded argument list.

Implementation Specific

Argument limits are implementation specific. However, this Linux Journal article suggests several ways to work around them, including increasing system limits. This may not be directly applicable to the OP, but it nonetheless useful in the general case.

Do Something Else

The OP's issue isn't actually a real problem. The question is imposing an arbitrary constraint that doesn't solve a real-world problem.

You can work around this easily enough by using loops. For example, with Bash 4:

for i in {1..100000}; do /bin/sh -c "/bin/true $i"; done

works just fine. It will certainly be slow, since you're spawning a process on each pass through the loop, but it certainly gets around the command-line limit you're experiencing.

Describe Your Real Problem

If a loop doesn't resolve your issue, please update the question to describe the problem you're actually trying to solve using really long argument lists. Exploring arbitrary line-length limits is an academic exercise, and not on-topic for Stack Overflow.

Argument list too long error when I run any command in Linux shell

Thank you all. I use set, which is suggested by chepner, and see what happened. The problem is caused by my environment PATH. PATH is too long. After I clear PATH and set it again. It works.
Thank you

Argument list too long error for rm, cp, mv commands

The reason this occurs is because bash actually expands the asterisk to every matching file, producing a very long command line.

Try this:

find . -name "*.pdf" -print0 | xargs -0 rm

Warning: this is a recursive search and will find (and delete) files in subdirectories as well. Tack on -f to the rm command only if you are sure you don't want confirmation.

You can do the following to make the command non-recursive:

find . -maxdepth 1 -name "*.pdf" -print0 | xargs -0 rm

Another option is to use find's -delete flag:

find . -name "*.pdf" -delete

Error when transferring huge files: Argument list too long

Your command is not correct try with below. It will work for you:-

find ./ -name  "sm20180416*" | xargs -I {} mv -f {} /ora_arch/ssmfep_backup/

Circumvent Argument list too long in script (for loop)

Argument list too long workaroud

Argument list length is something limited by your config.

getconf ARG_MAX
2097152

But after discuss around differences between bash specifics and system (os) limitations (see comments from that other guy), this question seem wrong:

Regarding discuss on comments, OP tried something like:

ls "/simple path"/image*.{jpg,png} | wc -l
bash: /bin/ls: Argument list too long

This happen because of OS limitation, not bash!!

But tested with OP code, this work finely

for file in ./"simple path"/image*.{jpg,png} ;do echo -n a;done | wc -c
70980

Like:

 printf "%c" ./"simple path"/image*.{jpg,png} | wc -c

Reduce line length by reducing fixed part:

First step: you could reduce argument length by:

cd "/drive1/"
ls images*.{jpg,png} | wc -l

But when number of file will grow, you'll be buggy again...

More general workaround:

find "/drive1/" -type f \( -name '*.jpg' -o -name '*.png' \) -exec myscript {} +

If you want this to NOT be recursive, you may add -maxdepth as 1st option:

find "/drive1/" -maxdepth 1 -type f \( -name '*.jpg' -o -name '*.png' \) \
    -exec myscript {} +

There, myscript will by run with filenames as arguments. The command line for myscript is built up until it reaches a system-defined limit.

myscript /drive1/file1.jpg '/drive1/File Name2.png' /drive1/...

From man find:

   -exec command {} +
         This  variant  of the -exec action runs the specified command on
         the selected files, but the command line is built  by  appending
         each  selected file name at the end; the total number of invoca‐
         tions of the command will  be  much  less  than  the  number  of
         matched  files.   The command line is built in much the same way
         that xargs builds its command lines.  Only one instance of  `{}'

Inscript sample

You could create your script like

#!/bin/bash

target=( "/drive1" "/Drive 2/Pictures" )

[ "$1" = "--run" ] && exec find "${target[@]}" -type f \( -name '*.jpg' -o \
                         -name '*.png' \) -exec $0 {} +

for file ;do
    echo Process "$file"
done

Then you have to run this with --run as argument.

work with any number of files! (Recursively! See maxdepth option)
permit many target
permit spaces and special characters in file and directrories names

you could run same script directly on files, without --run:

 ./myscript hello world 'hello world'
 Process hello
 Process world
 Process hello world

Using pure bash

Using arrays, you could do things like:

allfiles=( "/drive 1"/images*.{jpg,png} )
[ -f "$allfiles" ] || { echo No file found.; exit ;}

echo Number of files: ${#allfiles[@]}

for file in "${allfiles[@]}";do
    echo Process "$file"
done

Argument list too long while slurping JSON files

Built-in commands are immune to that limitation, and printf is one of them. In conjunction with xargs, it would help a lot to achieve this.

printf '%s\0' *.json | xargs -0 cat -- | jq -s .

Getting argument list too long error

Try this

$ find /file/collection/*/logs/ -name "*.log" -type f -maxdepth 1 | xargs grep hello

Error: grep: Argument list too long

Use find

find /home/*/public_html -type f -exec grep -l 'pattern' {} +

The + modifier makes it group the filenames in manageable chunks.

However, you can do it with grep -r. The arguments to this should be the directory names, not filenames.

grep -rl 'pattern' /home/*/public_html

This will just have 500+ arguments, not thousands of filenames.

How to get around the Linux Too Many Arguments limit

edit:

I was finally able to pass <= 256 KB as a single command line argument (see edit (4) in the bottom). However, please read carefully how I did it and decide for yourself if this is a way you want to go. At least you should be able to understand why you are 'stuck' otherwise from what I found out.

With the coupling of ARG_MAX to ulim -s / 4 came the introduction of MAX_ARG_STRLEN as max. length of an argument:

/*
 *  linux/fs/exec.c
 *
 *  Copyright (C) 1991, 1992  Linus Torvalds
 */

#ifdef CONFIG_MMU
/*
 * The nascent bprm->mm is not visible until exec_mmap() but it can
 * use a lot of memory, account these pages in current->mm temporary
 * for oom_badness()->get_mm_rss(). Once exec succeeds or fails, we
 * change the counter back via acct_arg_size(0).
 */

static bool valid_arg_len(struct linux_binprm *bprm, long len)
{
 return len <= MAX_ARG_STRLEN;
}

#else

static bool valid_arg_len(struct linux_binprm *bprm, long len)
{
  return len <= bprm->p;
}

#endif /* CONFIG_MMU */

static int copy_strings(int argc, struct user_arg_ptr argv,
      struct linux_binprm *bprm)
{

    str = get_user_arg_ptr(argv, argc);

    len = strnlen_user(str, MAX_ARG_STRLEN);
    if (!len)
      goto out;

    ret = -E2BIG;
    if (!valid_arg_len(bprm, len))
      goto out;

MAX_ARG_STRLEN is defined as 32 times the page size in linux/include/uapi/linux/binfmts.h:

/*
 * These are the maximum length and maximum number of strings passed to the
 * execve() system call.  MAX_ARG_STRLEN is essentially random but serves to
 * prevent the kernel from being unduly impacted by misaddressed pointers.
 * MAX_ARG_STRINGS is chosen to fit in a signed 32-bit integer.
 */
#define MAX_ARG_STRLEN (PAGE_SIZE * 32)
#define MAX_ARG_STRINGS 0x7FFFFFFF

The default page size is 4 KB so you cannot pass arguments longer than 128 KB.

I can't try it now but maybe switching to huge page mode (page size 4 MB) if possible on your system solves this problem.

For more detailed information and references see this answer to a similar question on Unix & Linux SE.

edits:

(1)
According to this answer one can change the page size of x86_64 Linux to 1 MB by enabling CONFIG_TRANSPARENT_HUGEPAGE and setting CONFIG_TRANSPARENT_HUGEPAGE_MADVISE to n in the kernel config.

(2)
After recompiling my kernel with the above configuration changes getconf PAGESIZE still returns 4096.
According to this answer CONFIG_HUGETLB_PAGE is also needed which I could pull in via CONFIG_HUGETLBFS. I am recompiling now and will test again.

(3)
I recompiled my kernel with CONFIG_HUGETLBFS enabled and now /proc/meminfo contains the corresponding HugePages_* entries mentioned in the corresponding section of the kernel documentation.
However, the page size according to getconf PAGESIZE is still unchanged. So while I should be able now to request huge pages via mmap calls, the kernel's default page size determining MAX_ARG_STRLEN is still fixed at 4 KB.

(4)
I modified linux/include/uapi/linux/binfmts.h to #define MAX_ARG_STRLEN (PAGE_SIZE * 64), recompiled my kernel and now your code produces:

227982
227982
239981
239981
252611
252611
265906
./testCL: line 11: ./foo: Argument list too long
279901
./testCL: line 11: ./foo: Argument list too long
294632
./testCL: line 11: ./foo: Argument list too long

So now the limit moved from 128 KB to 256 KB as expected.
I don't know about potential side effects though.
As far as I can tell, my system seems to run just fine.

Does "Argument List Too Long" Restriction Apply to Shell Builtins