Xargs with Multiple Arguments

xargs with multiple arguments

None of the solutions given so far deals correctly with file names containing space. Some even fail if the file names contain ' or ". If your input files are generated by users, you should be prepared for surprising file names.

GNU Parallel deals nicely with these file names and gives you (at least) 3 different solutions. If your program takes 3 and only 3 arguments then this will work:

(echo a1.txt; echo b1.txt; echo c1.txt;
 echo a2.txt; echo b2.txt; echo c2.txt;) |
parallel -N 3 my-program --file={1} --file={2} --file={3}

Or:

(echo a1.txt; echo b1.txt; echo c1.txt;
 echo a2.txt; echo b2.txt; echo c2.txt;) |
parallel -X -N 3 my-program --file={}

If, however, your program takes as many arguments as will fit on the command line:

(echo a1.txt; echo b1.txt; echo c1.txt;
 echo d1.txt; echo e1.txt; echo f1.txt;) |
parallel -X my-program --file={}

Watch the intro video to learn more: http://www.youtube.com/watch?v=OpaiGYxkSuQ

xargs for multiple arguments, each with two inputs

Calling Your Script

By default xargs crams as many arguments into each command as possible. It does not matter whether those arguments are in the same line or in different lines.

You can limit the number of arguments per command using -n

xargs -n 2 yourWgetScript.sh < yourFile

Alternatively, if the file lists the arguments for each call in a single line, you can instruct xargs to execute a new command every line instead of every two arguments

xargs -L 1 yourWgetScript.sh < yourFile

If your file is of the form

cmd1-arg1 cmd1-arg2
cmd2-arg1 cmd2-arg2
cmd3-arg1 cmd3-arg2
...

and there are no trailing spaces, special symbols, and so on, then those two commands are equivalent. But keep in mind that xargs has its own escaping mechanisms. If some of the arguments contain spaces or \"' you have to escape them using \.

More Efficient Alternative

It might be faster to use the following script instead of xargs yourWgetScript since the latter repeatedly starts a shell and a wget process for each URL which can be very slow.

awk '{print "https://stuff/" $1 "/" $2 "/etc"}' yourFile | xargs wget

Get xargs to allow multiple arguments to command

xargs -n 3 cmd

Executes cmd, with a maximum of three arguments at a time.

For example:

xargs -n 3 echo < file.txt
# gives
1 2 3
a b c
4 5 6

xargs -n 2 echo < file.txt
# gives
1 2
3 a
b c
4 5
6

can xargs separate parameters?

For those who find this from a search, the accepted answer did not work for me.

echo "'param 1' 'param 2'" | xargs -n1 | xargs -I@ echo \[@\] \[@\]

produces:

[param 1] [param 1]
[param 2] [param 2]

which does not meet the requirements given by the original poster to have xargs read in multiple entities, separate them, and send them to a single command ("echo" in the OP) as separate parameters. Xargs is not designed for this sort of task!

The bash answer can work.

p=(`echo "param1 param2"`); echo [${p[0]}] [${p[1]}]

produces:

[param1] [param2]

but this solution does not work with more than one line.

A correct solution with bash for sending pairs of lines as arguments to a single command is:

(echo 'param 1'; echo 'param 2'; echo 'param 3'; echo 'param 4') | while read line1; read line2; do echo "[$line1] [$line2]"; done

produces:

[param 1] [param 2]
[param 3] [param 4]

The GNU Parallel answer does work, but GNU Parallel must be make'd and installed. (The version packaged with Ubuntu is not GNU Parallel.)

Xargs with command that has multiple parameters

You can use the -I option for xargs:

nova list | grep SHUTOFF | cut '-d|' -f3 | xargs -I '{}' bash -c 'nova start {}'

Alternatively you can loop over the results:

for i in $(nova list | grep SHUTOFF | cut '-d|' -f3); do nova start $i; done

xargs pass multiple arguments to perl subroutine?

I am not sure about the details of your design, so I take it that you need a Perl one-liner to use shell's variables that are seen in the scope in which it's called.

A perl -e'...' executes a Perl program given under ''. For any variables from the environment where this program runs -- a pipeline, or a shell script -- to be available to the program their values need be passed to it. Ways to do this with a one-liner are spelled out in this post, and here is a summary.

A Perl program receives arguments passed to it on the command-line in @ARGV array. So you can invoke it in a pipeline as

... | perl -e'($v1, $v2) = @ARGV; ...' "$0" "$1"

or as

... | xargs -l perl -e'($v1, $v2) = @ARGV; ...'

if xargs is indeed used to feed the Perl program its input. In the first example the variables are quoted to protect possible interesting characters in them (spaces, *, etc) from being interpreted by the shell that sets up and runs the perl program.

If input contains multiple lines to process and the one-liner uses -n or -p for it then unpack arguments in a BEGIN block

... | perl -ne'BEGIN { ($v1, $v2) = splice(@ARGV,0,2) }; ...'  "$0" "$1" ...

which runs at compile time, so before the loop over input lines provided by -n/-p. The arguments other than filenames are now removed from @ARGV, so to leave only the filenames there for -n/-p, in case input comes from files.

There is also a rudimentary mechanism for command-line switches in a one-liner, via the -s switch. Please see the link above for details; I'd recommend @ARGV over this.

Finally, your calling code could set up environment variables which are then available to the Perl progam in %ENV. However, that doesn't seem to be suitable to what you seem to want.

Also see this post for another example.

Running multiple commands with xargs

cat a.txt | xargs -d $'\n' sh -c 'for arg do command1 "$arg"; command2 "$arg"; ...; done' _

...or, without a Useless Use Of cat:

<a.txt xargs -d $'\n' sh -c 'for arg do command1 "$arg"; command2 "$arg"; ...; done' _

---

To explain some of the finer points:

• The use of "$arg" instead of % (and the absence of -I in the xargs command line) is for security reasons: Passing data on sh's command-line argument list instead of substituting it into code prevents content that data might contain (such as $(rm -rf ~), to take a particularly malicious example) from being executed as code.

• Similarly, the use of -d $'\n' is a GNU extension which causes xargs to treat each line of the input file as a separate data item. Either this or -0 (which expects NULs instead of newlines) is necessary to prevent xargs from trying to apply shell-like (but not quite shell-compatible) parsing to the stream it reads. (If you don't have GNU xargs, you can use tr '\n' '\0' <a.txt | xargs -0 ... to get line-oriented reading without -d).

• The _ is a placeholder for $0, such that other data values added by xargs become $1 and onward, which happens to be the default set of values a for loop iterates over.

Using Xargs max-procs with multiple arguments from a file

Your second script, test.sh, expects two arguments, but xargs is feeding it only one (one word, in this case the complete line). You can fix it by first converting commas , to newlines (with a simple sed script) and then passing two arguments (now two lines) per call to test.sh (with -n2):

sed s/,/\\n/g file2.txt | xargs --max-procs 10 -n2 sh test.sh

Note that xargs supports a custom delimiter via -d option, and you could use it in case each line in file2.txt were ending with , (but then you should probably strip a newline prefixed to each first field).

How to use xargs to replace 2 arguments

I think it's best to do this with a while loop:

find . -iname "*.svg" -print0 |
    while IFS= read -r -d '' file; do
        svgexport "$file" "${file%.svg}.jpg"
    done

---

Related Topics