How to Use Scp with Xargs

Scp locate's output by Xargs

Typically, {} is a findism:

find ... -exec cmd {} \;

Where {} is the current file that find is working on.

You can get xargs to behave similar with:

locate labra | xargs -I{} echo {} more arguments

However, you'll quickly notice that it runs the commands multiple times instead of one call to scp.

So in the context of your example:

locate labra | xargs -I{} scp '{}' masi@11.11.11:~/Desktop/

Notice the single quotes around the {} as it'll be useful for paths with spaces in them.

How to Properly Use xargs With cp

xargs has a -I parameter, which essentially allows you to specify a substitution pattern and write a substitution pattern in a command (cp, in your case). Then, when the command is executed, all instances of that substitution pattern are replaced with the next line of input. So, your command string should look something like this:

head -n 150 ../indexes/index.txt | xargs -I {} cp {} ~/path/to/destination/

Without the -I, cp just sees the second line of input as the destination, which is not what's intended.

Update: So, a breakdown of what's happening:
As each line of input is read from head, since you've set up a pipe, the output of head goes to the pipe. When xargs gets a new line from the pipe, because you've specified ... -I {} cp {} ..., instead of trying to create one large argument string composed of each line of input and trying to pass that to cp, it executes cp using each line of input as the source operand.

To explain a little further, consider this scenario. We have a file called test.txt that contains the following:

some_file.txt
another_file.txt
a_third_file.txt

Now, suppose we want to write a command to copy all of those files to a given destination. If we simply write...

cat test.txt | xargs cp /path/to/destination

...then what cp effectively sees is this:

cp some_file.txt another_file.txt a_third_file.txt

In this form, it tries to treat a_third_file.txt as a directory and copy the previous two files to it (which is why in your case cp says the 150th path is not a directory). xargs -I {} fixes that:

cat test.txt | xargs -I {} cp {} ~/path/to/destination

Now, instead of cp only being executed once, cp gets executed three times:

cp some_file.txt ~/path/to/destination
cp another_file.txt ~/path/to/destination
cp a_third_file.txt ~/path/to/destination

xargs -a file to copy files to a folder

From the following link i figured it out:

https://superuser.com/questions/180251/copy-list-of-files

Heres the command:

xargs -a bakfiles.txt cp -t backups

scp multiple arguments of bash script

The basic problem is that when the shell expands $user@server:/path/run"$@"*.lz4, it doesn't copy the $user:... and *.lz4 parts for each argument, it just kind of blindly adds the list of arguments -- including word breaks between arguments -- into the middle. So if the args are 1 and 2, it essentially expands to:

scp $user@server:/path/run"1" "2"*.lz4 ./ 

...so $user@server:/path/run"1" and "2"*.lz4 are separate arguments to scp, which isn't useful. What you can do is create an array based on the arguments, and then use that as the source list for scp. Something like this:

sources=()
for runNum in "$@"; do
    sources+=("$user@server:/path/run${runNum}*.lz4")
done
scp "${sources[@]}" ./

And then use a separate loop for the lz4 command:

for runNum in "$@"; do
    lz4 -mdv --rm run${runNum}*.lz4
done

EDIT: to avoid having to authenticate multiple times, you can open a master SSH connection and let all the scp transfers piggyback on that. Here's an example, based heavily on Felix Rabe's answer here:

# Create the array of source file patterns to fetch
sources=()
for runNum in "$@"; do
    sources+=("$user@server:/path/run${runNum}*.lz4")
done

# Open master SSH connection:
# (This is the only time you have to enter the password)
sshSocket=~/"$user@server"
ssh -M -f -N -S "$sshSocket" "$user@server"

# Actually copy the files:
scp -o ControlPath="$sshSocket" "${sources[@]}" ./

# Close master connection:
ssh -S "$sshSocket" -O exit "$user@server"

Secure copying files from a remote server to local machine from a list in a text file

You get this error xargs: illegal option -- i because -i was deprecated. Use -I {} instead (you could also use a different replace string but {} is fine).

If the list is remote, the files are remote, you can do this to retrieve it locally and use it with xargs -I {}:

ssh user@server cat fileList.txt | xargs -I {} scp user@server:{} .

But this creates N+1 connections, and more importantly this copies all remote files (scattered in different directories you said) to the same local directory. Probably not what you want.

So, in order to recreate a similar hierarchy locally, let's say everything under /iscsi/archive/aat, you can:

• use cut -d/ to extract the part you want to be identical on both sides
• use a subshell to create the command that creates the target directory and copies the file there

Thus:

ssh user@server cat fileList.txt \
| cut -d/ -f4- \
| xargs -I {} sh -c 'mkdir -p $(dirname {}); scp user@server:/iscsi/archive/{} ./{}'

Should work, but that's starting to look messy, and you still have N+1 connections, so now rsync looks like a better option. If you have passwordless ssh connection, this should work:

rsync -a --files-from=<(ssh user@server cat fileList.txt) user@server:/ .

The leading / is stripped by rsync and in the end you'll get everything under ./iscsi/archive/....

You can also copy the files locally first, and then:

rsync -a --files-from=localCopyOfFileList.txt user@server:/ .

You can also manipulate that file to remove for example 2 levels:

rsync -a --files-from=localCopyOfFileList2.txt user@server:/iscsi/archive .

etc.

Scp multiple files created in last one day from server to local fails

find can find only local files. So run find on remote server, something along:

ssh username@server.xyz.com find /path-from-which-files-need-to-be-copied/ -type f -ctime -1 |
xargs -ILIST scp username@server.xyz.com:LIST /Users/abcUser/Documents/test/

Note that xargs by default parses \ and quotes in it's own way. The best way to pass the result of find is to use zero terminated stream:

ssh username@server.xyz.com find /path-from-which-files-need-to-be-copied/ -type f -ctime -1 -print0 |
xargs -0 -ILIST scp username@server.xyz.com:LIST /Users/abcUser/Documents/test/   

But xargs will invoke separate scp session for each file, which will be very slow. So optimize it by running a single scp for all files, I think you could do it something like this:

ssh username@server.xyz.com find /path-from-which-files-need-to-be-copied/ -type f -ctime -1 -printf 'username@server.xyz.com:%p\\0' |
xargs -0 sh -c 'scp "$@" "$0"' /Users/abcUser/Documents/test/

Scp the Three Newest Files Using Bash

perhaps the simplest solution, but it does not deal with spaces in filenames

scp `ls -t | head -3` user@server:.

using xargs has the advantage of dealing with spaces in file names, but will execute scp three times

ls -t | head -3 | xargs -i scp {} user@server:.

a loop based solution would look like this. We use while read here because the default delimiter for read is the newline character not the space character like the for loop

ls -t | head -3 | while read file ; do scp $file user@server ; done

saddly, the perfect solution, one which executes a single scp command while working nicely with white space, eludes me at the moment.

---

Related Topics