Automate Scp with Multiple Files with Expect Script

Automate SCP with multiple files with expect script

I found what I wanted with much more googleing. Thankyou for your help, hope this helps others

http://www.linuxquestions.org/questions/linux-general-1/scp-with-wildcard-in-expect-834813/

#!/usr/bin/expect -f

spawn bash -c "scp /path/* root@IP:/tmp/"
expect {
  -re ".*es.*o.*" {
    exp_send "yes\r"
    exp_continue
  }
  -re ".*sword.*" {
    exp_send "Password\r"
  }
}
interact

scp in Expect script works for few files but not for many files

The default timeout for expect is 10 seconds so expect eof would wait for at most 10 seconds which may be not enough for many files as you mentioned.

To fix, you can set timeout -1 before expect eof or just expect -timeout -1 eof.

scp multiple arguments of bash script

The basic problem is that when the shell expands $user@server:/path/run"$@"*.lz4, it doesn't copy the $user:... and *.lz4 parts for each argument, it just kind of blindly adds the list of arguments -- including word breaks between arguments -- into the middle. So if the args are 1 and 2, it essentially expands to:

scp $user@server:/path/run"1" "2"*.lz4 ./ 

...so $user@server:/path/run"1" and "2"*.lz4 are separate arguments to scp, which isn't useful. What you can do is create an array based on the arguments, and then use that as the source list for scp. Something like this:

sources=()
for runNum in "$@"; do
    sources+=("$user@server:/path/run${runNum}*.lz4")
done
scp "${sources[@]}" ./

And then use a separate loop for the lz4 command:

for runNum in "$@"; do
    lz4 -mdv --rm run${runNum}*.lz4
done

EDIT: to avoid having to authenticate multiple times, you can open a master SSH connection and let all the scp transfers piggyback on that. Here's an example, based heavily on Felix Rabe's answer here:

# Create the array of source file patterns to fetch
sources=()
for runNum in "$@"; do
    sources+=("$user@server:/path/run${runNum}*.lz4")
done

# Open master SSH connection:
# (This is the only time you have to enter the password)
sshSocket=~/"$user@server"
ssh -M -f -N -S "$sshSocket" "$user@server"

# Actually copy the files:
scp -o ControlPath="$sshSocket" "${sources[@]}" ./

# Close master connection:
ssh -S "$sshSocket" -O exit "$user@server"

---

Related Topics