Save The Result of Ls Command in The Remote Sftp Server on Local Machine

Save the result of ls command in the remote SFTP server on local machine

Here's how I would do it, with a more "traditional" approach that involves opening a file for writing the desired output (I like @kastelian 's log_file idea, though):

#!/usr/bin/expect

spawn sftp myuser@myftp.mydomain.com
expect "password:"
send "mypassword\n";
expect "sftp>"

set file [open /tmp/ls-output w]     ;# open for writing and set file identifier

expect ".*"     ;# match anything in buffer so as to clear it

send  "ls\r"

expect {
    "sftp>" {
        puts $file $expect_out(buffer)  ;# save to file buffer contents since last match (clear) until this match (sftp> prompt)
    }
    timeout {     ;# somewhat elegant way to die if something goes wrong
        puts $file "Error: expect block timed out"
    }
}

close $file

interact

The resulting file will hold the same two extra lines as in the log_file proposed solution: the ls command at the top, and the sftp> prompt at the bottom, but you should be able to deal with those as you like.

I have tested this and it works.

Let me know if it helped!

How do I pipe the output of an LS on remote server to the local filesystem via SFTP?

If you're willing to wait the 20 minutes for the data to scroll across your screen you can capture all the output using "script".

Call 'script' before you start your ssh or sftp session and it will capture all terminal output to your local disk. Type 'exit' to finish the capture.

NAME
     script -- make typescript of terminal session

SYNOPSIS
     script [-akq] [-t time] [file [command ...]]

DESCRIPTION
     The script utility makes a typescript of everything printed on your ter-
     minal.  It is useful for students who need a hardcopy record of an inter-
     active session as proof of an assignment, as the typescript file can be
     printed out later with lpr(1).

     If the argument file is given, script saves all dialogue in file.  If no
     file name is given, the typescript is saved in the file typescript.

     If the argument command is given, script will run the specified command
     with an optional argument vector instead of an interactive shell.

     The following options are available:

     -a      Append the output to file or typescript, retaining the prior con-
             tents.

     -k      Log keys sent to program as well as output.

     -q      Run in quiet mode, omit the start and stop status messages.

     -t time
             Specify time interval between flushing script output file.  A
             value of 0 causes script to flush for every character I/O event.
             The default interval is 30 seconds.

     The script ends when the forked shell (or command) exits (a control-D to
     exit the Bourne shell (sh(1)), and exit, logout or control-D (if
     ignoreeof is not set) for the C-shell, csh(1)).

     Certain interactive commands, such as vi(1), create garbage in the type-
     script file.  The script utility works best with commands that do not
     manipulate the screen.  The results are meant to emulate a hardcopy ter-
     minal, not an addressable one.

ENVIRONMENT
     The following environment variable is utilized by script:

     SHELL  If the variable SHELL exists, the shell forked by script will be
            that shell.  If SHELL is not set, the Bourne shell is assumed.
            (Most shells set this variable automatically).

SEE ALSO
     csh(1) (for the history mechanism).

HISTORY
     The script command appeared in 3.0BSD.

BUGS
     The script utility places everything in the log file, including linefeeds
     and backspaces.  This is not what the naive user expects.

     It is not possible to specify a command without also naming the script
     file because of argument parsing compatibility issues.

     When running in -k mode, echo cancelling is far from ideal.  The slave
     terminal mode is checked for ECHO mode to check when to avoid manual echo
     logging.  This does not work when in a raw mode where the program being
     run is doing manual echo.

Get latest file from sftp to local machine

You have your ls options wrong, try:

file=$(ssh username@servername 'ls -1tr /server/path' | tail -n 1)

You are using the -l (lowercase L for long listing) option instead of using -1 (number one) option (list one file per line) This causes your ls command to include the file permissions, file ownership, etc... which when used as input to scp causes the command to fail.

Also, if you are providing a filename to scp, you don't need the -r (recursive) option. The following should be all you need for your scp command. (don't forget to double-quote "$file" to prevent word-splitting if there are spaces in the filename)

scp username@servername:/path/path/"$file" /my/home/directory

List specific files on a remote windows machine from a linux machine using sftp

Will this insertion of grep in the pipeline do the trick?

echo "ls -l ${TARGET_ICT_DIR}\nquit\n" |
    sftp ${SFTP_LOGIN} |
    grep -E ' (anchor\.jar|rename\.txt|zipper.dat)\r?$' |
    tee /tmp/ver_ict_rel_dest_${L_CURR_PID}.txt

Do not know if the check for \r is necessary -- but... Windows... Also, in comment @ghoti noted that \r might not translate well to all versions of grep and shell. You could try something like the following if you run into such:

echo "ls -l ${TARGET_ICT_DIR}\nquit\n" |
    sftp ${SFTP_LOGIN} |
    grep -E " (anchor\.jar|rename\.txt|zipper.dat)$(printf '\r')?$" |
    tee /tmp/ver_ict_rel_dest_${L_CURR_PID}.txt

(Both variants of the ls -l | grep -E ... part was tested with dash and bash on Linux.)

SFTP - capture file list not working as expected

Here's a rough version of what you're looking for. It will require you to keep a list from the last sync around.
(How you authenticate for this is a different topic, but instead of saving the password in a script I'd recommend using ssh keys).

#!/bin/bash

lastlist=/tmp/sftp_list.old
newlist=/tmp/sftp_list.new
localpath=/home/foo/downloads
remotepattern='/home/bar/*.zip'
remotehost='account@0.0.0.0'
notifysubject="New remote files"
notifyaddress="foo@example.com bar@example.com"

cd "$localpath"
if echo "ls -1 $remotepattern" | sftp "$remotehost" >"$newlist"; then
    filelist=$(comm -13 "$lastlist" "$newlist")
    [ $(echo "$filelist" | wc -l) -gt 0 ] && {
        {
            echo "New remote files:"
            echo "$filelist"
        } | mail -s "$notifysubject" "$notifyaddress"

        # download new files
        echo "$filelist" | sed 's/^/get /' | sftp "$remotehost"

        rm -f "$lastlist"
        mv "$newlist" "$lastlist"
    }
else
    echo "Error connecting to the remote host"
fi

This downloads only new files, but it will fail in subtle ways if any of the filenames contain \n (which is very rare). A simple fix is to re-download all matching files instead of selecting by name: echo "get $remotepattern" | sftp "$remotehost"

Spring Integration - SFTP Polling - Inbound Adapter to fetch file name without copying to local directory

Use the outbound gateway with the list (LS) command.

https://docs.spring.io/spring-integration/docs/current/reference/html/sftp.html#using-the-ls-command

EDIT

@Bean
IntegrationFlow flow(DefaultSftpSessionFactory sf) {
    return IntegrationFlows.fromSupplier(() -> "dir", e -> e.poller(Pollers.fixedDelay(5000)))
            .handle(Sftp.outboundGateway(sf, Command.LS, "payload")
                    .options(Option.NAME_ONLY))
            .split()
            .log()
            .get();
}

Retrieving data from an SFTP server using JSch

Concerning your point 1, I suspect that the default directory after connecting is not what you expect. Try using an absolute remote path. Does sftpChannel.pwd() return the directory the file remote-data.txt is in on the remote machine ?

Concerning your point 2, looking at http://grepcode.com/file/repo1.maven.org/maven2/com.jcraft/jsch/0.1.42/com/jcraft/jsch/ChannelSftp.java#290 one sees that there is the following method in ChannelSftp:

 public void put(String src, String dst)

which indeed has a source and destination file name argument.

I guess you had already a look the Jsch sftp example at http://www.jcraft.com/jsch/examples/Sftp.java ?

---

Related Topics