How to Diff Directories Over Ssh

How to diff directories over ssh

If you needn't diff the detail in file, just get the difference of dir/file name, then try this:

(Note: need set "SSH login without password" , for detail , review this URL: http://www.linuxproblem.org/art_9.html)

diff <(ssh admin@10.0.0.10 ls -R /home/admin) <(ls -R /home/admin)

Diff between two remote folders through SSH

Try this command:

diff -B <(sshpass -p 'pa$$word1' ssh ubuntu1@images.server1.com "find /home/www/images/test_images -type f | sed 's/\/home\/www\/images\/test_images\///g'" | sort -n) <(sshpass -p 'pa$$word2' ssh ubuntu2@images.server2.com "find /var/www/site/images/test_images -type f | sed 's/\/var\/www\/site\/images\/test_images\///g'" | sort -n) | grep ">" | awk '{print $2}'

Explanation:

You can use diff -B <() <() for taking the diff between two streams. The command first uses sshpass to ssh into the two servers without having to enter your passwords interactively.

Each parameter for diff -B uses find command to recursively list all your images in the specified directory and uses sed to remove the root path of the files (because they are different for two servers - and to make it work for the diff command); and the sort command to sort them.

Since the output of the diff command returns either > or <, grep is used to filter out only the diffs from your Server 2. Last, awk prints out only the second column (removes the > column from the output).

NOTE: You need to install sshpass first. Use apt-get to install it as follows:

sudo apt-get install sshpass

You can extend this by piping other commands like rm. Hope this works for you.

Difficulties comparing local directory and remote directory using diff

diff -w /data/telescopedata/40in/shd/2016 <(ssh astro ls -l '/data/40in/shd/2016')

This is nonsense. You can't compare directory like this. The easiest way would be to use sshfs to mount your remote directory locally and compare them:

mkdir /tmp/cmp
sshfs astro:/data/40in/shd/2016 /mnt/cmp
diff -w /data/telescopedata/40in/shd/2016 /mnt/cmp/
fusermount -u /tmp/cmp

It should do the job just fine.

compare contents of two directories on remote server using unix

You can use rsync with the -n flag to find out if the files are in sync, without actually doing a sync.

For example, from server1:

rsync -n -avrc /abc/home/sample1/* server2:/abc/home/sample2/

This will print the names of all files (recursive, with the -r flag) that differ between server1:/abc/home/sample1/ and server2:/abc/home/sample2/

rsync used parameters explanation

-n, --dry-run - perform a trial run with no changes made

-a, --archive - archive mode; equals -rlptgoD (no -H,-A,-X)

-v, --verbose - increase verbosity

-r, --recursive - recurse into directories

-c, --checksum - skip based on checksum, not mod-time & size

compare multiple directories on remote hosts

Yes, GNU diff has an option --from-file that allows the comparison of one reference file or directory to many others.

diff -r --from-file=ref-dir dir1 dir2 ... dirN

Note that it will only compare ref-dir to dir1, ..., dirN; it won't compare dir1 to dir2, ..., dirN.

As for your remote directories, since you have ssh access to the machines, you can mount them locally with sshfs in order to execute diff over them.

Beyond compare remote with local file

One neat trick that I use in such cases is to use sshfs to make the remote directory appear as a local one to beyond compare:

sshfs user@host:/opt/data /tmp/transfer

Then you can use any diffing tool to compare your local data with the remote one via diff -Nur /local/data/dir /tmp/transfer or with beyond compare as usual and you can copy files back and forth as needed.

Note, you might need to install sshfs via your package manager, e.g. on Debian/Ubuntu via apt-get install sshfs

Possible options:

• -o PubkeyAuthentication=no: If you need password authentication, but ssh first tries too many private key authenticaitions, not needed if you use private key auth for this server or no private key auth at all
• -o sshfs_debug: Print out more information if authentication/connection fails
• -o idmap=user: Use the idmap=user option to translate the UID of the connecting user to the remote user
• -o gid=1000: uid, gid - set reported ownership of files to given values; uid is the numeric user ID of your user, gid is the numeric group ID of your user.
• -o nomap=error: Fail if user mapping cannot be done

See https://wiki.archlinux.org/index.php/SSHFS for more details.

Unix command to get list of directories over remote host

Try:

ssh "$host" '
find /home/abc/mydata -mindepth 1 -maxdepth 1 -type d |
while read -r d; do
    [ "$(find "$d" -mmin -1 -type f -print -quit 2>/dev/null)" ] &&
    ls -ldi "$d"
done
'

Notes:

• With mindepth/maxdepth, the first find will look only at level 1.
• A separate find will look inside each subfolder for regular files (-type f) recently modified. If other recently modified file types (subdirectories/sockets/symlinks) should trigger the output, remove -type f.
• The -t parameter to ssh is only necessary if you want colorized ls output.
• The output of ls -ldi ... is slightly different from that of find ... -ls, but I'm guessing it will do.
• The return value of the second level find is too vague to use here. Instead, we simply test if its output is nonempty.

---

Related Topics