Process Dies, If It Is Run via Paramiko Ssh Session and with "&" in the End

Process dies, if it is run via paramiko ssh session and with & in the end

With & you make your remote command exit instantly. The remote sshd will therefore likely (depends on implementation, but openssh does) kill all processes that were started from your command invocation. In your case, you just spawn a new process nohup tcpdump which will instantly return due to & at the end. The channel.recv_exit_status() will only block until the exit code for the & ooperation is ready which instantly is. Your code then just terminates, terminating your ssh session which will make the remote sshd kill the spawned nohup tcpdump proc. That why you end up with no tcpdump process.

Here's what you can do:

Since exec_command is going to spawn a new thread for your command, you can just leave it open and proceed with other tasks. But make sure to empty buffers every now and then (for verbose remote commands) to prevent paramiko from stalling.

ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh.connect(host, username=login, password=password)
transport = ssh.get_transport()
channel_tcpdump = transport.open_session()
channel_tcpdump.get_pty()
channel_tcpdump.set_combine_stderr(True)

cmd = "tcpdump -i eth1  port 443 -w /tmp/dump20150317183305940107.pcap"        # command will never exit
channel_tcpdump.exec_command(cmd)  # will return instantly due to new thread being spawned.
# do something else
time.sleep(15)      # wait 15 seconds
_,stdout,_ = ssh.exec_command("pgrep tcpdump") # or explicitly pkill tcpdump
print stdout.read()     # other command, different shell
channel_tcpdump.close()     # close channel and let remote side terminate your proc.
time.sleep(10)

Paramiko and exec_command - killing remote process?

While not the most efficient method, this should work. After you CTRL+C; In the KeyboardInterrupt handler you could exec_command("killall -u %s tail" % uname) like so:

#!/usr/bin/env python2

import paramiko
import select

import time
ltime = time.time()

# Or use random:
# import random
# ltime = random.randint(0, 500)

uname = "victorhooi"
client = paramiko.SSHClient()
client.load_system_host_keys()
client.connect('someserver', username=uname, password='blahblah')
transport = client.get_transport()
channel = transport.open_session()

channel.exec_command("tail -%df /home/victorhooi/macbeth.txt" % ltime)
while True:
    try:
        rl, wl, xl = select.select([channel],[],[],0.0)
        if len(rl) > 0:
            # Must be stdout
            print channel.recv(1024)
    except KeyboardInterrupt:
        print("Caught control-C")
        channel.close()
        try:
            # open new socket and kill the proc..
            client.get_transport().open_session().exec_command("kill -9 `ps -fu %s | grep 'tail -%df /home/victorhooi/macbeth.txt' | grep -v grep | awk '{print $2}'`" % (uname, ltime))
        except:
            pass
    
        client.close()
        exit(0)

This would kill any open processes named tail. That may cause issues though if you have tails open that you dont want to close, if thats the case you could grep a ps, get the pid and kill -9 it.

First, set tail to read n lines from end of file before following. set n to a unique nuber like time.time(), since tail doesn't care if that number is larger then the number of lines in the file, the large number from time.time()shouldnt cause issues and will be unique. Then grep for that unique number in the ps:

   client.get_transport().open_session().exec_command("kill -9 `ps -fu %s | grep 'tail -%df /home/victorhooi/macbeth.txt' | grep -v grep | awk '{print $2}'`" % (uname, ltime))

What does & at the end of a linux command mean?

The & makes the command run in the background.

From man bash:

If a command is terminated by the control operator &, the shell
executes the command in the background in a subshell. The shell does
not wait for the command to finish, and
the return status is 0.

Long-running ssh commands in python paramiko module (and how to end them)

1) You can just close the client if you wish. The server on the other end will kill the tail process.

2) If you need to do this in a non-blocking way, you will have to use the channel object directly. You can then watch for both stdout and stderr with channel.recv_ready() and channel.recv_stderr_ready(), or use select.select.

paramiko: automatically terminate remotely started processes

When the SSH connection is closed it'll not kill the running command on remote host.

The easiest solution is:

ssh.exec_command('python /home/me/loop.py', get_pty=True)
# ... do something ...
ssh.close()

Then when the SSH connection is closed, the pty (on remote host) will also be closed and the kernel (on remote host) will send the SIGHUP signal to the remote command. By default SIGHUP will terminate the process so the remote command will be killed.

---

According to the APUE book:

SIGHUP is sent to the controlling process (session leader) associated
with a controlling terminal if a disconnect is detected by the terminal
interface.

What's the difference between nohup and ampersand

nohup catches the hangup signal (see man 7 signal) while the ampersand doesn't (except the shell is confgured that way or doesn't send SIGHUP at all).

Normally, when running a command using & and exiting the shell afterwards, the shell will terminate the sub-command with the hangup signal (kill -SIGHUP <pid>). This can be prevented using nohup, as it catches the signal and ignores it so that it never reaches the actual application.

In case you're using bash, you can use the command shopt | grep hupon to find out whether
your shell sends SIGHUP to its child processes or not. If it is off, processes won't be
terminated, as it seems to be the case for you. More information on how bash terminates
applications can be found here.

There are cases where nohup does not work, for example when the process you start reconnects
the SIGHUP signal, as it is the case here.

Shell script to capture Process ID and kill it if exist

Actually the easiest way to do that would be to pass kill arguments like below:

ps -ef | grep your_process_name | grep -v grep | awk '{print $2}' | xargs kill

Hope it helps.

Why doesn't tcpdump run in background?

I'm not sure what you're trying to accomplish by having the startup script itself continue to run, but here's an approach that I think accomplishes what you're trying to do, namely start tcpdump and have it continue to run immune to hangups via nohup. I've simplified things a bit for illustrative purposes - feel free to add any variables back as you see fit, such as the nohup.out output directory, TIMESTAMP, etc.

Script #1: tcpdump_start.sh

#!/bin/sh
rm -f nohup.out
nohup /usr/sbin/tcpdump -ni eth0 -s 65535 -w file_result.pcap &

# Write tcpdump's PID to a file
echo $! > /var/run/tcpdump.pid

Script #2: tcpdump_stop.sh

#!/bin/sh
if [ -f /var/run/tcpdump.pid ]
then
        kill `cat /var/run/tcpdump.pid`
        echo tcpdump `cat /var/run/tcpdump.pid` killed.
        rm -f /var/run/tcpdump.pid
else
        echo tcpdump not running.
fi

To start tcpdump, just run tcpdump_start.sh.

To stop the tcpdump instance started with tcpdump_start.sh, just run tcpdump_stop.sh.

The captured packets will be written to the file_result.pcap file, and yes, it's a pcap file, not a text file, so it helps to name it with the proper file extension. The tcpdump statistics will be written to the nohup.out file when tcpdump is terminated.

---

Related Topics