Running Process of Remote Ssh Server in the Background Using Python Paramiko

Start a background process on remote machine using python

Login with ssh and start any job process with job&. Login with ssh in a different window and do ps to check for your job: you should see it running. Now logout of your first ssh and check again for your job process. You will notice that it is now gone. This happens because jobs are attached to a terminal by default and are sent a SIGHUP when the terminal is closed.

Now repeat the process with running nohup job& or disown job&. These both prevent the SIGHUP from killing the job process.

To fix your code you can use either of the following:

import fabric.api
fabric.api.execute(run_burnP6_bg, hosts=[remote_machine])

def run_burnP6_bg():
    fabric.api.run("nohup burnP6 &")

or with subprocess

import subprocess
cmd = 'ssh -f xyz@{1} '.format(ip_addr) + "'nohup burnP6 &'"
subprocess.call(cmd,shell=True)

these should prevent your job from dying when the ssh session ends.

Running remote command in background using paramiko

If you execute a command in the background, the session ends immediately, not waiting for the command to complete.

Try this command:

ssh -t user_name@host_name "ls && sleep 15 && ls &"

It's an equivalent of your Python code. And you will see that it produces no output either. It won't even wait the 15 seconds, it closes immediately. Same as your Python code.

$ ssh -t user_name@host_name "ls && sleep 15 && ls &"
Connection to host_name closed.
$

---

Anyway, I do not see a point of executing a command in the background, if you then wait for the command to finish.

Paramiko and exec_command - killing remote process?

While not the most efficient method, this should work. After you CTRL+C; In the KeyboardInterrupt handler you could exec_command("killall -u %s tail" % uname) like so:

#!/usr/bin/env python2

import paramiko
import select

import time
ltime = time.time()

# Or use random:
# import random
# ltime = random.randint(0, 500)

uname = "victorhooi"
client = paramiko.SSHClient()
client.load_system_host_keys()
client.connect('someserver', username=uname, password='blahblah')
transport = client.get_transport()
channel = transport.open_session()

channel.exec_command("tail -%df /home/victorhooi/macbeth.txt" % ltime)
while True:
    try:
        rl, wl, xl = select.select([channel],[],[],0.0)
        if len(rl) > 0:
            # Must be stdout
            print channel.recv(1024)
    except KeyboardInterrupt:
        print("Caught control-C")
        channel.close()
        try:
            # open new socket and kill the proc..
            client.get_transport().open_session().exec_command("kill -9 `ps -fu %s | grep 'tail -%df /home/victorhooi/macbeth.txt' | grep -v grep | awk '{print $2}'`" % (uname, ltime))
        except:
            pass
    
        client.close()
        exit(0)

This would kill any open processes named tail. That may cause issues though if you have tails open that you dont want to close, if thats the case you could grep a ps, get the pid and kill -9 it.

First, set tail to read n lines from end of file before following. set n to a unique nuber like time.time(), since tail doesn't care if that number is larger then the number of lines in the file, the large number from time.time()shouldnt cause issues and will be unique. Then grep for that unique number in the ps:

   client.get_transport().open_session().exec_command("kill -9 `ps -fu %s | grep 'tail -%df /home/victorhooi/macbeth.txt' | grep -v grep | awk '{print $2}'`" % (uname, ltime))

Process dies, if it is run via paramiko ssh session and with & in the end

With & you make your remote command exit instantly. The remote sshd will therefore likely (depends on implementation, but openssh does) kill all processes that were started from your command invocation. In your case, you just spawn a new process nohup tcpdump which will instantly return due to & at the end. The channel.recv_exit_status() will only block until the exit code for the & ooperation is ready which instantly is. Your code then just terminates, terminating your ssh session which will make the remote sshd kill the spawned nohup tcpdump proc. That why you end up with no tcpdump process.

Here's what you can do:

Since exec_command is going to spawn a new thread for your command, you can just leave it open and proceed with other tasks. But make sure to empty buffers every now and then (for verbose remote commands) to prevent paramiko from stalling.

ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh.connect(host, username=login, password=password)
transport = ssh.get_transport()
channel_tcpdump = transport.open_session()
channel_tcpdump.get_pty()
channel_tcpdump.set_combine_stderr(True)

cmd = "tcpdump -i eth1  port 443 -w /tmp/dump20150317183305940107.pcap"        # command will never exit
channel_tcpdump.exec_command(cmd)  # will return instantly due to new thread being spawned.
# do something else
time.sleep(15)      # wait 15 seconds
_,stdout,_ = ssh.exec_command("pgrep tcpdump") # or explicitly pkill tcpdump
print stdout.read()     # other command, different shell
channel_tcpdump.close()     # close channel and let remote side terminate your proc.
time.sleep(10)

Run multiple commands in different SSH servers in parallel using Python Paramiko

Indeed, the problem is that you close the SSH connection. As the remote process is not detached from the terminal, closing the terminal terminates the process. On Linux servers, you can use nohup. I do not know what is (if there is) a Windows equivalent.

Anyway, it seems that you do not need to close the connection. I understood, that you are ok with waiting for all the commands to complete.

stdouts = []
clients = []

# Start the commands
commands = zip(ip_list[1:], user_list[1:], password_list[1:])
for i, (ip, user, password) in enumerate(commands, 1):
    print("Open session in: " + ip + "...")
    client = paramiko.SSHClient()
    client.connect(ip, user, password)
    command = \
        f"cd {path} && " + \
        f"python {python_script} {cluster} -type worker -index {i} -batch 64 " + \
        f"> {path}/logs/'command output'/{ip_list[i]}.log 2>&1"
    stdin, stdout, stderr = client.exec_command(command)
    clients.append(client)
    stdouts.append(stdout)

# Wait for commands to complete
for i in range(len(stdouts)):
    stdouts[i].read()
    clients[i].close()

---

Note that the above simple solution with stdout.read() is working only because you redirect the commands output to a remote file. Were you not, the commands might deadlock.

Without that (or if you want to see the command output locally) you will need a code like this:

while any(x is not None for x in stdouts):
    for i in range(len(stdouts)):
        stdout = stdouts[i]
        if stdout is not None:
            channel = stdout.channel
            # To prevent losing output at the end, first test for exit,
            # then for output
            exited = channel.exit_status_ready()
            while channel.recv_ready():
                s = channel.recv(1024).decode('utf8')
                print(f"#{i} stdout: {s}")
            while channel.recv_stderr_ready():
                s = channel.recv_stderr(1024).decode('utf8')
                print(f"#{i} stderr: {s}")
            if exited:
                print(f"#{i} done")
                clients[i].close()
                stdouts[i] = None
    time.sleep(0.1)

If you do not need to separate the stdout and stderr, you can greatly simplify the code by using Channel.set_combine_stderr. See Paramiko ssh die/hang with big output.

---

Regarding your question about SSHClient.close: If you do not call it, the connection will be closed implicitly, when the script finishes, when Python garbage collector cleans up the pending objects. It's a bad practice. And even if Python won't do it, the local OS will terminate all connections of the local Python process. That's a bad practice too. In any case, that will terminate the remote processes along.

---

Related Topics