Process Stuck in Exit, Shows as Zombie But Cannot Be Reaped

Process stuck in exit, shows as zombie but cannot be reaped

I finally figured it out! The process was actually doing useful work all this time. The process held the last reference to a large file on a slow filesystem. When the process terminates, the last reference to the file is release, forcing the OS to reclaim the space. The file was so large that this required tens of thousands of I/O operations, taking 10 minutes or more.

How to reap zombie process in docker container with bash

I wrote small demo in c that can help to demonstrate that bash had reaped the zombie processes and how it would look like if he had not.

First to explain the definition of zombie process. The zombie process is a process who had finished the work and generated an exit code. The resources are kept by the kernel waiting for the parent to collect the exit code.

To have zombie, parent needs to ignore the child's exit (don't issue wait and ignore SIGCHLD).

Reaping the zombies

The following c code is creating two zombie processes. One belonging to the main process, and one that belongs to the first child.

#include <stdio.h>
#include <stdlib.h>
#include <signal.h>
#include <pthread.h>
#include <sys/wait.h>
#include <unistd.h>

int main()
{
    printf("Starting Program!\n");

    int pid = fork();
    if (pid == 0)
    {
        pid = fork(); // Create a child zombie
        if (pid == 0) {
            printf("Zombie process %i of the child process\n", getpid());
            exit(10);
        } else {
            printf("Child process %i is running!\n", getpid());
            sleep(10);  // wait 10s
            printf("Child process %i is exiting!\n", getpid());
            exit(0);
        }
    }
    else if (pid > 0)
    {
        pid = fork();
        if (pid == 0) {
            printf("Zombie process %i from the parent process\n", getpid());
        } else {
            printf("Parent process %i...\n", getpid());
            sleep(5);
            printf("Parent process will crash with segmentation failt!\n");
            int* p = 0;
            p = 10;
        }
    }
    else perror("fork()");
    exit(-1);
}

I also created a docker container that will compile the file and the child. The whole project is available in following git repository

After running the build, and the demo, the following printout is shown in the console:

root@d2d87f4aafbc:/zombie# ./zombie & ps -eaf --forest
[1] 8
Starting Program!
Parent process 8...
Zombie process 11 from the parent process
Child process 10 is running!
Zombie process 12 of the child process
UID          PID    PPID  C STIME TTY          TIME CMD
root           1       0  0 10:43 pts/0    00:00:00 /bin/bash
root           8       1  0 10:43 pts/0    00:00:00 ./zombie
root          10       8  0 10:43 pts/0    00:00:00  \_ ./zombie
root          12      10  0 10:43 pts/0    00:00:00  |   \_ [zombie] <defunct>
root          11       8  0 10:43 pts/0    00:00:00  \_ [zombie] <defunct>
root           9       1  0 10:43 pts/0    00:00:00 ps -eaf --forest
root@d2d87f4aafbc:/zombie# Parent process will crash with segmentation failt!
ps -eaf --forest
UID          PID    PPID  C STIME TTY          TIME CMD
root           1       0  0 10:43 pts/0    00:00:00 /bin/bash
root          10       1  0 10:43 pts/0    00:00:00 ./zombie
root          12      10  0 10:43 pts/0    00:00:00  \_ [zombie] <defunct>
root          13       1  0 10:43 pts/0    00:00:00 ps -eaf --forest
[1]+  Exit 255                ./zombie
root@d2d87f4aafbc:/zombie# Child process 10 is exiting!
ps -eaf --forest
UID          PID    PPID  C STIME TTY          TIME CMD
root           1       0  0 10:43 pts/0    00:00:00 /bin/bash
root          14       1  0 10:43 pts/0    00:00:00 ps -eaf --forest

The main process (PID 8) creates two children.

A child (PID 10) that will create a zombie child (PID 12) and will sleep for 10 seconds.
A child that will become zombie (PID 11).

After the creation of the processes, the parent process will sleep for 5s and create segmentation fault, leaving the zombies.

When the main process dies, the PID 11 is inherited by bash and it's cleaned up (reaped). PID 10 is still working (sleeping is a kind of work for a process) he is left alone by bash, since PID 11 had not invoked wait, the PID 12 is still zombie.

After 5 seconds, PID 11 had finished sleeping and exited. Bash had reaped and inherited PID 12 after which bash had reaped PID 12

Leaving zombies

The other c application is just executing the bash as a child process, leaving it to be the PID 1, and he will ignore the zombies.

# docker run -ti --rm test /zombie/ignore
root@b9d49363cb57:/zombie# ./zombie & ps -eaf --forest
[1] 10
Starting Program!
Parent process 10...
Zombie process 13 from the parent process
Child process 12 is running!
Zombie process 14 of the child process
UID          PID    PPID  C STIME TTY          TIME CMD
root           1       0  0 11:18 pts/0    00:00:00 /zombie/ignore
root           7       1  0 11:18 pts/0    00:00:00 sh -c /bin/bash
root           8       7  0 11:18 pts/0    00:00:00  \_ /bin/bash
root          10       8  0 11:18 pts/0    00:00:00      \_ ./zombie
root          12      10  0 11:18 pts/0    00:00:00      |   \_ ./zombie
root          14      12  0 11:18 pts/0    00:00:00      |   |   \_ [zombie] <defunct>
root          13      10  0 11:18 pts/0    00:00:00      |   \_ [zombie] <defunct>
root          11       8  0 11:18 pts/0    00:00:00      \_ ps -eaf --forest
root@b9d49363cb57:/zombie# pParent process will crash with segmentation failt!
ps -eaf --forest
UID          PID    PPID  C STIME TTY          TIME CMD
root           1       0  0 11:18 pts/0    00:00:00 /zombie/ignore
root           7       1  0 11:18 pts/0    00:00:00 sh -c /bin/bash
root           8       7  0 11:18 pts/0    00:00:00  \_ /bin/bash
root          15       8  0 11:18 pts/0    00:00:00      \_ ps -eaf --forest
root          12       1  0 11:18 pts/0    00:00:00 ./zombie
root          14      12  0 11:18 pts/0    00:00:00  \_ [zombie] <defunct>
root          13       1  0 11:18 pts/0    00:00:00 [zombie] <defunct>
[1]+  Exit 255                ./zombie
root@b9d49363cb57:/zombie# Child process 12 is exiting!
ps -eaf --forest
UID          PID    PPID  C STIME TTY          TIME CMD
root           1       0  0 11:18 pts/0    00:00:00 /zombie/ignore
root           7       1  0 11:18 pts/0    00:00:00 sh -c /bin/bash
root           8       7  0 11:18 pts/0    00:00:00  \_ /bin/bash
root          16       8  0 11:18 pts/0    00:00:00      \_ ps -eaf --forest
root          12       1  0 11:18 pts/0    00:00:00 [zombie] <defunct>
root          13       1  0 11:18 pts/0    00:00:00 [zombie] <defunct>
root          14       1  0 11:18 pts/0    00:00:00 [zombie] <defunct>
root@b9d49363cb57:/zombie#

So now, we have 3 zombies left in the system, hanging.

zombie process can't be killed

Zombie processes are already dead, so they cannot be killed, they can only be reaped, which has to be done by their parent process via wait*(). This is usually called the child reaper idiom, in the signal handler for SIGCHLD:

while (wait*(... WNOHANG ...)) {
    ...
}

Linux, waitpid, WNOHANG, child process, zombie

I finally find out there were some fd leaks during deep tracing by lsof.

After fd leaks were fixed, the problem was gone.

Zombie process is generated when call system() in Perl threads

Threads are listed in the perldoc as "discouraged". Personally, I find they work fine, they're just somewhat counter intuitive - they aren't lightweight constructs like might be assumed (based on other threading models).

I will note - the generic solution to self-reaping zombies is to set $SIG{'CHLD'} e.g.: http://perldoc.perl.org/perlipc.html but that's probably not a good idea if you're capturing the return code. You could probably do an open and a waitpid instead though.

So I wouldn't normally suggest their use, unless you've a scenario where you need to do a lot of inter-thread communication. Parallel::ForkManager is generally much more efficient.

If you do have to use them - I wouldn't do what you're doing, and spawning a thread per 'job' and instead use a worker threads model with Thread::Queue.

I can't say for certain, but I suspect one of your problems is this line:

$cmd = "$HOME/worker.sh "."$arg";

Because perl will be interpolating $HOME - and you don't define it, therefore it's null.

You really should be turning on strict and warnings and cleaning up any errors as a result - your code has quite a few.

But that said - unless I'm missing something your code is much more complicated than it needs to be - it looks like all you're doing here is running parallel ssh commands.

So I'd suggest what you'd be better off with is something like this:

#!/usr/bin/env perl
use strict;
use warnings;

use threads;
use Thread::Queue;

my @servers = qw/hostA hostB/;

my $cmd         = '$HOME/worker.sh --node';
my $threadcount = 2;

my $hostq  = Thread::Queue->new();
my $errorq = Thread::Queue->new();

sub worker {
    while ( my $hostname = $hostq->dequeue ) {
        my $output =
            qx( ssh -o StrictHostKeyChecking=no $hostname \"source /etc/profile; $cmd\" );
        if ($?) {
            $errorq->enqueue("$hostname: $output");
        }
    }
}

$hostq->enqueue(@servers);
for ( 1 .. $threadcount ) {
    my $thr = threads->create( \&worker );
}
$hostq->end();

foreach my $thr ( threads->list ) {
    $thr->join;
}
$errorq->end();
while ( my $error = $errorq->dequeue ) {
    print "ERROR: $error\n";
}

Alternatively, with Parallel::ForkManager:

#!/usr/bin/env perl
use strict;
use warnings;

my @servers = qw/hostA hostB/;

my $cmd     = '$HOME/worker.sh --node';
my $manager = Parallel::ForkManager->new(5);    #fork limit.

foreach my $hostname (@servers) {
    $manager->start and next;
    my $output =
        qx( ssh -o StrictHostKeyChecking=no $hostname \"source /etc/profile; $cmd\" );
    if ($?) {
        print "ERROR: $hostname $output\n";
    }
    $manager->finish;
}

$manager->wait_all_children();

Remove persistent zombie process from the command line

The only kill command that reaps the zombie process is the one that kills its parent, since it will then reparent to init which will then proceed to reap it.

Not really possible when it's a kernel thread though...

Not able to kill a fork'ed process

You are killing yourself. fork() returns 0 if you are in the forked process, or the child process id (PID) in the 'master' process.

So, the upper branch of your if() clause is performed in the master process, where you copy the child's process ID (stored in a) to child_pid.

In the lower branch you are in the child process, where take the child_pid, which is your own and then happily kill() yourself... That's why you never get the line 'Killed with...'

As paxdiablo pointed out, since this is a child process it will remain a zombie until you fetch the exit status with wait() or the master process exits.

BTW, I'm not sure what you want to do with this code:

If you want to exit the child process gracefully you can just do an exit().
If you want to kill your child process, keep track of the child PID (returned by fork()) and then kill() it from your master process.
If you want to kill the master process from your child (odd as that may sound) be careful, as that may take the child process(es) with it. You have to detach the child process from the master process (see the manual page for daemon()).

Process Stuck in Exit, Shows as Zombie But Cannot Be Reaped