Can Upstart Expect/Respawn Be Used on Processes That Fork More Than Twice

Can upstart expect/respawn be used on processes that fork more than twice?

I came up with an ugly hack to make this work. It works for my application on my system. YMMV.

1. start the application in the pre-start section
2. in the script section run a script that runs as long as the application runs. The pid of this script is what upstart will track.
3. in the post-stop section kill the application

example

env DAEMON=/usr/bin/forky-application

pre-start script
    su -s /bin/sh -c "$DAEMON" joeuseraccount
end script

script 
    sleepWhileAppIsUp(){
        while pidof $1 >/dev/null; do
            sleep 1
        done
    }

    sleepWhileAppIsUp $DAEMON
end script

post-stop script
    if pidof $DAEMON;
    then
        kill `pidof $DAEMON`
        #pkill  $DAEMON # post-stop process (19300) terminated with status 1
    fi
end script

a similar approach could be taken with pid files.

delayed_job with multiple workers and upstart

Out of the box, it appears that upstart expect does not support the behavior outlined in https://github.com/collectiveidea/delayed_job#running-jobs , as there are multiple workers that each fork twice to daemonize.

As outlined in this question about upstart: Can upstart expect/respawn be used on processes that fork more than twice? , you can use a bit of scripting to shepherd the processes yourself in the different hooks.

Another option would be to use upstart job instances (http://upstart.ubuntu.com/cookbook/#instance) to start multiple jobs that do not fork.

Ubuntu upstart gets incorrect PID from Play 1.3

This is because getJavaVersion() spawns a subprocess, which bumps the PID count, which breaks Upstart, the latter which expects Play to fork exactly none, once or twice, depending on which expect stanza you use.

I've fixed this in a pull request.

Upstart script for bitcoind, respawn feature

oom never

is your first problem. You need this:

oom score never

Additionally, do not use oom score never except for critical system services. Try -500 or -700 instead. That should be a higher priority than most processes, but not the ones that are essential for any running system. So you should use:

oom score -500

The second issue is that you are using start-stop-daemon. You should just ditch that, as Upstart can handle everything. So the resulting script would look like this:

description "bitcoind"

start on filesystem
stop on runlevel [!2345]

oom score -500
chdir /root/.bitcoin

respawn
respawn limit 10 60 # 10 times in 60 seconds

exec /usr/bin/bitcoind

The last issue could be that you have not defined normal exit properly. You need to specify which return codes and signals constitute a normal exit, so that Upstart knows to respawn if the signals and return codes do not match. See the Upstart cookbook on how to do this: http://upstart.ubuntu.com/cookbook/#normal-exit.

Upstart stop if running

A colleague came up with a great solution

if ( initctl status node-App | grep start ); then
  initctl stop node-App
fi

Very simple and works properly when run from Jenkins

---

Related Topics