Correct Daemon Behaviour (From Pep 3143) Explained

Correct daemon behaviour (from PEP 3143) explained

PEP 3142 took these requirements from Unix Network Programming ('UNP') by the late W. Richard Stevens. The explanation below is quoted or summarised from that book. It's not so easily found online, and it may be illegal to download. So I borrowed it from the library. Pages referred to are in the second edition, Volume 1 (1998). (The PEP refers to the first edition, 1990.)

Close all open file descriptors.

"We close any open descriptors inherited from the process that executed the daemon (ie the shell). [..] Some daemons open /dev/null for reading and writing and duplicate the descriptor to standard input, standard output and standard error."

(This 'Howdy World' Python daemon demonstrates this.)

"This guarantees that the common descriptors are open, and a read from any of these descriptors returns 0 (End Of File) and the kernel just discards anything written to any of these three descriptors. The reason for opening these descriptors is so that any library function called by the daemon that assumes it can read from standard input or write to standard output or standard error, will not fail. Alternately, some daemons open a log file that they will write to while running and duplicate its descriptor to standard output and standard error". (UNP p. 337)

Change current working directory

"A printer daemon might change to the printer's spool directory, where it does all its work. [...] The daemon could have been started anywhere in the filesystem, and if it remains there, that filesystem cannot be unmounted." (UNP p 337)

Why would you want to unmount a filesystem? Two reasons:

1. You want to separate (and be able mount and unmount) directories that can fill up with user data from directories dedicated to the OS.

2. If you start a daemon from, say, a USB-stick, you want to be able to unmount that stick without interfering with the daemon.

Reset the file access creation mask.

"So that if the daemon creates its own files, permission bits in the inherited file mode creation mask do not affect the permission bits of the new files." (UNP, p 337)

Run in the background.
By definition,

"a daemon is a process that runs in the background and is independent of control from all terminals". (UNP p 331)

Disassociate from process group.
In order to understand this, you need to understand what a process group is, and that means you need to know what fork does.

What fork does

fork is the only way (in Unix) to create a new process. (in Linux, there is also clone). Key in understanding fork is that it returns twice when called (once): once in the calling process (= parent) with the process ID of the newly created process (= child), and once in the child. "All descriptors known by the parent when forking, are shared with the child when fork returns." (UNP p 102).
When a process wants to execute another program, it creates a new process by calling fork, which creates a copy of itself. Then one of them (usually the child) calls the new program. (UNP, p 102)

Why disassociate from process group

The point is that a session leader may acquire a controlling terminal. A daemon should never do this, it must stay in the background. This is achieved by calling fork twice: the parent forks to create a child, the child forks to create a grandchild. Parent and child are terminated, but grandchild remains. But because it's a grandchild, it's not a session leader, and therefor can't acquire a controlling terminal. (Summarised from UNP par 12.4 p 335)

The double fork is discussed in more detail here, and in the comments below.

Ignore terminal I/O signals.

"Signals generated from terminal keys must not affect any daemons started from that terminal earlier". (UNP p. 331)

Disassociate from control terminal and don't reacquire a control terminal.
By now, the reasons are obvious:

"If the daemon is started from a terminal, we want to be able to use that terminal for other tasks at a later time. For example, if we start the daemon from a terminal, log off the terminal, and someone else logs in on that terminal, we do not want any daemon error messages appearing during the next user's terminal session." (UNP p 331)

Correctly handle the following circumstances:

• Started by System V init process

  • A daemon should be launchable at boot time, obviously.

• Daemon termination by SIGTERM signal

  • SIGTERM means Signal Terminate. At shutdown, the init process normally sends SIGTERM to all processess, waits usually 5 to 20 seconds to give them time to clean up and terminate. (UNP, p 135) Also, a child can send SIGTERM to its parent, when its parent should stop doing what it's doing. (UNP p 408)

• Children generate SIGCLD signal

  • Stevens discusses SIGCHLD, not SIGCLD. The difference between them isn't important for understanding daemon behaviour. If a child terminates, it sends SIGCHLD to it's parent. If a parent doesn't catch it, the child becomes a zombie (UNP p 118). Oh what fun.

On a final note, when I started to find answers to my question in UNP, it soon struck me I really should read more of it. It's 900+ (!) pages, from 1998 (!) but I believe the concepts and the explanations in UNP stand the test of time, gloriously. Stevens not only knew very well what he was talking about, he also understood what was difficult about it, and made it easier to understand. That's really rare.

How to start daemon process from python on windows?

Using the answer Janne Karila pointed out this is how you can run a process that doen't die when its parent dies, no need to use the win32process module.

DETACHED_PROCESS = 8
subprocess.Popen(executable, creationflags=DETACHED_PROCESS, close_fds=True)

DETACHED_PROCESS is a Process Creation Flag that is passed to the underlying CreateProcess function.

Using appropriate POSIX signals

If you're in control of the child processes, you can pretty much do as you please, but SIGTERM is the self-documenting signal for this. It asks a process to terminate, politely: the process chooses how to handle the signal and may perform cleanup actions before actually exiting (or may ignore the signal).

The standard way to kill a process, then, is to first send a SIGTERM; then wait for it to terminate with a grace period of, say, five seconds (longer if termination can take a long time, e.g. because of massive disk I/O). If the grace period has expired, send a SIGKILL. That's the "hard" version of SIGTERM and cannot be ignored, but also leaves the process no chance of neatly cleaning up after itself. Having to send a SIGKILL should be considered an issue with the child process and reported as such.

How do I implement a simple cross platform Python daemon?

Two options come to mind:

1. Port your program into a windows service. You can probably share much of your code between the two implementations.

2. Does your program really use any daemon functionality? If not, you rewrite it as a simple server that runs in the background, manages communications through sockets, and perform its tasks. It will probably consume more system resources than a daemon would, but it would be quote platform independent.

How do you create a daemon in Python?

Current solution

A reference implementation of PEP 3143 (Standard daemon process library) is now available as python-daemon.

Historical answer

Sander Marechal's code sample is superior to the original, which was originally posted in 2004. I once contributed a daemonizer for Pyro, but would probably use Sander's code if I had to do it over.

change signal disposition of already running process (SIGHUP)

Use disown(1)

disown: disown [-h] [-ar] [jobspec
...]
Remove jobs from current shell.

Removes each JOBSPEC argument from the table of active jobs.  Without
any JOBSPECs, the shell uses its notion of the current job.

Options:
  -a        remove all jobs if JOBSPEC is not supplied
  -h        mark each JOBSPEC so that SIGHUP is not sent to the job if
            the shell receives a SIGHUP
  -r        remove only running jobs

Detaching a process from terminal, entirely

"disown" is a bash builtin that
removes a shell job from the shell's
job list. What this basically means is
that you can't use "fg", "bg" on it
anymore, but more importantly, when
you close your shell it won't hang or
send a SIGHUP to that child anymore.
Unlike "nohup", "disown" is used after
the process has been launched and
backgrounded.

Python Multiprocessing Kill Processes

You need to .join() on your processes in a worker Queue, which will lock them to the calling application until all of them succeed or kill when the parent is killed, and run them in daemon mode.

http://forums.xkcd.com/viewtopic.php?f=11&t=94726

end daemon processes with multiprocessing module

http://docs.python.org/2/library/multiprocessing.html#the-process-class

http://www.python.org/dev/peps/pep-3143/#correct-daemon-behaviour

---

Related Topics