What Happens When a Signal Is Received While Already in a Signal Handler

What happens when a signal is received while already in a signal handler?

In your concrete example (the same signal being received), the signal is delivered after the signal handler has finished (so bullet point #2 is correct). Note, however, that you may "lose" signals.

The reason for that is that while a signal is being inside its handler, it is blocked. Blocked signals are set to pending, but not queued. The term "pending" means that the operating system remembers that there is a signal waiting to be delivered at the next opportunity, and "not queued" means that it does this by setting a flag somewhere, but not by keeping an exact record of how many signals have arrived.

Thus, you may receive 2 or 3 (or 10) more SIGCHLD while in your handler, but only see one (so in some cases, bullet point #1 can be correct, too).

Note that several flags that you can pass to sigaction can affect the default behaviour, such as SA_NODEFER (prevents blocking signal) and SA_NOCLDWAIT (may not generate signal at all on some systems).

Now of course, if you receive a different type of signal, there's no guarantee that it won't interrupt your handler. For that reason, one preferrably doesn't use non signal safe functions.

Waiting for a signal inside the signal handler itself

Since you're using signal rather than sigaction, it's important to note the following from the POSIX standard:

When a signal occurs, and func points to a function, it is implementation-defined whether the equivalent of a:

signal(sig, SIG_DFL);

is executed or the implementation prevents some implementation-defined set of signals (at least including sig) from occurring until the current signal handling has completed.

This reflects the two historic signal implementations, SVID and BSD.
Since you're using Ubuntu 18.04, you're likely using glibc, which implements the latter (BSD) semantics. SIGUSR1 is masked while its handler is executing.

Since you want the SVID semantics, in which no signals are masked and you need to reestablish the signal handler each time the signal handler is called, you should replace your signal(SIGUSR1, sig_handler); calls with the following:

struct sigaction sa = { .sa_handler = sig_handler, .sa_flags = SA_NODEFER|SA_RESETHAND };
sigemptyset(&sa.mask);
sigaction(SIGUSR1, &sa, NULL);

The SA_NODEFER flag together with the empty mask means no signals will be masked; SA_RESETHAND means the signal action will be reset to SIG_DFL.

In addition, as the other answers said, you shouldn't be calling printf from within the signal handler. The Linux signal safety man page says which functions can be called. sigaction, signal, and pause are OK. You can use write to write strings instead of printf.

Interruption of signal handler with other signal?

Yes, the execution of a signal handler may itself be interrupted by the delivery of another signal.

There are a few nuances, however.

By default, user-defined signal handlers temporarily block the very signal which invoked them. This is the default behavior of sigaction unless the SA_NODEFER flag is set. (This is also the behavior of the older, discouraged signal function, at least on most implementations.)

Additionally, sigaction can explicitly block signals during the handler's execution by setting the sa_mask member of the const struct sigaction. Most code you see will explicitly empty this member during struct initialization, though it is often more robust to sigfillset this member and not worry about interruptions.

So, again, yes. Handle EINTR, errno, &c. as appropriate, or, better yet, design the handler to do no more than set a sig_atomic_t flag and avoid many worries.

How exactly signal handler will work if it receives another signal while processed?

Here's your misconception:

and this first handler will quit here, and num ++ will not work.

When one signal handler interrupts another, once the interrupting one finishes, the interrupted one resumes where it left off. It doesn't just end early.

By the way, lots of problems with your code:

1. Since you're not using SA_SIGINFO, your handler should only take one parameter and should go in sa_handler instead of sa_sigaction.
2. You're not initializing most fields of the struct sigaction, so really weird stuff could end up happening when you call sigaction on it.
3. You're really restricted in what you're allowed to do from inside a signal handler; in particular, you're not allowed to access a global variable of type int. Change num to be a volatile sig_atomic_t.
4. You should basically never use write outside of a loop, since partial writes are allowed to happen at basically any time.

As for why sending SIGUSR1 twice doesn't always run the handler twice, that's because non-real-time signals are allowed to coalesce, so if your second kill happens before the signal handler for the first one starts running, then the second one effectively won't do anything.

What happens if during a signal handling in UNIX, the same signal gets sent to the program?

To answer the second part of your question, "is it true that signal handlers should do as minimal work as possible?" the answer is yes, because there is a very minimal set of functions that are "async signal safe" and therefore able to be called from signal handlers. Async signal safety is kind of an enhanced form of re-entrancy. If foo() is async signal safe, that means that it's safe to call foo() within a signal handler, even if foo() was already executing when the signal was raised.

You can get the full list of async signal safe functions by looking that the section 7 man page for signal (man 7 signal). Calling any function other than one of these from within a signal handler, directly or indirectly, invokes undefined behavior.

The "write a byte to a pipe" approach is a great way to deal with signals without being restricted to async signal safe functions, especially if your program is already oriented around a select loop.

What happens to threads when process receives signal?

Some intro to threads and signals (signal(7))

• Signal Disposition is per-process

The signal disposition is a per-process attribute: in a multithreaded application, the disposition of a particular signal is the same for all threads.

• Signal may be process-directed or thread-directed

Process-directed Signals: A process-directed signal is one that is targeted at (and thus pending for) the process as a whole. A process-directed signal may be delivered to any one of the threads that do not currently have the signal blocked. If more than one of the threads has the signal unblocked, then the kernel chooses an arbitrary thread to which to deliver the signal.

Thread-directed Signals: A thread-directed signal is one that is targeted at a specific thread. The set will consist of the union of the set of pending process-directed signals and the set of signals pending for the calling thread.

• Asynchronous and Synchronous Signal Handling

You can configure your program to tell how to deal with signals. You can ignore them (few can't be ignored), register a signal handler which will be invoked when that specific signal is received (asynchronous), or block it to deal with it later (synchronous).

Coming to your case,

"The question is: what happens to threads when signal handling function is running?"

The signal is delivered once to any thread that is configured to receive it. The thread, which is asynchronously handling the signal, stops whatever it is doing and jumps to the configured signal handler. The flow of the execution in the remaining threads is unaffected.

If threads continue running their jobs, is there a way to freeze them while debugging handler is working?

There is no standard way of doing this. You need to build your own mechanism for enabling this.

To research further, some clarity is required regarding where the debug handler is executed. In each thread or in main() or in a specific thread?

Edit

Assuming main() implements the logging functionality, below tries to implement the base minimal for the same. Comments are added which enables to walk through code and understand the implementation.

#define THREAD_MAX_COUNT 100

#include <pthread.h>
#include <semaphore.h>
#include <signal.h>
#include <sys/signalfd.h>
#include <stdio.h>
#include <unistd.h>
#include <stdlib.h>

int debug;
sigset_t debug_mask;
pthread_t main_tid;

void* thread_func(void* th_data)
{
    /* .... */

    for ( ; ; ) {

        if (debug) {    // If debug procedure starts
            printf("Freezing %d\n", *((int*) th_data));

            pthread_kill(main_tid, SIGRTMIN);  // Notify the main thread about the thread's freeze.
            int signo;
            sigwait(&debug_mask, &signo);  // Wait till logging is done. main() will signal once it is done. 

            printf("Resuming %d\n", *((int*) th_data));
        }

        /* ... */
    }

    return NULL;
}

int main() {

    /* Block SIGINT SIGRTMIN*/

    sigset_t sigmask;
    sigemptyset(&sigmask);
    sigaddset(&sigmask, SIGINT);
    sigaddset(&sigmask, SIGRTMIN);

    pthread_sigmask(SIG_BLOCK, &sigmask, NULL);

    /* Set debug variables */

    debug = 0;
    sigemptyset(&debug_mask);
    sigaddset(&debug_mask, SIGRTMIN);
    main_tid = pthread_self();

    /* Get signalfd for SIGINT */

    int sigfd = signalfd(-1, &sigmask, 0);
    struct signalfd_siginfo sigbuf;

    /* Select variable initializations */

    fd_set rd_set, tr_set;
    FD_ZERO(&rd_set);
    FD_SET(sigfd, &rd_set);

    int td_count = 0;
    pthread_t tids[THREAD_MAX_COUNT];

    for ( ; ; ) {
        /* Wait for signal */
        tr_set = rd_set;

        select(sigfd + 1, &tr_set, NULL, NULL, NULL);

        if (FD_ISSET(sigfd, &tr_set)) {
            /* Read the pending signal */
            read(sigfd, &sigbuf, sizeof(sigbuf));

            /* Start logging */
            debug = 1;

            int signo;
            for (int count = 0; count < td_count; count++) {
                /* Wait for all threads to freeze */
                sigwait(&debug_mask, &signo);
            }

            printf("Logging...\n");
            sleep(3);

            /* End logging and resume threads */
            debug = 0;

            for (int count = 0; count < td_count; count++)
                pthread_kill(tids[count], SIGRTMIN);

            /* Note below code is for testing purpose; Creates new thread on each interruption */
            int* td_data = malloc(sizeof(int));
            *td_data = td_count;

            pthread_create(tids + td_count, NULL, thread_func, td_data);

            td_count++;
        }
    }

    return 0;
}

Terminal Session:

$ gcc SO.c -lpthread 
$ ./a.out 
^CLogging...
^CFreezing 0
Logging...
Resuming 0
^CFreezing 0
Freezing 1
Logging...
Resuming 0
Resuming 1
^CFreezing 0
Freezing 1
Freezing 2
Logging...
Resuming 1
Resuming 0
Resuming 2
^CFreezing 2
Freezing 3
Freezing 1
Freezing 0
Logging...
Resuming 1
Resuming 3
Resuming 2
Resuming 0
^CFreezing 1
Freezing 4
Freezing 3
Freezing 0
Freezing 2
Logging...
Resuming 1
Resuming 2
Resuming 0
Resuming 4
Resuming 3
^CFreezing 3
Freezing 0
Freezing 4
Freezing 2
Freezing 5
Freezing 1
Logging...
Resuming 0
Resuming 1
Resuming 2
Resuming 5
Resuming 3
Resuming 4
^\Quit (core dumped)

calling signal handler again in itself

The problem is that the behaviour of signal() varies across UNIX versions, and has also varied historically across different versions of Linux (quoted from Linux man). Especially:

In the original UNIX systems, when a handler that was established using
signal() was invoked by the delivery of a signal, the disposition of
the signal would be reset to SIG_DFL, and the system did not block
delivery of further instances of the signal. This is equivalent to
calling sigaction(2) with the following flags:

sa.sa_flags = SA_RESETHAND | SA_NODEFER;

So in such a system you have to call signal() again after a signal has been delivered. Because of these portability issues, the man page starts with:

The behavior of signal() varies across UNIX versions, and has also varied historically across different versions of Linux. Avoid its use:
use sigaction(2) instead. See Portability below.

What happens if ctx.rip and ctx.rsp in a signal handler is modified

I've figured it out, many thanks to Ian's hint:

https://groups.google.com/forum/#!topic/golang-nuts/BA7Dqp_zcwk

The root cause seems similar to the "uncertainty principle".

As an observer, by adding a println call in asyncPreempt
as well as asyncPreempt2 influences the actual behavior
after signal handling. The println involves stack split check,
which calls the morestack.

It took me a while to realize that morestack stores its caller
pc in g.m.morebuf.pc since getcallerpc in newstack
always returns the pc from morestack, which doesn't tell
too much information.

//go:nosplit
func asyncPreempt2() {
    // println("asyncPreempt2 is called") // comment here omits calling morestack.
    gp := getg()
    gp.asyncSafePoint = true
    if gp.preemptStop {
        mcall(preemptPark)
    } else {
        mcall(gopreempt_m)
    }
    println("asyncPreempt2 finished")
    gp.asyncSafePoint = false
}

TLDR: after signal handler, kernal restores asyncPreempt's rip and directly switches to it, nothing happens between runtime.asyncPreempt and runtime.sigtramp.

Read blocks again after returning from signal handler

According to everything I know and the man page on read(2),

EINTR  The call was interrupted by a signal before any data  was  read;
              see signal(7).

read should return -1 and set errno to EINTR.

You are reading too much into that. read() can return -1 and set errno to EINTR, which combination should be interpreted to mean that it was interrupted by a signal (before any data were read). It's even safe to say that if read fails on account of being interrupted by a signal, then that's the way to expect it to manifest. But that does not mean that read is guaranteed to fail in the event that a signal is received while it is blocking.

However, the output of the program suggests that it blocks again on
read after returning from the signal handler.

That is indeed one possible behavior. In particular, it is part of the BSD semantics for signal() and signal handling, and that is the default for Glibc (subject to the _BSD_SOURCE feature-test macro), so that's what you would expect by default on both Mac (because BSD) and most Linux (because Glibc). The "Portability" section of the manual page for Glibc's implementation of signal() goes into the matter in some detail.

Additionally this answer on the same thread says that signal() calls
sigaction() underneath, so why is the behaviour in case of system
calls different for the two?

The key thing about sigaction() is that it provides mechanisms for specifying all the details for the handling of the signal whose disposition is set, including, notably,

• whether (certain) system calls resume after the signal is handled;
• whether the signal disposition is reset upon receipt of the signal; and
• whether the signal is blocked while its handler is running.

Thus, if some implementation of signal() operates by calling sigaction(), that has no inherent implication for these or other ancillary behaviors upon subsequent receipt of the signal. Nor does it mean that registering a handler for that signal directly via sigaction() must produce the same effect as doing so indirectly via signal() -- it all depends on the arguments to sigaction(). The caller gets to choose.

Note in particular the advice from the manpage linked above:

The only portable use of signal() is to set a signal's disposition
to SIG_DFL or SIG_IGN. The semantics when
using signal() to establish a signal handler vary across systems (and POSIX.1 explicitly permits this variation); do not use it for this purpose.

(emphasis in the original.)

Supposing that you want system calls not to be resumed after being interrupted by SIGINT and handled by your handler, you get that from sigaction by avoiding specifying SA_RESTART among the flags. Most likely, you don't want any of the other flags, either, so that would mean something (for example) like this:

// with this form, flags and signal mask are initialized by default to all-bits-zero
sigaction(SIGINT, & (struct sigaction) { .sa_handler = handler }, NULL);

---

Related Topics