Simple Linux Signal Handling

[Q-3] Does the terminate variable in my example have to be volatile? I've
seen many examples where this variable is volatile, and others where
it is not.

The flag terminate should be volatile sig_atomic_t:

Because handler functions can be called asynchronously. That is, a handler might be called at any point in the program, unpredictably. If two signals arrive during a very short interval, one handler can run within another. And it is considered better practice to declare volatile sig_atomic_t, this type are always accessed atomically, avoid uncertainty about interrupting access to a variable. volatile tells the compiler not to optimize and put it into register. (read: Atomic Data Access and Signal Handling for detail expiation).

One more reference: 24.4.7 Atomic Data Access and Signal Handling.
Furthermore, the C11 standard in 7.14.1.1-5 indicates that only objects of volatile sig_atomic_t can be accessed from a signal handler (accessing others has undefined behavior).

[Q-4] I've read that signal() is now deprecated, and to use sigaction(). Are
there any really good examples to show how to convert from the
previous signal() call? I'm having trouble with the new structure that
I have to create/pass and how it all fits together.

The example below (and the link in the comments) can be helpful:

// 1. Prepare struct 
struct sigaction sa;
sa.sa_handler =  sighandler;

// 2. To restart functions if interrupted by handler (as handlers called asynchronously)
sa.sa_flags = SA_RESTART; 

// 3. Set zero 
sigemptyset(&sa.sa_mask);

/* 3b. 
 // uncomment if you wants to block 
 // some signals while one is executing. 
sigaddset( &sa.sa_mask, SIGINT );
*/ 

// 4. Register signals 
sigaction( SIGINT, &sa, NULL );

references:

Beginning Linux Programming, 4th Edition: in this book, exactly your code is explained with sigaction() nicely in "Chapter 11: Processes and Signals".
The sigaction documentation, including an example (quick learning).
The GNU C Library: Signal Handling

_{*I started from 1, Presently I am reading 3 GNU-library}

[Q-5] Is the second call to signal() necessary? Is there something similar that I need to be concerned with for sigaction()?

Why you set it to default-action before program termination is unclear to me. I think the following paragraph will give you an answer:

Handling Signals

The call to signal establishes signal handling for only one occurrence of a signal. Before the signal-handling function is called, the library resets the signal so that the default action is performed if the same signal occurs again. Resetting signal handling helps to prevent an infinite loop if, for example, an action performed in the signal handler raises the same signal again. If you want your handler to be used for a signal each time it occurs, you must call signal within the handler to reinstate it. You should be cautious in reinstating signal handling. For example, if you continually reinstate SIGINT handling, you may lose the ability to interrupt and terminate your program.

The signal() function defines the handler of the next received signal only, after which the default handler is reinstated. So it is necessary for the signal handler to call signal() if the program needs to continue handling signals using a non-default handler.

Read a discussion for further reference: When to re-enable signal handlers.

[Q-1a] Is any signal handling necessary?

Yes, Linux will do cleanup for you. For example if you don't close a file or a socket, Linux will do the cleanup after the process terminates. But Linux may not necessary perform the clean up immediately and it may take some time (may be to keep system performance high or some other issues). For example if you don't close a tcp-socket and the program terminates the kernel will not close the socket immediately to ensure all data has been transmitted, TCP guarantees delivery if possible.

[Q-1b] Therefore, can I just replace the signal handler with just an infinite loop and let the OS gracefully exit the threads, de-allocate the memory, etc?

No, operating system performs do clean-up only after program terminates. While a process executes, resources that are allocated to that process don't get claimed by the OS. (The OS can't know whether your process is in an infinite loop or not - this is an unsolvable problem). If you want that after process termination the OS performs the clean-up operations for you, then you don't need to handle signals (even in case your process abnormally terminated by a signal).

[Q] All I'm trying to accomplish to to have my: main loop run until either ctrlc or power is disconnected or something really bad happens.

No, there is a limitation! You can't catch all signals. Some signals are not catchable e.g. SIGKILL and SIGSTOP and both are termination signals. Quoting one:

— Macro: int SIGKILL

The SIGKILL signal is used to cause immediate program termination. It cannot be handled or ignored, and is therefore always fatal. It is also not possible to block this signal.

So you can't make a program that cannot be interrupted (an uninterrupted program)!

_{I am not sure but may be you can do something like this in Windows systems: by writing TSRs (some sort of kernel-mode hooking). I remember from my thesis time that some viruses couldn't be terminated even from task manager but I also believe that they trick user by admin permissions.}

I hope this answer will help you.

What is best practice for signal handling in production multi-threaded program on Linux?

Is this a good practice,

Yes absolutely. Handling termination signals in a multi-threaded environment any other way(by not having a single thread responsible for them) is virtually impossible.

or I should provide a custom action for every signal?

No. You'd normally want to handle SIGINT, SIGTERM and SIGHUP. SIGKILL can't be handled, and I'd leave SIGQUIT alone so it could be used to core-dump the application.

Signal handling with multiple threads in Linux

This is slightly nuanced, based on which version of the Linux kernel you are using.

Assuming 2.6 posix threads, and if you are talking about the OS sending SIGTERM or SIGHUP, the signal is sent to process, which is received by and handled by root thread. Using POSIX threads, you can also sent SIGTERM to individual threads as well, but I suspect you are asking about what happens when the OS sends the signal to the process.

In 2.6, SIGTERM will cause child threads to exit "cleanly", where as 2.4, child threads were left in an indeterminate state.

How to avoid using printf in a signal handler?

You can use some flag variable, set that flag inside signal handler, and based on that flag call printf() function in main() or other part of program during normal operation.

It is not safe to call all functions, such as printf, from within a signal handler.
A useful technique is to use a signal handler to set a flag and then check that flag
from the main program and print a message if required.

Notice in example below, signal handler ding() set a flag alarm_fired to 1 as SIGALRM caught and in main function alarm_fired value is examined to conditionally call printf correctly.

static int alarm_fired = 0;
void ding(int sig) // can be called asynchronously
{
  alarm_fired = 1; // set flag
}
int main()
{
    pid_t pid;
    printf("alarm application starting\n");
    pid = fork();
    switch(pid) {
        case -1:
            /* Failure */
            perror("fork failed");
            exit(1);
        case 0:
            /* child */
            sleep(5);
            kill(getppid(), SIGALRM);
            exit(0);
    }
    /* if we get here we are the parent process */
    printf("waiting for alarm to go off\n");
    (void) signal(SIGALRM, ding);
    pause();
    if (alarm_fired)  // check flag to call printf
      printf("Ding!\n");
    printf("done\n");
    exit(0);
}

_{Reference: Beginning Linux Programming, 4th Edition, In this book exactly your code is explained (what you want), Chapter 11: Processes and Signals, page 484}

Additionally, you need to take special care in writing handler functions because they can be called asynchronously. That is, a handler might be called at any point in the program, unpredictably. If two signals arrive during a very short interval, one handler can run within another. And It is considered better practice to declare volatile sigatomic_t, this type are always accessed atomically, avoid uncertainty about interrupting access to a variable. (read: Atomic Data Access and Signal Handling for detail expiation).

Read Defining Signal Handlers :to learn how to write a signal handler function that can be established with the signal() or sigaction() functions.

List of authorized functions in manual page, calling this function inside signal handler is safe.

C Programming - Counting signals in signal handler

To the best of my knowledge, you can't use signals for that. If two signals of the same kind are sent to a process before it gets scheduled to handle the first one, it will only see one signal. Think of it as a bit mask, there is one bit for each pending signal, and when the process gets scheduled it will receive them all. But if it is waiting for some other process, and a signal for which the bit in the mask is already set, then nothing more happens.

A better solution would probably be to open a pipe to each subprocess, and each of them writes a message when done. When the parent has read the message from all children, it can continue. There are other synchronisation methods, but this would probably be the simplest.

Simple Linux Signal Handling