Linux, Need Accurate Program Timing. Scheduler Wake Up Program

insuring CPU time after waking up - linux scheduler

You said you are observing delays while writing. I think in this situation you can use
schedule_timeout function. Device drivers use this technique while writing to register so that they dont lockup the system. Recently, I have come across a problem where writing to register is causing delays; I am planning to do schedule_timeout in my case too.

Setting priority, scheduling mode will not help here.

Microsecond accurate (or better) process timing in Linux

If you are looking for this level of timing resolution, you are probably trying to do some micro-optimization. If that's the case, you should look at PAPI. Not only does it provide both wall-clock and virtual (process only) timing information, it also provides access to CPU event counters, which can be indispensable when you are trying to improve performance.

http://icl.cs.utk.edu/papi/

Getting accurate time measurement with `perf-stat`

There are a variety of reasons you can see variation when you repeatedly benchmark what appears to be the same code. I have covered some of the reasons in another answer and it would be worthwhile to keep those in mind.

However, based on experience and playing the probabilities, we can eliminate many of those up front. What's left are the most likely causes of your relatively large deviations for short programs from a cold start:

CPU power saving and frequency scaling features.
Actual runtime behavior differences, i.e., different code executed in the runtime library, VM, OS or other supporting infrastructure each time your run your program.
Some caching effect, or code or data alignment effect that varies from run to run.

You can probably separate these three effects with a plain perf stat without overriding the event list, like:

$ perf stat true

 Performance counter stats for 'true':

          0.258367      task-clock (msec)         #    0.427 CPUs utilized          
                 0      context-switches          #    0.000 K/sec                  
                 0      cpu-migrations            #    0.000 K/sec                  
                41      page-faults               #    0.159 M/sec                  
           664,570      cycles                    #    2.572 GHz                    
           486,817      instructions              #    0.73  insn per cycle         
            92,503      branches                  #  358.029 M/sec                  
             3,978      branch-misses             #    4.30% of all branches        

       0.000605076 seconds time elapsed

Look first at the 2.572 GHz line. This shows the effective CPU frequency, calculating by dividing the true number of CPU cycles by the task-clock value (CPU time spent by the program). If this varies from run to run, the wall-clock time performance deviation is partly or completely explained by this change, and the most likely cause is (1) above, i.e., CPU frequency scaling, including both scaling below nominal frequency (power saving), and above (turbo boost or similar features).

The details of disabling frequency scaling depends on the hardware, but a common one that works on most modern Linux distributions is cpupower -c all frequency-set -g performance to inhibit below-nominal scaling.

Disabling turbo boost is more complicated and may depend on the hardware platform and even the specific CPU, but for recent x86 some options include:

Writing 0 to /sys/devices/system/cpu/intel_pstate/no_turbo (Intel only)
Doing a wrmsr -p${core} 0x1a0 0x4000850089 for each ${core} in your system (although one on each socket is probably enough on some/most/all chips?). (Intel only)
Adjust the /sys/devices/system/cpu/cpu0/cpufreq/scaling_max_freq value to set a maximum frequency.
Use the userspace governor and /sys/devices/system/cpu/cpu0/cpufreq/scaling_setspeed to set a fixed frequency.

Another option is to simply run your test repeatedly, and hope that the CPU quickly reaches a steady state. perf stat has built-in support for that with the --repeat=N option:

   -r, --repeat=<n>
       repeat command and print average + stddev (max: 100). 0 means forever.

Let's say you observe that the frequency is always the same (within 1% or so), or you have fixed the frequency issues but some variance remains.

Next, check the instructions line. This is a rough indicator of how much total work your program is doing. If it varies in the same direction and similar relative variance to your runtime variance, you have a problem of type (2): some runs are doing more work than others. Without knowing what your program is, it would be hard to say more, but you can use tools like strace, perf record + perf annotate to track that down.

If instructions doesn't vary, and frequency is fixed, but runtime varies, you have a problem of type (3) or "other". You'll want to look at more performance counters to see which correlate with the slower runs: are you having more cache misses? More context switches? More branch mispredictions? The list goes on. Once you find out what is slowing you down, you can try to isolate the code that is causing it. You can also go the other direction: using traditional profiling to determine what part of the code slows down on the slow runs.

Good luck!

Is it possible to guarantee a thread wakes up and runs every one second?

Depends what you mean by "exactly." No clock can ever be "exact." and sleep(1) only guarantees to sleep for at least one second. It could sleep longer, but on modern Linux, probably only a tiny bit longer.

If what you really want is, for the long-term average number of iterations to be one per second, then you need to do this (pseudo-code because I don't have access to a real computer right now):

dueDate = get_current_time();
while (true) {
    sleepInterval = dueDate - get_current_time();
    if (sleepInterval > 0) {
        sleep(sleepInterval);
    }
    doWhateverNeedsToBeDone();
    dueDate += oneSecond;
}

This prevents errors in the timing from accumulating. The sleep() call might not wake at exactly the right time, but the next dueDate always is incremented by exactly one second.

It helps if your get_current_time()-like function and your sleep()-like function both work on a fine grain. Linux has clocks that return time in nanoseconds, and it has the ability to sleep for a given number of nanoseconds, but I forget the actual names of those functions. You can find them easily enough in Google.

If that trick doesn't give you enough accuracy using regular Linux system calls, then you may need to run a "real-time" enabled version of Linux, and use priviliged system calls to enable real-time scheduling for your process.

C/C++ - Looking for Realtime Timing Events under Linux

It almost doesn't matter what you are going to use — a select(), epoll(), usleep(), settimer(). Some methods can "take" a higher-resolution timeout, some can't. But Linux is not a real-time OS and should not even expect it to wake up your process at precisely that time. Now, even if you spin the CPU, the kernel can easily suspend your process shall it decide to allocate that CPU to some other process or an interrupt handler.

From the API perspective (as well as recent trends), you should probably stick with timerfd() — you can tell it what clock to use, and it integrates nicely with the rest of Linux event dispatching mechanisms (epoll, AIO, etc).

Other than that, its outside your control as a programmer. A great effort is usually needed in order to get near-real-time behavior from Linux. It ranges from building a custom patched kernel and configuring affinity for the entire world including shielding of interrupts, to patching your BIOS and tuning the hardware.

As it is, assuming you run a fresh vanilla kernel on a commodity x86_64 server which is not highly loaded, expect your wake-up time to be somewhere around 50 µs (you can easily benchmark it to get a rough idea for your system).

Which function is called for waking up a cpu on Linux

There isn't a function to wake the CPU. In the idle state, the CPU is waiting for a system-level interrupt. Usually from a driver after a hardware event or from a previously scheduled timer like a cron job.

This answer has more details: https://stackoverflow.com/a/15096339/50177

µs-precision wait in C for linux that does not put program to sleep?

Achieving this low resolutions (less than 1ms) is almost impossible in conventional multitasking operating systems without hardware support, but there is a software technique which could help you. (I've tested it before)

Software delay loop isn't accurate solution because of processes preemption by operating system's scheduler. But you can patch your kernel with RT_PREEMPT and enable it via CONFIG_RT_PREEMPT, now you have a kernel with realtime scheduling support, the realtime kernel let you run a process with realtime priority, the process with realtime priority run until it wants nobody could preempt it, so if you run a delay loop the process will not preempted by operating system so you could created accurate delays with these loops.