Differencebetween Clock_Monotonic & Clock_Monotonic_Raw

What is the difference between CLOCK_MONOTONIC & CLOCK_MONOTONIC_RAW?

CLOCK_MONOTONIC never experiences discontinuities due to NTP time adjustments, but it does change in frequency as NTP learns what error exists between the local oscillator and the upstream servers.

CLOCK_MONOTONIC_RAW is simply the local oscillator, not disciplined by NTP. This could be very useful if you want to implement some other time synchronization algorithm against a clock which is not fighting you due to NTP. While ntpd (the reference implementation of NTP protocol and the most widespread NTP daemon) is reputed to be "gentle" with time adjustments, it's more accurate to say it's gentle with the absolute time. It's willing to slew the clock by 500ppm which is pretty dramatic if you're in a position to measure your clock frequency against some other standard.

The utility of CLOCK_MONOTONIC_RAW is going to be limited until facilities like pthread_timedwait_monotonic offer an option to use that timebase.

CLOCK_MONOTONIC vs CLOCK_MONOTONIC_RAW truncated values

I think your conclusion that CLOCK_MONOTONIC_RAW is "truncated" is wrong. Rather, the resolution of the hardware clock source is probably just microseconds. The nonzero low digits you're seeing in CLOCK_MONOTONIC are because the timestamps from the hardware clock source are being scaled, per adjustments made via adjtime/NTP, to correct for imprecision in the hardware clock rate that would otherwise make it drift relative to real time.

To test this hypothesis, you should take a large number of timer samples with CLOCK_MONOTONIC and look for a pattern in the low digits. I suspect you'll find that all your timestamps differ by a multiple of some number of nanoseconds close to but not exactly 1000, e.g. maybe 995 or 1005 or so.

Difference between CLOCK_REALTIME and CLOCK_MONOTONIC?

CLOCK_REALTIME represents the machine's best-guess as to the current wall-clock, time-of-day time. As Ignacio and MarkR say, this means that CLOCK_REALTIME can jump forwards and backwards as the system time-of-day clock is changed, including by NTP.

CLOCK_MONOTONIC represents the absolute elapsed wall-clock time since some arbitrary, fixed point in the past. It isn't affected by changes in the system time-of-day clock.

If you want to compute the elapsed time between two events observed on the one machine without an intervening reboot, CLOCK_MONOTONIC is the best option.

Note that on Linux, CLOCK_MONOTONIC does not measure time spent in suspend, although by the POSIX definition it should. You can use the Linux-specific CLOCK_BOOTTIME for a monotonic clock that keeps running during suspend.

Is CLOCK_MONOTONIC process (or thread) specific?

The answer Maxim and comments to that answered the second part of your question, I believe. To expand on the answer for the first part, POSIX 2008 states

If the Monotonic Clock option is supported, all implementations shall support a clock_id of CLOCK_MONOTONIC defined in <time.h>. This clock represents the monotonic clock for the system. For this clock, the value returned by clock_gettime() represents the amount of time (in seconds and nanoseconds) since an unspecified point in the past (for example, system start-up time, or the Epoch). This point does not change after system start-up time.

In particular, note "the monotonic clock for the system". That is, per-system and not per-process, it keeps ticking even though your process is not running. Also, "This point does not change after system start-up time.", which again implies that it keeps ticking regardless of whether a particular process is running or sleeping.

So, either you have found a bug in the Linux implementation, or more likely, in your test program.

Linux clock_gettime(CLOCK_MONOTONIC) strange non-monotonic behavior

man clock_gettime says:

CLOCK_MONOTONIC_RAW (since Linux 2.6.28; Linux-specific)
Similar to CLOCK_MONOTONIC, but provides access to a raw hardware-based time that is not subject to NTP adjustments.

Since CLOCK_MONOTONIC_RAW is not subject of NTP adjustments, I guess CLOCK_MONOTONIC could be.

We had similar problems with Redhat Enterprise 5.0 with 2.6.18 kernel and some specific Itanium processor. We couldn't reproduce it with other processor on the same OS. It was fixed in RHEL 5.3 with slightly newer kernel and some Redhat patches.