How to Get a Faster Output Pipe Than /Dev/Null

Can I get a faster output pipe than /dev/null?

Output to /dev/null is implemented in the kernel, which is pretty bloody fast. The output pipe isn't your problem now, it's the time it takes to build the strings that are getting sent to /dev/null. I would recommend you go through the program and comment out (or guard with if $be_verbose) all the lines that are useless print statements. I'm pretty sure that'll give you a noticeable speedup.

In general, does redirecting outputs to /dev/null enhance performance?

Yes. Outputting to the screen requires a lot of painting. Outputting to a file is much faster, because it's running at disc speed. Outputting to /dev/null is faster yet, because the output goes nowhere.

How to output as fast as possible a fixed buffer?

Well it seems that linux scheduler and IO priorities played had a big role in the slowdown.

Also, spectre and other cpu vunerability mitigations came to play.

After further optimization, to achieve a faster speed I had to tune this things:

1) program nice level (nice -n -20)
2) program ionice level (ionice -c 1 -n 7)
3) pipe size increased 8 times.
4) disable cpu mitigations by adding "pti=off spectre_v2=off l1tf=off" in kernel command line
5) tuning the linux scheduler

echo -n -1 >/proc/sys/kernel/sched_rt_runtime_us
echo -n -1 >/proc/sys/kernel/sched_rt_period_us
echo -n -1 >/proc/sys/kernel/sched_rr_timeslice_ms
echo -n 0 >/proc/sys/kernel/sched_tunable_scaling

Now the program outputs (on the same pc) 8.00 GB/sec!

If you have other ideas you're welcome to contribute.

Will printf still have a cost even if I redirect output to /dev/null?

Pretty much.

When you redirect the stdout of the program to /dev/null, any call to printf(3) will still evaluate all the arguments, and the string formatting process will still take place before calling write(2), which writes the full formatted string to the standard output of the process. It's at the kernel level that the data isn't written to disk, but discarded by the handler associated with the special device /dev/null.

So at the very best, you won't bypass or evade the overhead of evaluating the arguments and passing them to printf, the string formatting job behind printf, and at least one system call to actually write the data, just by redirecting stdout to /dev/null. Well, that's a true difference on Linux. The implementation just returns the number of bytes you wanted to write (specified by the 3rd argument of your call to write(2)) and ignores everything else (see this answer). Depending on the amount of data you're writing, and the speed of the target device (disk or terminal), the difference in performance may vary a lot. On embedded systems, generally speaking, cutting off the disk write by redirecting to /dev/null can save quite some system resources for a non-trivial amount of written data.

Although in theory, the program could detect /dev/null and perform some optimizations within the restrictions of standards they comply to (ISO C and POSIX), based on general understanding of common implementations, they practically don't (i.e. I am unaware of any Unix or Linux system doing so).

The POSIX standard mandates writing to the standard output for any call to printf(3), so it's not standard-conforming to suppress the call to write(2) depending on the associated file descriptors. For more details about POSIX requirements, you can read Damon's answer. Oh, and a quick note: All Linux distros are practically POSIX-compliant, despite not being certified to be so.

Be aware that if you replace printf completely, some side effects may go wrong, for example printf("%d%n", a++, &b). If you really need to suppress the output depending on the program execution environment, consider setting a global flag and wrap up printf to check the flag before printing — it isn't going to slow down the program to an extent where the performance loss is visible, as a single condition check is much faster than calling printf and doing all the string formatting.

How can I redirect all output to /dev/null?

Redirection operators are evaluated left-to-right. You wrongly put 2>&1 first, which points 2 to the same place, as 1 currently is pointed to which is the local terminal screen, because you have not redirected 1 yet. You need to do either of the following:

2>/dev/null 1>/dev/null google-chrome &

Or

2>/dev/null 1>&2 google-chrome &

The placement of the redirect operators in relation to the command does not matter. You can put them before or after the command.



Related Topics



Leave a reply



Submit