What Stdout.Sync = True Means

What STDOUT.sync = true means?

Normally puts does not write immediately to STDOUT, but buffers the strings internally and writes the output in bigger chunks. This is done because IO operations are slow and usually it makes more sense to avoid writing every single character immediately to the console.

This behavior leads to problems in certain situations. Imagine you want to build a progress bar (run a loop that outputs single dots between extensive calculations). With buffering the result might be that there isn't any output for a while and then suddenly multiple dots are printed at once.

To avoid this behavior and instead write immediately to STDOUT you can set STDOUT into sync mode like this:

STDOUT.sync = true

From the docs:

When sync mode is true, all output is immediately flushed to the underlying operating system and is not buffered internally.

Why is Ruby STDOUT buffering when I don't expect it to?

As @Ry points out in the comments, $stdout.sync is true by default in IRB, but this is not necessarily the same for scripts.

So you should set $stdout.sync = true to be sure to prevent buffering.

How to write to stdout with Ruby on Bluemix?

You can add the following line to your config.ru file. This will disable buffering to stdout and allows your puts commands to stdout to appear correctly in the log output.

$stdout.sync = true

See the answer at What STDOUT.sync = true means? for more details about how puts buffers.

What is exact meaning of iostream is sync with ios_base::sync_with_stdio

The default ostream used by std::cout and stdio(like printf) is stdout, but it is not necessarily so.

The output could alway be redirected to the other destination. Referencing this: http://www.tldp.org/LDP/abs/html/io-redirection.html

Are async writes to a redirected stdout synchronous or async?

If my program outputs logs to stdout, and it's called in a manner where the output is redirected to a file or piped to another program, will my program stall if the buffer isn't consumed in a timely manner by the OS or pipe recipient?

Yes, it will.

When you call Console.SetOut(), this code executes:

public static void SetOut(TextWriter newOut) {
    if (newOut == null)
        throw new ArgumentNullException("newOut");
    Contract.EndContractBlock();
#pragma warning disable 618
    new SecurityPermission(SecurityPermissionFlag.UnmanagedCode).Demand();
#pragma warning restore 618
#if FEATURE_CODEPAGES_FILE    // if no codepages file then we are locked into default codepage  and this field is not used              
    _isOutTextWriterRedirected = true;
#endif
    newOut = TextWriter.Synchronized(newOut);
    lock(InternalSyncObject) {
        _out = newOut;
    }
}

I.e., the new output TextWriter is always a writer returned by TextWriter.Syncronized(). That in turn always returns a SyncTextWriter object. And if you look at the WriteLineAsync() method for that class, it does two things:

• It is itself a synchronized method, which means if multiple threads attempt to use the writer, only one thread at a time will actually do so (i.e. all methods with that annotation are mutually exclusive between threads).
• More germane to your question though is that the WriteLineAsync() method calls the synchronous WriteLine() method on the underlying TextWriter that was used to create that SyncTextWriter object.

That second point means that the method will not return until the operation has completed. It always returns a completed task. It has no way to return an incomplete task, and if the operation is blocked for any reason, such as that the consumer of the output stream is not consuming it and thus is causing the output buffer to get filled, the call will not return.

This will cause the thread in which you make such a call to block until the write operation can complete, i.e. whatever condition was causing it to block in the first place has been resolved.

Now, all that said:

• If you are writing to a file, whether because you called SetOut() or because the process stdout has been redirected, this may slow the program down depending on how much output your program generates relative to the other work it does, but I doubt you'd experience anything characterized as a "stall". I.e. disk I/O is slow, but it doesn't generally block for extended periods of time.
• If you call SetOut() and pass some other type of writer, presumably you have control over it and can simply make sure it's written correctly and won't block when being written to.
• If the stdout is redirected externally to some other destination, such as a second .NET program running which has started your process and redirected the Process.StandardOutput stream, then it behooves that other program to ensure that it keeps up, if it cares about your program being able to continue unimpeded. But that's always the case anyway; the consumer of a process's output streams should always make sure to read them as quickly as possible.

I don't personally think it's the concern of a program to itself protect itself against redirected streams that might block, other than of course to not redirect the console streams internally to a destination object that might. Usually, this is considered the job of the receiving process, since it's the one that presumably cares whether the redirected program can do its work.

If it was really a concern, you could buffer output yourself within the process, such as having a BlockingCollection<string> object to mediate between threads that want to write to output, and a consumer thread that does the actual writing. But that's not all that useful; it just kicks the can down the road, because you don't want to keep writing to the collection indefinitely, if the consuming thread can't be counted on to be able to proceed unimpeded. If the consumer really gets blocked, the collection will just get too large and throw an OutOfMemoryException at some point. You'll have similar problems with any other mechanism that seeks to delegate the writing to a different thread, with the goal of letting the current thread run without blocking.

IMHO, better to just write normally (i.e. don't even bother using the ...Async methods, unless that's happening because you have code that abstracted the writing as the TextWriter and it's intended to work asynchronously when that's supported), and count on the consuming end of the stream to "do the right thing", and not block your process.

Why doesn't Logger output to STDOUT get redirected to files?

It's a buffering problem, you can set the standard output to sync and this should resolve it.

#!/usr/bin/env ruby

require 'logger'

$stdout.sync = true
logger = Logger.new($stdout)

loop do
  logger.info "This is a test haha"
  sleep 1
end

Ruby's output in the Windows console VS mingw64 VS cygwin64

Here's me answering my own question.

Little background

If you do puts STDOUT.sync before the code then you will see that no matter if you are using Windows console or UNIX-like terminal, it will say that STDOUT.sync is set to false. That's weird thing because Windows console is flushing output immediately and UNIX-like terminals don't. I'm not sure why that's happening.

Solution

You can do STDOUT.flush (or $stdout.flush) to flush buffer or you can set STDOUT.sync (or $stdout.sync) to true. Both variants are completely friendly with Windows console. So the code will be as following:

puts "Dale"
STDOUT.flush
sleep 1
puts "Cooper"

or more recommended:

STDOUT.sync = true
puts "Dale"
sleep 1
puts "Cooper"

Determining whenever it's Windows console or UNIX-like terminal

Here is a little trick suggested by @eryksun to know if code is being runned in Windows console or UNIX-like terminal. STDOUT.isatty works kind of inverted when runned under Windows, but nevertheless it does the trick.

if STDOUT.isatty
    # Windows console
else
    # UNIX-like terminal
end

Keep in mind that this assumes that you already know that the code is being runned under Windows. A good way to check OS is described here.

References

Major source for the answer can be found here. Idea for the answer belongs to @eryksun.

STDOUT.sync,
STDOUT.sync = (question about this method),
STDOUT.flush, STDOUT.isatty.

Line-oriented streaming in Ruby (like grep)

It looks like your best bet is to use STDOUT.syswrite and STDOUT.sysread - the following seemed to have reasonably good performance, despite being ugly code:

STDIN.sync = true
STDOUT.syswrite "Looking for #{ARGV[0]}\n"

def next_line
  mybuff = @overflow || ""
  until mybuff[/\n/]
    mybuff += STDIN.sysread(8)
  end
  overflow = mybuff.split("\n")
  out, *others = overflow
  @overflow = others.join("\n")
  out
rescue EOFError => e
  false  # NB: There's a bug here, see below
end

line = next_line
while line
  STDOUT.syswrite "#{line}\n" if line =~ /#{ARGV[0]}/i
  line = next_line
end

Note: Not sure you need #sync with #sysread, but if so you should probably sync STDOUT too. Also, it reads 8 bytes at a time into mybuff - you should experiment with this value, it's highly inefficient / CPU heavy. Lastly, this code is hacky and needs a refactor, but it works - tested it using ls -l ~/* | ruby rgrep.rb doc (where 'doc' is the search term)

---

Second note: Apparently, I was so busy trying to get it to perform well, I failed to get it to perform correctly! As Dmitry Shevkoplyas has noted, if there is text in @overflow when EOFError is raised, that text will be lost. I believe if you replace the catch with the following, it should fix the problem:

rescue EOFError => e
  return false unless @overflow && @overflow.length > 0
  output = @overflow
  @overflow = ""
  output
end

(if you found that helpful, please upvote Dmitry's answer!)

---

Related Topics