Ruby Popen3 -- How to Repeatedly Write to Stdin & Read Stdout Without Re-Opening Process

ruby popen3 -- how to repeatedly write to stdin & read stdout without re-opening process?

You can have some success using expect library, and have the child process to explicitly mark the end of each output, like:

require 'expect'
require 'open3'

Open3.popen3("/bin/bash") do
    | input, output, error, wait_thr |
    input.sync = true
    output.sync = true

    input.puts "ls /tmp"
    input.puts "echo '----'"
    puts output.expect("----", 5)

    input.puts "cal apr 2014"
    input.puts "echo '----'"
    puts output.expect("----", 5)
end

As a bonus, expect has a timeout option.

Ruby: Read large data from stdout and stderr of an external process on Windows

After a lot of different trial and error attempts, I eventually came up with using two threads, one to read from each stream (generator.rb is just a script I wrote to output things to standard out and err):

require 'open3'

data = {}

Open3.popen3("ruby generator.rb") do |stdin, out, err, external|
  # Create a thread to read from each stream
  { :out => out, :err => err }.each do |key, stream|
    Thread.new do
      until (line = stream.gets).nil? do
        data[key] = line
      end
    end
  end

  # Don't exit until the external process is done
  external.join
end

puts data[:out]
puts data[:err]

It simply outputs the last line sent to standard output and error by the calling program, but could obviously be extended to do additional processing (with different logic in each thread). A method I was using before I finally came up with this was resulting in some failures due to race conditions; I don't know if this code is still vulnerable, but I've yet to experience a similar failure.

How to pipe a value into a popen3

Use this:

Open3.popen3(cmd) do |stdin, stdout, stderr, wait|
  stdin.puts("test")
  stdin.close
  # unless (err = stderr.read).empty? then raise err end
  stdout.read
end

Thing to note here is that you need to close (or, maybe flush) the stdin in order for popen3 to feed it to the command. Then, you can read the stdout in a simple manner.

Also, make sure to keep in mind the following excerpt from the docs (Open3#popen3):

You should be careful to avoid deadlocks. Since pipes are fixed length
buffers, ::popen3(“prog”) {|i, o, e, t| o.read } deadlocks if the
program generates too much output on stderr. You should read stdout
and stderr simultaneously (using threads or IO.select). However, if
you don’t need stderr output, you can use ::popen2. If merged stdout
and stderr output is not a problem, you can use ::popen2e. If you
really need stdout and stderr output as separate strings, you can
consider ::capture3.

Access STDIN of child process without capturing STDOUT or STDERR

If you’re on a platform that supports it, you could do this with pipe, fork and exec:

# create a pipe
read_io, write_io = IO.pipe

child = fork do
  # in child

  # close the write end of the pipe
  write_io.close

  # change our stdin to be the read end of the pipe
  STDIN.reopen(read_io)

  # exec the desired command which will keep the stdin just set
  exec 'the_child_process_command'
end

# in parent

# close read end of pipe
read_io.close

# write what we want to the pipe, it will be sent to childs stdin
write_io.write "this will go to child processes stdin"
write_io.close

Process.wait child

IO#select returns IO objects despite EOF -- is this expected?

I don't blame you for being a little confused, the official docs are, shall we say, a little thin on what select is supposed to do.

IO.select is probably just a thin wrapper around the select system call so we'll have a look at that (which is quite well documented). From the Linux man pages:

Those listed in readfds will be watched to see if characters become available
for reading (more precisely, to see if a read will not block; in particular, a
file descriptor is also ready on end-of-file)

Emphasis mine. So, select is more about "will it block" than it is about "are there bytes waiting for me" and an EOF is a non-blocking state so select considers a file descriptor that is in an end-of-file condition to be ready for reading.

---

Related Topics