Ruby Pipes: How to Tie the Output of Two Subprocesses Together

ruby - IO.popen not working lame stdin and stdout encoding

Did you try the pipe | character?

Tested this on windows with ruby installer

require 'open3'

command = 'dir /B | sort /R'  # a windows example command
Open3.popen3(command) {|stdin, stdout, stderr, wait_thr|
  pid = wait_thr.pid
  puts stdout.read  #<a list of files in cwd in reverse order>
}

Other ways: Ruby pipes: How do I tie the output of two subprocesses together?

EDIT:
using IO::pipe

require 'open3'

command1 = 'dir /B'
command2 = 'sort /R'

reader,writer = IO.pipe
Open3.popen3(command1) {|stdin, stdout, stderr, wait_thr|
  writer.write stdout.read
}
writer.close

stdout, stderr, status = Open3.capture3(command2, :stdin_data => reader.read)
reader.close

puts "status: #{status}"   #pid and exit code
puts "stderr: #{stderr}"   #use this to debug command2 errors
puts stdout

Embedding the two also appears to work, yet, as the blog you referred to said, one must wait for the first command to finish (not real-time -- test with a ping command)

stdout2 = ''
Open3.popen3(command1) {|stdin, stdout, stderr, wait_thr|
  stdout2, stderr2, status2 = Open3.capture3(command2, :stdin_data => stdout.read)
}
puts stdout2

Ruby spawn process, capturing STDOUT/STDERR, while behaving as if it were spawned regularly

[edited: see the bottom for the amended update]

Figured it out :)

To really understand the problem I read up a lot on how a PTY works. I don't think I really understood it properly until I drew it out. Basically PTY could be used for a Terminal emulator, and that was the simplest way to think of the data flow for it:

keyboard -> OS -> terminal -> master pty -> termios -> slave pty -> shell
                                               |
                                               v
 monitor <- OS <- terminal <- master pty <- termios

(note: this might not be 100% correct, I'm definitely no expert on the subject, just posting it incase it helps anybody else understand it)

So the important bit in the diagram that I hadn't really realised was that when you type, the only reason you see your input on screen is because it's passed back (left-wards) to the master.

So first thing's first - this ruby script should first set the tty to raw (IO.console.raw!), it can restore it after execution is finished (IO.console.cooked!). This'll make sure the keyboard inputs aren't printed by this parent Ruby script.

Second thing is the slave itself should not be raw, so the slave.raw! call is removed. To explain this, I originally added this because it removes extra return carriages from the output: running echo hello results in "hello\r\n". What I missed was that this return carriage is a key instruction to the terminal emulator (whoops).

Third thing, the process should only be talking to the slave. Passing STDIN felt convenient, but it upsets the flow shown in the diagram.

This brings up a new problem on how to pass user input through, so I tried this. So we basically pass STDIN to the master:

  input_thread = Thread.new do
    STDIN.each_char do |char|
      master.putc(char) rescue nil
    end
  end

that kind of worked, but it has its own issues in terms of some interactive processes weren't receiving a key some of the time. Time will tell, but using IO.copy_stream instead appears to solve that issue (and reads much nicer of course).

input_thread = Thread.new { IO.copy_stream(STDIN, master) }

---

update 21st Aug:

So the above example mostly worked, but for some reason keys like CTRL+c still wouldn't behave correctly. I even looked up other people's approach to see what I could be doing wrong, and effectively it seemed the same approach - as IO.copy_stream(STDIN, master) was successfully sending 3 to the master. None of the following seemed to help at all:

master.putc 3
master.putc "\x03"
master.putc "\003"

Before I went and delved into trying to achieve this in a lower level language I tried out 1 more thing - the block syntax. Apparently the block syntax magically fixes this problem.

To prevent this answer getting a bit too verbose, the following appears to work:

require 'pty'
require 'io/console'

def run
  output = ""

  IO.console.raw!

  input_thread = nil

  PTY.spawn('bash') do |read, write, pid|
    Signal.trap(:WINCH) { write.winsize = STDOUT.winsize }
    input_thread = Thread.new { IO.copy_stream(STDIN, write) }

    read.each_char do |char|
      STDOUT.print char
      output.concat(char)
    end

    Process.wait(pid)
  end

  input_thread.kill if input_thread

  IO.console.cooked!
end

Bundler.send(:with_env, Bundler.clean_env) do
  run
end

Open another terminal to display information

Okay, I don't know what operating system you're using or what's your console of choice, but on my Ubuntu 12.10 installation this opened up an xterm window with my results:

a = "Hello World!"

system(%Q[xterm -hold -e /bin/bash -l -c "echo #{a}"])

As for an explanation:

• -hold prevents the window from closing as soon as the program executes.
• -e specifies what program and its arguments are to be run (the /bin/bash ... part.).
• /bin/bash invokes the unix shell bash
• -l makes bash act as if it had been invoked as a login shell
• -c will receive console output
• echo displays a message on the screen.

Check the man pages for xterm and bash for more information.

If you add more details, perhaps we can clarify a better answer although, truthfully, I prefer Sergio's answer of outputting to a file and having another console tailing that file.

How to test output from a subprocess with RSpec

expect { `echo 1` }.to output("1").to_stdout

doesn't work for two reasons:

1. echo runs in a subprocess. RSpec's output matcher doesn't handle output from subprocesses by default. But you can use to_stdout_from_any_process instead of to_stdout to handle subprocesses, although it's a bit slower.

2. output only works for output sent to the same standard output stream as the Ruby process. Backticks open a new standard output stream, send the command's standard output to it and return the contents of the stream when the command completes. I don't think you care whether you run your subprocess with backticks or some other way, so just use system (which sends the command's standard output to the Ruby process's standard output stream) instead.

Addressing those two points gives us this expectation, which passes:

expect { system("echo 1") }.to output("1\n").to_stdout_from_any_process

(I had to change the expected value for it to pass, since echo adds a newline.)

As MilesStanfield pointed out, in the case you gave it's equivalent and easier to just test the output of backticks rather than use output:

expect { `echo 1` }.to eq "1\n"

That might or might not work in the more complicated case that you presumably have in mind.

---

Related Topics