How to Write (Large) Files with Ruby Eventmachine

What is the best way to read files in an EventMachine-based app?

EM.attach/watch cannot be used on files, as select/epoll on a disk-based file descriptor will always return readable.

Ultimately, it depends on what you're trying to do. If it's a small file, just File.read it. If it is larger, you can read small chunks over time. For example, EM::FileStreamer does this to send large file over the network.

Another common use-case is to tail a file and read in new contents when it changes. This can be achieved using EM.watch_file: http://github.com/jordansissel/eventmachine-tail

High memory consumption downloading large files on Sinatra and Thin

The way streaming in sinatra works under eventmachine is that for each call to
out << chunk sinatra schedules a call in eventmachine to send the chunk. Problem with your code is that it is blocking eventmachines event-loop until the entire file is read and the read is done. So nothing will be sent untill the entire data is in memory.

this could be worked around by doing something like:

get "/" do
s3_object = # large S3 object (not loaded into memory)
  stream :keep_open do |out|
    reader = lambda {
       chunk = s3_object.read
       break if chunk == nil
       out << chunk
       EM::next_tick &reader
    }
    reader.call
  end
end

this will read one chunk as soon as eventmachine is ready without blocking the event loop. Of course in this case s3_object.read need to only return one chunk at the time.

Read file in EventMachine asynchronously

There is no asynchronous file IO support in EventMachine, the best way to achieve what you're trying to do is to read a couple of lines on each tick and send them off to the database. The most important is to not read too large chunks since that would block the reactor.

EM.run do
  io = File.open('path/to/file')
  read_chunk = proc do
    lines_sent = 10
    10.times do
      if line = io.gets
        send_to_db(line) do
          # when the DB call is done
          lines_sent -= 1
          EM.next_tick(read_chunk) if lines_sent == 0
        end
      else
        EM.stop
      end
    end
  end
  EM.next_tick(read_chunk)
end

See What is the best way to read files in an EventMachine-based app?

EventMachine how to write keyboard handler reacting on keypress

If you want to receive unbuffered input from the terminal, you should turn off canonical-mode on standard input. (I also turn off echo to make the screen easier to read.) Add this before your code calls #open_keyboard or within your handler initializer:

require 'termios'
# ...
attributes = Termios.tcgetattr($stdin).dup
attributes.lflag &= ~Termios::ECHO # Optional.
attributes.lflag &= ~Termios::ICANON
Termios::tcsetattr($stdin, Termios::TCSANOW, attributes)

For example:

require 'termios'
require 'eventmachine'

module UnbufferedKeyboardHandler
  def receive_data(buffer)
    puts ">>> #{buffer}"
  end
end

EM.run do
  attributes = Termios.tcgetattr($stdin).dup
  attributes.lflag &= ~Termios::ECHO
  attributes.lflag &= ~Termios::ICANON
  Termios::tcsetattr($stdin, Termios::TCSANOW, attributes)

  EM.open_keyboard(UnbufferedKeyboardHandler)
end

EventMachine how to write keyboard handler reacting on keypress

If you want to receive unbuffered input from the terminal, you should turn off canonical-mode on standard input. (I also turn off echo to make the screen easier to read.) Add this before your code calls #open_keyboard or within your handler initializer:

require 'termios'
# ...
attributes = Termios.tcgetattr($stdin).dup
attributes.lflag &= ~Termios::ECHO # Optional.
attributes.lflag &= ~Termios::ICANON
Termios::tcsetattr($stdin, Termios::TCSANOW, attributes)

For example:

require 'termios'
require 'eventmachine'

module UnbufferedKeyboardHandler
  def receive_data(buffer)
    puts ">>> #{buffer}"
  end
end

EM.run do
  attributes = Termios.tcgetattr($stdin).dup
  attributes.lflag &= ~Termios::ECHO
  attributes.lflag &= ~Termios::ICANON
  Termios::tcsetattr($stdin, Termios::TCSANOW, attributes)

  EM.open_keyboard(UnbufferedKeyboardHandler)
end

---

Related Topics