How to Http Post Stream Data from Memory in Ruby

Stream the response body of an HTTP GET to an HTTP POST with Ruby

Answering my own question: here's my solution. Note that in order to make this work, I needed to monkey patch Net::HTTP so I could access the socket in order to manually read chunks from the response object. If you have a better solution, I'd still like to see it.

require 'net/http'
require 'excon'

# provide access to the actual socket
class Net::HTTPResponse
  attr_reader :socket
end

source_uri = URI("https://example.org/very_large_file")
target_uri = URI("https://example2.org/rest/resource")

Net::HTTP.start(source_uri.host, source_uri.port, use_ssl: source_uri.scheme == 'https') do |http|
  request = Net::HTTP::Get.new source_uri

  http.request request do |response|
    len = response.content_length
    p "reading #{len} bytes..."
    read_bytes = 0
    chunk = ''

    chunker = lambda do
      begin
        if read_bytes + Excon::CHUNK_SIZE < len
          chunk = response.socket.read(Excon::CHUNK_SIZE).to_s
          read_bytes += chunk.size
        else
          chunk = response.socket.read(len - read_bytes)
          read_bytes += chunk.size
        end
      rescue EOFError
        # ignore eof
      end
      p "read #{read_bytes} bytes"
      chunk
    end

    Excon.ssl_verify_peer = false
    Excon.post(target_uri.to_s, :request_block => chunker)

  end
end

Ruby on Rails 3: Streaming data through Rails to client

It looks like this isn't available in Rails 3

https://rails.lighthouseapp.com/projects/8994/tickets/2546-render-text-proc

This appeared to work for me in my controller:

self.response_body =  proc{ |response, output|
  output.write "Hello world"
}

Ruby: How to post a file via HTTP as multipart/form-data?

I like RestClient. It encapsulates net/http with cool features like multipart form data:

require 'rest_client'
RestClient.post('http://localhost:3000/foo', 
  :name_of_file_param => File.new('/path/to/file'))

It also supports streaming.

gem install rest-client will get you started.

Streaming web uploads to socket with Rack

You are right: the Rack spec mandates rewindable input, which implies buffering. It seems Rack is not the tool for this job.

You may want to try FastCGI, which does indeed allow non-buffered streaming. Or maybe a Java Servlet. My 2¢: Do you really need it? If not, don't worry, disk space is really cheap. If so, do you really need to do it in Ruby?

edit: Mongrel::HTTPRequest does not support unbuffered large streaming inputs (without monkeypatching)

---

Related Topics