How to Download a File Over Http Using Ruby

How do I download a binary file over HTTP?

The simplest way is the platform-specific solution:

 #!/usr/bin/env ruby
`wget http://somedomain.net/flv/sample/sample.flv`

Probably you are searching for:

require 'net/http'
# Must be somedomain.net instead of somedomain.net/, otherwise, it will throw exception.
Net::HTTP.start("somedomain.net") do |http|
    resp = http.get("/flv/sample/sample.flv")
    open("sample.flv", "wb") do |file|
        file.write(resp.body)
    end
end
puts "Done."

Edit: Changed. Thank You.

Edit2: The solution which saves part of a file while downloading:

# instead of http.get
f = open('sample.flv')
begin
    http.request_get('/sample.flv') do |resp|
        resp.read_body do |segment|
            f.write(segment)
        end
    end
ensure
    f.close()
end

How to download via HTTP only piece of big file with ruby

This seems to work when using sockets:

require 'socket'                  
host = "download.thinkbroadband.com"                 
path = "/1GB.zip" # get 1gb sample file
request = "GET #{path} HTTP/1.0\r\n\r\n"
socket = TCPSocket.open(host,80) 
socket.print(request)        

# find beginning of response body
buffer = ""                    
while !buffer.match("\r\n\r\n") do
  buffer += socket.read(1)  
end           

response = socket.read(100) #read first 100 bytes of body
puts response

I'm curious if there is a "ruby way".

Downloading a specific part of a file from a http server

You can do this with any library that lets you set request headers (all curl -r does is set the Range header), which should be pretty much any HTTP library. Net::HTTP, for its part, has a set_range convenience method that takes as arguments either a single Range object (e.g. 60...80) or a start index and length:

require "net/http"
require "uri"

url = URI.parse("http://example.com/foo")
req = Net::HTTP::Get.new(url.path)

req.set_range(60, 20)

res = Net::HTTP.new(url.host, url.port).start do |http|
  http.request(req)
end
puts res.body

How can I download a file from a URL and save it in Rails?

Try this:

require 'open-uri'
open('image.png', 'wb') do |file|
  file << open('http://example.com/image.png').read
end

Download the last parts of a file, using Ruby?

You have create a range request (partial download), here is the info how to do it: How to make an HTTP GET with modified headers?

You'll need the size of the file, so you need another request to fetch only the headers to parse that info, preferably with a HEAD command (or a GET with Range: bytes=0-0).

Rails: How to to download a file from a http and save it into database

Use open-url (in the Ruby stdlib) to grab the files, then use a gem like paperclip to store them in the db as attachments to your models.

UPDATE:

Attachment_fu does not accept the raw bytes, it needs a "file-like" object. Use this example of a LocalFile along with the code below to dump the image into a temp file then send that to your model.

  http = Net::HTTP.new('www.google.com')
  http.start() { |http|
     req = Net::HTTP::Get.new("/intl/en_ALL/images/srpr/logo1w.png")
     response = http.request(req)
     tempfile = Tempfile.new('logo1w.png')
     File.open(tempfile.path,'w') do |f|
       f.write response.body
     end
     attachment = Attachment.new(:uploaded_data => LocalFile.new(tempfile.path))
     attachement.save
  }

How to download a delayed file via HTTP in Ruby?

Here is the modification according to the post and comment:

require 'open-uri'

def http_download(uri, filename)
  bytes_total = nil
  index = 1
  begin
    open(
      uri,
      read_timeout: 500,
      content_length_proc: lambda { |content_length|
        bytes_total = content_length
      },
      progress_proc: lambda { |bytes_transferred|
        if bytes_total
          print("\r#{bytes_transferred} of #{bytes_total} bytes")
        else
          print("\r#{bytes_transferred} bytes (total size unknown)")
        end
      }
    ) do |io|
      # if "application/json" == io.content_type
      if io.is_a? StringIO
        raise " --> Failed, server is processing. Retry the request ##{index}"
      else # Tempfile
        puts "\n--> Succeed, writing to #{filename}"
        File.open(filename, 'w'){|wf| wf.write io.read}
      end
    end
  rescue => e
    puts e
    return if e.is_a? OpenURI::HTTPError # Processing error

    index += 1
    return if index > 10

    sleep index and retry
  end
end

---

Related Topics