Using Www:Mechanize to Download a File to Disk Without Loading It All in Memory First

Using rest-client to download a file to disk without loading it all in memory first

Another way is to use raw_response. This saves directly to a file, usually in /tmp and handles redirects without a problem.
See Streaming Responses. Here's their example:

>> raw = RestClient::Request.execute(
           method: :get,
           url: 'http://releases.ubuntu.com/16.04.2/ubuntu-16.04.2-desktop-amd64.iso',
           raw_response: true)
=> <RestClient::RawResponse @code=200, @file=#<Tempfile:/tmp/rest-client.20170522-5346-1pptjm1>, @request=<RestClient::Request @method="get", @url="http://releases.ubuntu.com/16.04.2/ubuntu-16.04.2-desktop-amd64.iso">>
>> raw.file.size
=> 1554186240
>> raw.file.path
=> "/tmp/rest-client.20170522-5346-1pptjm1"

Save WWW::Mechanize::File to disk using FileUtils

Well, WWW::Mechanize::File has a save_as instance method, so I suppose something like this might work:

agent.get('http://example.com/foo.torrent').save_as 'a_file_name'

Ruby - Working with Mechanize::File response without saving to disk

It seems what you want to do is not possible with rubyzip. From rubyzip library's TODO file:

• SUggestion: ZipInputStream/ZipOutputStream should accept an IO object in addition to a filename.

Can I use Mechanize to download files with .docx and .xlsx,.txt extensions?

1. The type of file doesn't matter; any file accessible over the net can be obtained via mechanize, which is a tool for automating interaction with Mechanize.

2. The file will be stored in the directory where the program was run. Use Mechanize::Download instead of Mechanize::FileSaver to specify where the file should be downloaded to. Example code here: https://stackoverflow.com/a/9105153/429758 (Specify the full path in the filename)

3. Mechanize doesn't use a browser while downloading. For all intents and purposes, Mechanize acts like a web browser with no user interface via http://ruby.about.com/od/tasks/a/The-Mechanize-2-0-Handbook.htm

Do checkout the EXAMPLES page on mechanize documentation for further examples about how to use mechanize.

Confusion with mechanize installation

Mechanize is the main gem, and is used to automate interactions with websites (including downloading files).

mechanize-downloader is a gem for downloading files with a progress bar. It extends Mechanize to provide a progress bar as the files are being downloaded.

Running gem install mechanize-downloader installs Mechanize as well since it is a dependency. Mechanize is a standard, and it's easy to get help should you run into problems.

Take a look at "Using WWW:Mechanize to download a file to disk without loading it all in memory first" for an example of how to download files using Mechanize.

For a more general introduction to working with Mechanize, check out "#191 Mechanize.

"Mechanize examples"" is another useful resource.

Is there a way to programmatically login to a website and download a csv file?

If you're trying to log into websites, then you'll need to use something like Mechanize (tutorial). Once your script is logged in you'll want to use Mechanize::Download - see here: Using WWW:Mechanize to download a file to disk without loading it all in memory first That's assuming you want to store the file locally. If the HTTP request allows you to simply read a URL (like http://example.com/data.csv) then you might be able to read that csv file directly in memory.

---

Related Topics