Using Ruby with Mechanize to Log into a Website

Using Ruby with Mechanize to log into a website

This is the approach I usually take. It hasn't failed me:

username_field = form.field_with(:name => "user_session[username]")
username_field.value = "whatever_user"
password_field = form.field_with(:name => "user_session[password]")
password_field.value = "whatever_pwd"
form.submit

In Mechanize (Ruby), how to login then scrape?

The agent variable retains the session and cookies.

So you first do your login, as you did, and then you write agent.get(---your-pdf-link-here--).

In your example code is a small error: the result of the submit is in search_results and then you continue to use page to search for the links?

So in your case, I guess it should look like (untested of course) :

# step 1, login:
agent = Mechanize.new
agent.pluggable_parser.pdf = Mechanize::FileSaver

page = agent.get("http://elwatan.com/sso/inscription/inscription_payant.php")

form = page.form_with(:id => 'form-login-page')
form.login = "my_mail"
form.password = "my_pasword"
page = form.submit

# step 2, get the PDF:
page.parser.xpath('//th/a').each do |link|
  agent.get link['href']
end

Using a Ruby script to login to a website via https

Mechanize

Mechanize is ruby library which imititates the behaviour of a web browser. You can click links, fill out forms und submit them. It even has a history and remebers cookies. It seems your problem could be easily solved with the help of mechanize.

The following example is taken from http://docs.seattlerb.org/mechanize/EXAMPLES_rdoc.html:

require 'rubygems'
require 'mechanize'

a = Mechanize.new
a.get('http://rubyforge.org/') do |page|
  # Click the login link
  login_page = a.click(page.link_with(:text => /Log In/))

  # Submit the login form
  my_page = login_page.form_with(:action => '/account/login.php') do |f|
    f.form_loginname  = ARGV[0]
    f.form_pw         = ARGV[1]
  end.click_button

  my_page.links.each do |link|
    text = link.text.strip
    next unless text.length > 0
    puts text
  end
end

---

Related Topics