Authorization Issue with Cron Crawler Inserting Data into Google Spreadsheet Using Google API in Ruby

Authorization issue with cron crawler inserting data into Google spreadsheet using Google API in Ruby

Solved thanks to Ruby google_drive gem oAuth2 saving

I needed to get a refresh token and make my code use it like below.

CLIENT_ID = '!!!'
CLIENT_SECRET = '!!!'
OAUTH_SCOPE = 'https://www.googleapis.com/auth/drive'
REDIRECT_URI = 'urn:ietf:wg:oauth:2.0:oob'
REFRESH_TOKEN = '!!!'

client = Google::APIClient.new
client.authorization.client_id = CLIENT_ID
client.authorization.client_secret = CLIENT_SECRET
client.authorization.scope = OAUTH_SCOPE
client.authorization.redirect_uri = REDIRECT_URI
client.authorization.refresh_token = REFRESH_TOKEN
client.authorization.refresh! 

access_token = client.authorization.access_token
session = GoogleDrive.login_with_oauth(access_token)

want to keep running my single ruby crawler that dont need html and nothing

If your computer is going to be on everyday at 9am, you could schedule it using cron.

If your computer is not going to be on everyday at 9am, you can simply buy a cheap VPS server from digitalocean, linode or one of the many other VPS hosting providers and setup a cron job there just like you did with your local computer.

The cron job would look like this:

0 0 9 1/1 * ? * ruby /path/to/my/ruby/file.rb

You can read more about setting up cron jobs here: https://askubuntu.com/questions/2368/how-do-i-set-up-a-cron-job

Ruby google_drive gem oAuth2 saving

I figured out this mess.

auth.scope =
    "https://www.googleapis.com/auth/drive " +
    "https://spreadsheets.google.com/feeds/"

The auth.scope is missing a url. REFERENCE from github

auth.scope =
    "https://docs.google.com/feeds/" +
    "https://www.googleapis.com/auth/drive " +
    "https://spreadsheets.google.com/feeds/"

You can reuse your access_token, but first you need to get it. WARNING: The auth.access_token is only good for an hour. If you need another one you need to call auth.refresh! This will issue you another access_token.

require 'google/api_client'
require 'google_drive'

client = Google::APIClient.new(
  :application_name => 'Example Ruby application',
  :application_version => '1.0.0'
)
auth = client.authorization
auth.client_id = "YOUR CLIENT ID"
auth.client_secret = "YOUR CLIENT SECRET"
auth.scope =
    "https://www.googleapis.com/auth/drive " +
    "https://spreadsheets.google.com/feeds/"
auth.redirect_uri = "urn:ietf:wg:oauth:2.0:oob"
print("1. Open this page:\n%s\n\n" % auth.authorization_uri)
print("2. Enter the authorization code shown in the page: ")
auth.code = $stdin.gets.chomp
auth.fetch_access_token!
access_token = auth.access_token

system'clear'
print "Save your access token\n\n"
print access_token  
print "\nSave your refresh token\n\n"
print auth.refresh_token 

This chunk of code should print out your access_token/refresh_token in your console. Now you can hard code your application.

require "google/api_client"
require "google_driver"
client = Google::APIClient.new(
  :application_name => 'Example Ruby application',
  :application_version => '1.0.0'
)
auth = client.authorization
auth.client_id = "Your Client ID"
auth.client_secret = "Your client secret"

access_token = "token you saved from the terminal"

session = GoogleDrive.login_with_oauth(access_token)

for file in session.files
   p file.title
end

And eventually your "access_token" will expire. At this point you need to "refresh" it. You can call to see if your access_token is expired by calling

auth.expires_at

You can get a new access token with this example.

require 'google/api_client'
require 'google_drive'

$client_id = "YOUR CLIENT ID"
$client_secret = "YOUR CLIENT SECRET"
$refresh_token = "SAVED REFRESH TOKEN"

 client = Google::APIClient.new(:application_name => 'Example Ruby application', :application_version => '1.0.0')
    auth = client.authorization
    auth.client_id = $client_id
    auth.client_secret = $client_secret
    auth.scope = "https://docs.google.com/feeds/" + "https://www.googleapis.com/auth/drive " + "https://spreadsheets.google.com/feeds/"
    auth.redirect_uri = "urn:ietf:wg:oauth:2.0:oob"
    auth.refresh_token = $refresh_token

    #here's the magic sauce 
    auth.refresh! 

    #as you can see above auth.access_token wasn't passed a value
    # but if you call it now you'll see you have a new access_token

    auth.access_token
    => new token from server 

using variable in xpath on ruby with nokogiri

The problem is with your use of single quotes instead of double quotes.

change this:

title = page.xpath('//span[@class="tit"][#{x}]').inner_html

to this:

title = page.xpath("//span[@class=\"tit\"][#{x}]").inner_html

for proper variable expansion. Note also the escaping of internal double quotes.

---

Related Topics