Passing Csrftoken with Python Requests

Passing csrftoken with python Requests

If you are going to set the referrer header, then for that specific site you need to set the referrer to the same URL as the login page:

import sys
import requests

URL = 'https://portal.bitcasa.com/login'

client = requests.session()

# Retrieve the CSRF token first
client.get(URL)  # sets cookie
if 'csrftoken' in client.cookies:
    # Django 1.6 and up
    csrftoken = client.cookies['csrftoken']
else:
    # older versions
    csrftoken = client.cookies['csrf']

login_data = dict(username=EMAIL, password=PASSWORD, csrfmiddlewaretoken=csrftoken, next='/')
r = client.post(URL, data=login_data, headers=dict(Referer=URL))

When using unsecured http, the Referer header is often filtered out and otherwise easily spoofable anyway, so most sites no longer require the header to be set. However, when using an SSL connection and if it is set, it does make sense for the site to validate that it at least references something that could logically have initiated the request. Django does this when the connection is encrypted (uses https://), and actively requires it then.

Get CSRF token using python requests

See the following code example. You can use it directly to login into a website that only uses cookies to store login information.

import requests

LOGIN_URL = 'https://examplenotarealpage.com'
headers = {
    'accept': 'text/html,application/xhtml+xml,application/xml',
    'user-agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36'
}

response = requests.get(LOGIN_URL, headers=headers, verify=False)

headers['cookie'] = '; '.join([x.name + '=' + x.value for x in response.cookies])
headers['content-type'] = 'application/x-www-form-urlencoded'
payload = {
    'username': 'user_name',
    'password': 'randompass123'
}

response = requests.post(LOGIN_URL, data=payload, headers=headers, verify=False)
headers['cookie'] = '; '.join([x.name + '=' + x.value for x in response.cookies])

There are a few possible locations of the CSRF token. Different websites use different ways to pass it to browser. Here are some of them:

• It can come with response headers, in that case getting it is easy.

• Sometimes page meta holds the CSRF token. You have to parse the html content of the page to get it. Find the proper CSS selector for it. See an example:

  from bs4 import BeautifulSoup
  soup = BeautifulSoup(response.text, 'lxml')
  csrf_token = soup.select_one('meta[name="csrf-token"]')['content']
  

• It can be inside of a script tag with JavaScript code. Getting it will be tricky. But, you can always use regex to isolate it.

Login with python requests and csrf-token

It's worth noting that your request, when using the Python Requests module, will not be the exact same as a standard user request. In order to fully mimic a realistic request, and thus not be blocked by any firewall or security measures by the site, you will need to copy both all POST parameters, GET parameters and finally headers.

You can use a tool such as Burp Suite to intercept the login request. Copy the URL it is sending it to, copy all POST parameters also, and finally copy all headers. You should be using the requests.Session() function in order to store cookies. You may also want to do a initial session GET request to the homepage in order to pick up cookies as it is not realistic for a user to send a login request without first visiting the homepage.

I hope that makes sense, header parameters can be passed like so:

import requests

headers = {
    'User-Agent': 'My User Agent (copy your real one for a realistic request).'
}

data = {
    'username': 'John',
    'password': 'Doe'
}

s = requests.Session()
s.get("https://mywebsite.com/")
s.post("https://mywebsite.com/", data=data, headers=headers)

How do I pass a CSRF token using the python-requests library?

You can either send the CSRF token as a POST parameter or a HTTP header.

Edit: a Referer HTTP header is also required by Django's CSRF protection. It needs to have the same origin as the request.

Using POST parameters:

post_data = {'email': email, 'answer': answer, 'csrftoken': crsf_token_value}
headers = {'Referer': URL}
response = request.post(URL, data=post_data, headers=headers)

Using HTTP headers:

post_data = {'email': email, 'answer': answer}
headers = {'X-CSRFToken': csrf_token_value, 'Referer': URL}
response = request.post(URL, data=post_data, headers=headers)

Scrape with requests page with CSRF token

_csrf is provided on that page as a form parameter you would have to parse out with something like BeautifulSoup.

<input type="hidden" name="_csrf" value="666363ca-ffff-ffff-ffff-41a61e158e0f" />

---

Related Topics