Sending "User-Agent" Using Requests Library in Python

Sending User-agent using Requests library in Python

The user-agent should be specified as a field in the header.

Here is a list of HTTP header fields, and you'd probably be interested in request-specific fields, which includes User-Agent.

If you're using requests v2.13 and newer

The simplest way to do what you want is to create a dictionary and specify your headers directly, like so:

import requests

url = 'SOME URL'

headers = {
    'User-Agent': 'My User Agent 1.0',
    'From': 'youremail@domain.example'  # This is another valid field
}

response = requests.get(url, headers=headers)

If you're using requests v2.12.x and older

Older versions of requests clobbered default headers, so you'd want to do the following to preserve default headers and then add your own to them.

import requests

url = 'SOME URL'

# Get a copy of the default headers that requests would use
headers = requests.utils.default_headers()

# Update the headers with your custom ones
# You don't have to worry about case-sensitivity with
# the dictionary keys, because default_headers uses a custom
# CaseInsensitiveDict implementation within requests' source code.
headers.update(
    {
        'User-Agent': 'My User Agent 1.0',
    }
)

response = requests.get(url, headers=headers)

How do I change my user-agent using Requests Library

In your header,you even didn't add it to your request.

page = requests.get('http://www.useragentstring.com/',headers=headers)

Now it show me the right user-agent:

How to use Python requests to fake a browser visit a.k.a and generate User Agent?

Provide a User-Agent header:

import requests

url = 'http://www.ichangtou.com/#company:data_000008.html'
headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36'}

response = requests.get(url, headers=headers)
print(response.content)

FYI, here is a list of User-Agent strings for different browsers:

• List of all Browsers

---

As a side note, there is a pretty useful third-party package called fake-useragent that provides a nice abstraction layer over user agents:

fake-useragent

Up to date simple useragent faker with real world database

Demo:

>>> from fake_useragent import UserAgent
>>> ua = UserAgent()
>>> ua.chrome
u'Mozilla/5.0 (Windows NT 6.2; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1667.0 Safari/537.36'
>>> ua.random
u'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.67 Safari/537.36'

how to remove the User-Agent header when send request in python

The requests library is built on top of the urllib3 library. So, when you pass None User-Agent header to the requests's post method, the urllib3 set their own default User-Agent

import requests

r = requests.post("https://httpbin.org/post", headers={
    "User-Agent": None,
})

print(r.json()["headers"]["User-Agent"])

Output

python-urllib3/1.26.7

Here the urllib3 source of connection.py

class HTTPConnection(_HTTPConnection, object):
    ...

    def request(self, method, url, body=None, headers=None):
        if headers is None:
            headers = {}
        else:
            # Avoid modifying the headers passed into .request()
            headers = headers.copy()
        if "user-agent" not in (six.ensure_str(k.lower()) for k in headers):
            headers["User-Agent"] = _get_default_user_agent()
        super(HTTPConnection, self).request(method, url, body=body, headers=headers) 

So, you can monkey patch it to disable default User-Agent header

import requests
from urllib3 import connection

def request(self, method, url, body=None, headers=None):
    if headers is None:
        headers = {}
    else:
        # Avoid modifying the headers passed into .request()
        headers = headers.copy()
    super(connection.HTTPConnection, self).request(method, url, body=body, headers=headers)

connection.HTTPConnection.request = request

r = requests.post("https://httpbin.org/post", headers={
    "User-Agent": None,
})

print(r.json()["headers"])

Output

{
'Accept': '*/*', 
'Accept-Encoding': 'gzip, deflate', 
'Content-Length': '0', 
'Host': 'httpbin.org', 
'X-Amzn-Trace-Id': 'Root=1-61f7b53b-26c4c8f6498c86a24ff05940'
}

Also, consider to provide browser-like User-Agent like this Mozilla/5.0 (Macintosh; Intel Mac OS X 12_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.99 Safari/537.36. Maybe it solves your task with less effort

---

Related Topics