Proxies With Python 'Requests' Module

Proxies with Python 'Requests' module

The proxies' dict syntax is {"protocol": "scheme://ip:port", ...}. With it you can specify different (or the same) proxie(s) for requests using http, https, and ftp protocols:

http_proxy  = "http://10.10.1.10:3128"
https_proxy = "https://10.10.1.11:1080"
ftp_proxy   = "ftp://10.10.1.10:3128"

proxies = { 
              "http"  : http_proxy, 
              "https" : https_proxy, 
              "ftp"   : ftp_proxy
            }

r = requests.get(url, headers=headers, proxies=proxies)

Deduced from the requests documentation:

Parameters:

method – method for the new Request object.

url – URL for the new Request object.

...

proxies – (optional) Dictionary mapping protocol to the URL of the proxy.

...

---

On linux you can also do this via the HTTP_PROXY, HTTPS_PROXY, and FTP_PROXY environment variables:

export HTTP_PROXY=10.10.1.10:3128
export HTTPS_PROXY=10.10.1.11:1080
export FTP_PROXY=10.10.1.10:3128

On Windows:

set http_proxy=10.10.1.10:3128
set https_proxy=10.10.1.11:1080
set ftp_proxy=10.10.1.10:3128

what is the proper way to use proxies with requests in python

Ok is working yea !!! .
The credits for solving this goes to (Olvin Rogh) Thanks Olvin for your help and pointing out my problem. I was adding colon ":" inside the keys
This code is working now.

PROXY = {'https': 'https://143.208.200.26:7878',
         'http': 'http://143.208.200.26:7878'}

with requests.Session() as session:
    session.proxies = PROXY
    r = session.get('http://ip-api.com/json')
    print(json.dumps(r.json(), indent=2))

\n' in proxies in requests module of python

Just follow the traceback and look what's used to parse the proxy-url.

requests uses in the HTTPAdapter.get_connection() urllib3's parse_url method.

However, a regex is used to match the url (with re.match()). The regex end with a $.

From the related python docs:

$

Matches the end of the string or just before the newline at the end of the string, and in MULTILINE mode also matches before a newline. foo matches both ‘foo’ and ‘foobar’, while the regular expression foo$ matches only ‘foo’. More interestingly, searching for foo.$ in 'foo1\nfoo2\n' matches ‘foo2’ normally, but ‘foo1’ in MULTILINE mode; searching for a single $ in 'foo\n' will find two (empty) matches: one just before the newline, and one at the end of the string.

So the regex matches the string up to the end or up to the newline. Hence, the re more or less ignores the last new line.

But if you now have two line breaks in your proxy url, you have a multi-line-string and even if it ignores the last newline, there is still a newline that does not match the regex.

---

Here is a minimal example that has the same behavior:

>>> import re
>>> regex = re.compile(r'^\d$')  # matches exactly one digit
>>> print(re.match(regex, '1'))
<re.Match object; span=(0, 1), match='1'>
>>> print(re.match(regex, '1\n'))
<re.Match object; span=(0, 1), match='1'>
>>> print(re.match(regex, '1\n\n'))
None

---

Related Topics