How to Disable the Security Certificate Check in Python Requests

How do I disable the security certificate check in Python requests

From the documentation:

requests can also ignore verifying the SSL certificate if you set
verify to False.

>>> requests.get('https://kennethreitz.com', verify=False)
<Response [200]>

If you're using a third-party module and want to disable the checks, here's a context manager that monkey patches requests and changes it so that verify=False is the default and suppresses the warning.

import warnings
import contextlib

import requests
from urllib3.exceptions import InsecureRequestWarning

old_merge_environment_settings = requests.Session.merge_environment_settings

@contextlib.contextmanager
def no_ssl_verification():
    opened_adapters = set()

    def merge_environment_settings(self, url, proxies, stream, verify, cert):
        # Verification happens only once per connection so we need to close
        # all the opened adapters once we're done. Otherwise, the effects of
        # verify=False persist beyond the end of this context manager.
        opened_adapters.add(self.get_adapter(url))

        settings = old_merge_environment_settings(self, url, proxies, stream, verify, cert)
        settings['verify'] = False

        return settings

    requests.Session.merge_environment_settings = merge_environment_settings

    try:
        with warnings.catch_warnings():
            warnings.simplefilter('ignore', InsecureRequestWarning)
            yield
    finally:
        requests.Session.merge_environment_settings = old_merge_environment_settings

        for adapter in opened_adapters:
            try:
                adapter.close()
            except:
                pass

Here's how you use it:

with no_ssl_verification():
    requests.get('https://wrong.host.badssl.example/')
    print('It works')

    requests.get('https://wrong.host.badssl.example/', verify=True)
    print('Even if you try to force it to')

requests.get('https://wrong.host.badssl.example/', verify=False)
print('It resets back')

session = requests.Session()
session.verify = True

with no_ssl_verification():
    session.get('https://wrong.host.badssl.example/', verify=True)
    print('Works even here')

try:
    requests.get('https://wrong.host.badssl.example/')
except requests.exceptions.SSLError:
    print('It breaks')

try:
    session.get('https://wrong.host.badssl.example/')
except requests.exceptions.SSLError:
    print('It breaks here again')

Note that this code closes all open adapters that handled a patched request once you leave the context manager. This is because requests maintains a per-session connection pool and certificate validation happens only once per connection so unexpected things like this will happen:

>>> import requests
>>> session = requests.Session()
>>> session.get('https://wrong.host.badssl.example/', verify=False)
/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py:857: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning)
<Response [200]>
>>> session.get('https://wrong.host.badssl.example/', verify=True)
/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py:857: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning)
<Response [200]>

Disable Python requests SSL validation for an imported module

Note: This solution is a complete hack.

Short answer: Set the CURL_CA_BUNDLE environment variable to an empty string.

Before:

$ python
import requests
requests.get('http://www.google.com')
<Response [200]>

requests.get('https://www.google.com')
...
File "/usr/local/lib/python2.7/site-packages/requests-2.17.3-py2.7.egg/requests/adapters.py", line 514, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: ("bad handshake: Error([('SSL routines', 'SSL3_GET_SERVER_CERTIFICATE', 'certificate verify failed')],)",)

After:

$ CURL_CA_BUNDLE="" python
import requests
requests.get('http://www.google.com')
<Response [200]>

requests.get('https://www.google.com')
/usr/local/lib/python2.7/site-packages/urllib3-1.21.1-py2.7.egg/urllib3/connectionpool.py:852: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings InsecureRequestWarning)
<Response [200]>

How it works

This solution works because Python requests overwrites the default value for verify from the environment variables CURL_CA_BUNDLE and REQUESTS_CA_BUNDLE, as can be seen here:

if verify is True or verify is None:
verify = (os.environ.get('REQUESTS_CA_BUNDLE') or
    os.environ.get('CURL_CA_BUNDLE'))

The environment variables are meant to specify the path to the certificate file or CA_BUNDLE and are copied into verify. However, by setting CURL_CA_BUNDLE to an empty string, the empty string is copied into verify and in Python, an empty string evaluates to False.

Note that this hack only works with the CURL_CA_BUNDLE environment variable - it does not work with the REQUESTS_CA_BUNDLE. This is because verify is set with the following statement:

verify = (os.environ.get('REQUESTS_CA_BUNDLE') or os.environ.get('CURL_CA_BUNDLE'))

It only works with CURL_CA_BUNDLE because '' or None is not the same as None or '', as can be seen below:

print repr(None or "")
# Prints: ''
print repr("" or None )
# Prints: None

How do I disable the ssl check in python 3.x?

Use urllib.request.urlopen with custom ssl context:

import ssl
import urllib.request

ctx = ssl.create_default_context()
ctx.check_hostname = False
ctx.verify_mode = ssl.CERT_NONE

with urllib.request.urlopen(url_string, context=ctx) as u, \
        open(file_name, 'wb') as f:
    f.write(u.read())

Alternatively, if you use requests library, it could be simpler:

import requests

with open(file_name, 'wb') as f:
    resp = requests.get(url_string, verify=False)
    f.write(resp.content)

Is it safe to disable SSL certificate verification in pythons's requests lib?

The correct answer here is "it depends".

You've given us very little information to go on, so I'm going to make some assumptions and list them below (if any of them do not match, then you should reconsider your choice):

1. You are constantly connecting to the same website in your CRON job
2. You know the website fairly well and are certain that the certificate-related errors are benign
3. You are not sending sensitive data to the website in order to scrape it (such as login and user name)

If that is the situation (which I am guessing it is) then it should be generally harmless. That said, whether or not it is "safe" depends on your definition of that word in the context of two computers talking to each other over the internet.

As others have said, Requests does not attempt to render HTML, parse XML, or execute JavaScript. Because it simply is retrieving your data, then the biggest risk you run is not receiving data that can be verified came from the server you thought it was coming from. If, however, you're using requests in combination with something that does the above, there are a myriad of potential attacks that a malicious man in the middle could use against you.

There are also options that mean you don't have to forgo verification. For example, if the server uses a self-signed certificate, you could get the certificate in PEM format, save it to a file and provide the path to that file to the verify argument instead. Requests would then be able to validate the certificate for you.

So, as I said, it depends.

---

Update based on Albert's replies

So what appears to be happening is that the website in question sends only the leaf certificate which is valid. This website is relying on browser behaviour that currently works like so:

The browser connects to the website and notes that the site does not send it's full certificate chain. It then goes and retrieves the intermediaries, validates them, and completes the connection. Requests, however, uses OpenSSL for validation and OpenSSL does not contain any such behaviour. Since the validation logic is almost entirely in OpenSSL, Requests has no way to emulate a browser in this case.

Further, Security tooling (e.g., SSLLabs) has started counting this configuration against a website's security ranking. It's increasingly the opinion that websites should send the entire chain. If you encounter a website that doesn't, contacting them and informing them of that is the best course forward.

If the website refuses to update their certificate chain, then Requests' users can retrieve the PEM encoded intermediary certificates and stick them in a .pem file which they then provide to the verify parameter. Requests presently only includes Root certificates in its trust store (as every browser does). It will never ship intermediary certificates because there are just too many. So including the intermediaries in a bundle with the root certificate(s) will allow you to verify the website's certificate. OpenSSL will have a PEM encoded file that has each link in the chain and will be able to verify up to the root certificate.

---

Related Topics