Python Requests - How to Use System Ca-Certificates (Debian/Ubuntu)

Python Requests - How to use system ca-certificates (debian/ubuntu)?

From https://stackoverflow.com/a/33717517/1695680

To make python requests use the system ca-certificates bundle, it needs to be told to use it over its own embedded bundle

export REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt

---

Requests embeds its bundles here, for reference:

/usr/local/lib/python2.7/site-packages/requests/cacert.pem
/usr/lib/python3/dist-packages/requests/cacert.pem

Or in newer versions use additional package to obtain certificates from:
https://github.com/certifi/python-certifi

To verify from which file certificates are loaded, you can try:

Python 3.8.5 (default, Jul 28 2020, 12:59:40) 
>>> import certifi
>>> certifi.where()
'/etc/ssl/certs/ca-certificates.crt'

Why python requests not use the system ssl cert by default?

I find the root cause, the requests in fact will call where using from certifi import where to find the proper CA on this pc.

But, there are 2 kinds of ways to install certifi module in ubuntu:

Option 1:

apt-get install -y python3-certifi

Option 2:

pip3 install certifi

NOTE: if directly using pip3 install requests, it will implicitly install certifi using pip3 if no debian package installed for python3-certifi.

However, looks Canonical (the backers of Ubuntu) made some changes for certifi, so if using python3-certifi from apt, the code of def where is next or similar varies from different versions:

root@4e1aab76e082:/# cat /usr/lib/python3/dist-packages/certifi/core.py
def where():
    return "/etc/ssl/certs/ca-certificates.crt"

But if using pip3 to install, it will be:

root@4e1aab76e082:/# cat /usr/local/lib/python3.6/dist-packages/certifi/core.py
def where():
    _CACERT_CTX = get_path("certifi", "cacert.pem")

That's /usr/local/lib/python3.6/dist-packages/certifi/cacert.pem in this package.

So the solution on ubuntu is: use apt-get install python3-certifi to install the ubuntu variant of certifi, uninstall the pip one. Then, we can no change anything of app code.

UPDATE:

I find another way which works with official certifi, using next variable, I could also let python requests module go to fetch the CA from where I specified without change app code:

export REQUESTS_CA_BUNDLE='/etc/ssl/certs/ca-certificates.crt'

How to get Python requests to trust a self signed SSL certificate?

try:

r = requests.post(url, data=data, verify='/path/to/public_key.pem')

Why Python isn't loading the ca certificate?

requests does not use the defaults from ssl; it uses envvar REQUESTS_CA_BUNDLE or CURL_CA_BUNDLE if set and otherwise uses the (spunoff) certifi module which depending on how you installed requests&certifi which you didn't say and your environment which you didn't identify might use a system default (which might or might not be the same as the OpenSSL used in ssl) or might be its own copy of Mozilla. In the latter case it should include Digicert Global Root CA as needed for that site, because Firefox (also Mozilla) does.

Look at requests.certs.where() or python -m requests.certs

Mostly dupe

Python Requests - How to use system ca-certificates (debian/ubuntu)?

How to force requests use the certificates on my ubuntu system

Why python requests not use the system ssl cert by default?

can connect to URL with curl but not with requests (i.e. requests ignoring my CA bundle?)

i tried your stuff on my system (Manjaro Linux, python 3.10) i can make a connection. I downloaded the complete certificate chain from the website (with my browser). After that i can use it with:

r = requests.get(url=URL, verify=<path to pem file>)

and with

export REQUESTS_CA_BUNDLE=<path to pem>

r = requests.get(url=URL)

I tried the export within pyCharm.

So the python stuff is working and you may have a problem in your certificates. Without this stuff i get the ssl error (of course), because python does not use the system certs as you mentioned correct. In my pem-file i have 3 certificates. Maybe you have only 1 and the others are in the global store, so that curl does not need the complete chain, instead of python. You should try to download the complete chain with your browser and try again.

---

Related Topics