Python Urllib2 Ssl Error

urllib and SSL: CERTIFICATE_VERIFY_FAILED Error

If you just want to bypass verification, you can create a new SSLContext. By default newly created contexts use CERT_NONE.

Be careful with this as stated in section 17.3.7.2.1

When calling the SSLContext constructor directly, CERT_NONE is the default. Since it does not authenticate the other peer, it can be insecure, especially in client mode where most of time you would like to ensure the authenticity of the server you’re talking to. Therefore, when in client mode, it is highly recommended to use CERT_REQUIRED.

But if you just want it to work now for some other reason you can do the following, you'll have to import ssl as well:

input = input.replace("!web ", "")      
url = "https://domainsearch.p.mashape.com/index.php?name=" + input
req = urllib2.Request(url, headers={ 'X-Mashape-Key': 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' })
gcontext = ssl.SSLContext()  # Only for gangstars
info = urllib2.urlopen(req, context=gcontext).read()
Message.Chat.SendMessage ("" + info)

This should get round your problem but you're not really solving any of the issues, but you won't see the [SSL: CERTIFICATE_VERIFY_FAILED] because you now aren't verifying the cert!

To add to the above, if you want to know more about why you are seeing these issues you will want to have a look at PEP 476.

This PEP proposes to enable verification of X509 certificate signatures, as well as hostname verification for Python's HTTP clients by default, subject to opt-out on a per-call basis. This change would be applied to Python 2.7, Python 3.4, and Python 3.5.

There is an advised opt out which isn't dissimilar to my advice above:

import ssl

# This restores the same behavior as before.
context = ssl._create_unverified_context()
urllib.urlopen("https://no-valid-cert", context=context)

It also features a highly discouraged option via monkeypatching which you don't often see in python:

import ssl

ssl._create_default_https_context = ssl._create_unverified_context

Which overrides the default function for context creation with the function to create an unverified context.

Please note with this as stated in the PEP:

This guidance is aimed primarily at system administrators that wish to adopt newer versions of Python that implement this PEP in legacy environments that do not yet support certificate verification on HTTPS connections. For example, an administrator may opt out by adding the monkeypatch above to sitecustomize.py in their Standard Operating Environment for Python. Applications and libraries SHOULD NOT be making this change process wide (except perhaps in response to a system administrator controlled configuration setting).

If you want to read a paper on why not validating certs is bad in software you can find it here!

python ignore certificate validation urllib2

urllib2 does not verify server certificate by default. Check this documentation.

Edit: As pointed out in below comment, this is not true anymore for newer versions (seems like >= 2.7.9) of Python. Refer the below ANSWER

Python Urllib2 SSL error

To summarize the comments about the cause of the problem and explain the real problem in more detail:

If you check the trust chain for the OpenSSL client you get the following:

 [0] 54:7D:B3:AC:BF:... /CN=*.s3.amazonaws.com 
 [1] 5D:EB:8F:33:9E:... /CN=VeriSign Class 3 Secure Server CA - G3
 [2] F4:A8:0A:0C:D1:... /CN=VeriSign Class 3 Public Primary Certification Authority - G5
[OT] A1:DB:63:93:91:... /C=US/O=VeriSign, Inc./OU=Class 3 Public Primary Certification Authority

The first certificate [0] is the leaf certificate sent by the server. The following certifcates [1] and [2] are chain certificates sent by the server. The last certificate [OT] is the trusted root certificate, which is not sent by the server but is in the local storage of trusted CA. Each certificate in the chain is signed by the next one and the last certificate [OT] is trusted, so the trust chain is complete.

If you check the trust chain instead by a browser (e.g. Google Chrome using the NSS library) you get the following chain:

 [0] 54:7D:B3:AC:BF:... /CN=*.s3.amazonaws.com 
 [1] 5D:EB:8F:33:9E:... /CN=VeriSign Class 3 Secure Server CA - G3
[NT] 4E:B6:D5:78:49:... /CN=VeriSign Class 3 Public Primary Certification Authority - G5

Here [0] and [1] are again sent by the server, but [NT] is the trusted root certificate. While this looks from the subject exactly like the chain certificate [2] the fingerprint says that the certificates are different. If you would take a closer looks at the certificates [2] and [NT] you would see, that the public key inside the certificate is the same and thus both [2] and [NT] can be used to verify the signature for [1] and thus can be used to build the trust chain.

This means, that while the server sends the same certificate chain in all cases there are multiple ways to verify the chain up to a trusted root certificate. How this is done depends on the SSL library and on the known trusted root certificates:

                          [0] (*.s3.amazonaws.com)
                           |
                          [1] (Verisign G3) --------------------------\
                           |                                          |
      /------------------ [2] (Verisign G5 F4:A8:0A:0C:D1...)         |
      |                                                               |
      |              certificates sent by server                      |
 .....|...............................................................|................
      |              locally trusted root certificates                |
      |                                                               |
     [OT] Public Primary Certification Authority        [NT] Verisign G5 4E:B6:D5:78:49
     OpenSSL library                                    Google Chrome (NSS library)

But the question remains, why your verification was unsuccessful.
What you did was to take the trusted root certificate used by the browser (Verisign G5 4E:B6:D5:78:49) together with OpenSSL. But the verification in browser (NSS) and OpenSSL work slightly different:

• NSS: build trust chain from certificates send by the server. Stop building the chain when we got a certificate signed by any of the locally trusted root certificates.
• OpenSSL_ build trust chain from the certificates sent by the server. After this is done check if we have a trusted root certificate signing the latest certificate in the chain.

Because of this subtle difference OpenSSL is not able to verify the chain [0],[1],[2] against root certificate [NT], because this certificate does not sign the latest element in chain [2] but instead [1]. If the server would instead only sent a chain of [0],[1] then the verification would succeed.

This is a long known bug and there exist patches and hopefully the issue if finally addressed in OpenSSL 1.0.2 with the introduction of the X509_V_FLAG_TRUSTED_FIRST option.

SSL: CERTIFICATE_VERIFY_FAILED with urllib

There is command line program that you can run on MacOsX that will install the certificates:

sudo /Applications/Python\ 3.6/Install\ Certificates.command

The sudo command will prompt for you password to elevate your privileges.

---

Related Topics