Python Urllib2 Basic Auth Problem

Python urllib2 Basic Auth Problem

The problem could be that the Python libraries, per HTTP-Standard, first send an unauthenticated request, and then only if it's answered with a 401 retry, are the correct credentials sent. If the Foursquare servers don't do "totally standard authentication" then the libraries won't work.

Try using headers to do authentication:

import urllib2, base64

request = urllib2.Request("http://api.foursquare.com/v1/user")
base64string = base64.b64encode('%s:%s' % (username, password))
request.add_header("Authorization", "Basic %s" % base64string)   
result = urllib2.urlopen(request)

Had the same problem as you and found the solution from this thread: http://forums.shopify.com/categories/9/posts/27662

python urllib2 basic authentication

I could not figure out why this method did not work but I was successful in calling the API using this code:

base64string = base64.encodestring('%s:%s' % (username, password)).replace('\n', '')

request = urllib2.Request(url)

request.add_header("Authorization", "Basic %s" % base64string) 

result = urllib2.urlopen(request)
        data = result.read()

urllib2 basic authentication oddites

About an year ago, I went thro' the same process and documented how I solved the problem - The direct and simple way to authentication and the standard one. Choose what you deem fit.

HTTP Authentication in Python

There is an explained description, in the missing urllib2 document.

Python urllib2, basic HTTP authentication, and tr.im

This seems to work really well (taken from another thread)

import urllib2, base64

request = urllib2.Request("http://api.foursquare.com/v1/user")
base64string = base64.encodestring('%s:%s' % (username, password)).replace('\n', '')
request.add_header("Authorization", "Basic %s" % base64string)   
result = urllib2.urlopen(request)

Basic authentication error with urllib2 since python 2.7 update

You can't have a new line character \n at the end of your header. Instead of using base64.encodestring, use base64.b64encode.

I don't think this has anything to do with an update to Python, since this behaviour has been there since the base64 module was included back in Python 2.4 (see the bolded text):

Encode the string s, which can contain arbitrary binary data, and
return a string containing one or more lines of base64-encoded data.
encodestring() returns a string containing one or more lines of
base64-encoded data always including an extra trailing newline ('\n').

HTTP Basic Authentication not working with Python 3

The solution given here works without any modifications.

from bs4 import BeautifulSoup
import urllib.request

# create a password manager
password_mgr = urllib.request.HTTPPasswordMgrWithDefaultRealm()

# Add the username and password.
# If we knew the realm, we could use it instead of None.
top_level_url = "http://example.com/foo/"
password_mgr.add_password(None, top_level_url, username, password)

handler = urllib.request.HTTPBasicAuthHandler(password_mgr)

# create "opener" (OpenerDirector instance)
opener = urllib.request.build_opener(handler)

# use the opener to fetch a URL
u = opener.open(url)

soup = BeautifulSoup(u.read(), 'html.parser')

The previous code works as well. You just have to decode the utf-8 encoded string otherwise the header contains a byte-sequence.

from bs4 import BeautifulSoup
import urllib.request, base64, urllib.error

request = urllib.request.Request(url)
string = '%s:%s' % ('username','password')

base64string = base64.standard_b64encode(string.encode('utf-8'))

request.add_header("Authorization", "Basic %s" % base64string.decode('utf-8'))
try:
    u = urllib.request.urlopen(request)
except urllib.error.HTTPError as e:
    print(e)
    print(e.headers)

soup = BeautifulSoup(u.read(), 'html.parser')

print(soup.prettify())

urllib2 HTTPPasswordMgr not working - Credentials not sent error

From Python urllib2 Basic Auth Problem

The problem [is] that the Python libraries, per HTTP-Standard, first send an unauthenticated request, and then only if it's answered with a 401 retry, are the correct credentials sent. If the ... servers don't do "totally standard authentication" then the libraries won't work.

This particular API does not respond with a 401 Unauthorized on the first attempt, it responds with an XML response containing the message that credentials were not sent with a 200 OK response code.

---

Related Topics