How to Percent-Encode Url Parameters in Python

How can I percent-encode URL parameters in Python?

Python 2

From the documentation:

urllib.quote(string[, safe])

Replace special characters in string
using the %xx escape. Letters, digits,
and the characters '_.-' are never
quoted. By default, this function is
intended for quoting the path section
of the URL.The optional safe parameter
specifies additional characters that
should not be quoted — its default
value is '/'

That means passing '' for safe will solve your first issue:

>>> urllib.quote('/test')
'/test'
>>> urllib.quote('/test', safe='')
'%2Ftest'

About the second issue, there is a bug report about it. Apparently it was fixed in Python 3. You can workaround it by encoding as UTF-8 like this:

>>> query = urllib.quote(u"Müller".encode('utf8'))
>>> print urllib.unquote(query).decode('utf8')
Müller

By the way, have a look at urlencode.

Python 3

In Python 3, the function quote has been moved to urllib.parse:

>>> import urllib.parse
>>> print(urllib.parse.quote("Müller".encode('utf8')))
M%C3%BCller
>>> print(urllib.parse.unquote("M%C3%BCller"))
Müller

Percent-encoding a string in Python

There are a number of ways to do this - a few are described here: How to urlencode a querystring in Python?

Basically you're trying to perform "URL encoding". String replace and/or regex are not the best options for this, as you guessed, since there are several builtin options to take care of things for you (such as some non-obvious conversions that need to be handled)

...and don't use the first option shown on that linked page - you probably want to use

urllib.parse.quote_plus(...)

How to prevent python requests from percent encoding my URLs?

It is not good solution but you can use directly string:

r = requests.get(url, params='format=json&key=site:dummy+type:example+group:wheel')

---

BTW:

Code which convert payload to this string

payload = {
    'format': 'json', 
    'key': 'site:dummy+type:example+group:wheel'
}

payload_str = "&".join("%s=%s" % (k,v) for k,v in payload.items())
# 'format=json&key=site:dummy+type:example+group:wheel'

r = requests.get(url, params=payload_str)

---

EDIT (2020):

You can also use urllib.parse.urlencode(...) with parameter safe=':+' to create string without converting chars :+ .

As I know requests also use urllib.parse.urlencode(...) for this but without safe=.

import requests
import urllib.parse

payload = {
    'format': 'json', 
    'key': 'site:dummy+type:example+group:wheel'
}

payload_str = urllib.parse.urlencode(payload, safe=':+')
# 'format=json&key=site:dummy+type:example+group:wheel'

url = 'https://httpbin.org/get'

r = requests.get(url, params=payload_str)

print(r.text)

I used page https://httpbin.org/get to test it.

percent encoding URL with python

Use

urllib.quote_plus(url, safe=':')

Since you don't want the colon encoded you need to specify that when calling urllib.quote():

>>> expected = 'https:%2F%2Fdl.dropbox.com%2Fu%2F94943007%2Ffile.kml'
>>> url = 'https://dl.dropbox.com/u/94943007/file.kml'
>>> urllib.quote(url, safe=':') == expected
True

urllib.quote() takes a keyword argument safe that defaults to / and indicates which characters are considered safe and therefore don't need to be encoded. In your first example you used '' which resulted in the slashes being encoded. The unexpected output you pasted below where the slashes weren't encoded probably was from a previous attempt where you didn't set the keyword argument safe at all.

Overriding the default of '/' and instead excluding the colon with ':' is what finally yields the desired result.

Edit: Additionally, the API calls for spaces to be encoded as plus signs. Therefore urllib.quote_plus() should be used (whose keyword argument safe doesn't default to '/').

How to URL encode in Python 3?

You misread the documentation. You need to do two things:

1. Quote each key and value from your dictionary, and
2. Encode those into a URL

Luckily urllib.parse.urlencode does both those things in a single step, and that's the function you should be using.

from urllib.parse import urlencode, quote_plus

payload = {'username':'administrator', 'password':'xyz'}
result = urlencode(payload, quote_via=quote_plus)
# 'password=xyz&username=administrator'

python: encode a url with percentage signs?

Use urllib.quote() for the url path leaving everything else as is:

from urllib import quote
from urlparse import urlparse, urlunparse

url = "http://www.website.com/search/si/1/doctors/Vancouver, BC"

scheme, netloc, path, params, query, fragment = urlparse(url)
path = quote(path)
print urlunparse((scheme, netloc, path, params, query, fragment))

prints:

http://www.website.com/search/si/1/doctors/Vancouver%2C%20BC

See also:

• How can I normalize a URL in python

python requests module request params encoded url is different with the intended url

You can manually encode your _params to construct your query string and then concatenate it to your _url.

You can use urllib.parse.urlencode[Python-Docs] to convert
your _params dictionary to a percent-encoded ASCII text string. The resulting
string is a series of key=value pairs separated by & characters,
where both key and value are quoted using the quote_via function. By
default, quote_plus() is used to quote the values, which
means spaces are quoted as a + character and / characters are
encoded as %2F, which follows the standard for GET requests
(application/x-www-form-urlencoded). An alternate function that can be
passed as quote_via is quote(), which will encode spaces
as %20 and not encode / characters. For maximum control of what is
quoted, use quote and specify a value for safe.

from urllib.parse import quote_plus, quote, urlencode
import requests

url_template = "http://something/?{}"
_headers = { ... }
_params = {"action": "log", "datetime": "0900 (대한민국 표준시)"}
_url = url_template.format(urlencode(_params, safe="()", quote_via=quote))

response = requests.get(_url, headers=_headers)

How to urlencode a querystring in Python?

You need to pass your parameters into urlencode() as either a mapping (dict), or a sequence of 2-tuples, like:

>>> import urllib
>>> f = { 'eventName' : 'myEvent', 'eventDescription' : 'cool event'}
>>> urllib.urlencode(f)
'eventName=myEvent&eventDescription=cool+event'

Python 3 or above

Use urllib.parse.urlencode:

>>> urllib.parse.urlencode(f)
eventName=myEvent&eventDescription=cool+event

Note that this does not do url encoding in the commonly used sense (look at the output). For that use urllib.parse.quote_plus.

Python percent encoding only certain characters in a URL

How about a.replace('#','%23')?

---

Related Topics