Python Requests Library Redirect New Url

how to get redirect url using python requests

To get the resultant URL after you've been redirected, you can do r.url.

r = requests.get('https://youtu.be/dQw4w9WgXcQ') 
print(r.url) # https://www.youtube.com/watch?v=dQw4w9WgXcQ&feature=youtu.be

r.history is for URLs prior to the final one, so it's only returning your original URL because you were only redirected once.

How to get the final destination URL after redirections in python requests?

As suggested here: Python Requests library redirect new url

You can use the response history to get the final URL. In this case, the final URL will return a 200, however, it will have the "final final" redirect in the HTML. You can parse the final HTML to get the redirectURL.

I would use something like beautifulsoup4 to make parsing very easy - pip install beautifulsoup4

import requests
from bs4 import BeautifulSoup
from urllib.request import unquote
from html import unescape

doi_link  = 'https://doi.org/10.1016/j.artint.2018.07.007'
response = requests.get(url= doi_link ,allow_redirects=True )
for resp in response.history:
     print(resp.status_code, resp.url)

# use final response
# parse html and get final redirect url
soup = BeautifulSoup(response.text, 'html.parser')
redirect_url = soup.find(name="input" ,attrs={"name":"redirectURL"})["value"]

# get final response. unescape and unquote url from the HTML
final_url = unescape(unquote(redirect_url))
print(final_url)
article_resp = requests.get(final_url)

---

python requests library redirect new page to post request

Solved it using selenium webdriver and wait after login or any other GET request.
So that background request completes it's execution and return to the required page.

Whenever you face redirections while Sending any GET request, add SLEEP(seconds) method it can solve many problems

---

Related Topics