How to Set Max_Retries for Requests.Request

Can I set max_retries for requests.request?

It is the underlying urllib3 library that does the retrying. To set a different maximum retry count, use alternative transport adapters:

from requests.adapters import HTTPAdapter

s = requests.Session()
s.mount('http://stackoverflow.com', HTTPAdapter(max_retries=5))

The max_retries argument takes an integer or a Retry() object; the latter gives you fine-grained control over what kinds of failures are retried (an integer value is turned into a Retry() instance which only handles connection failures; errors after a connection is made are by default not handled as these could lead to side-effects).

---

Old answer, predating the release of requests 1.2.1:

The requests library doesn't really make this configurable, nor does it intend to (see this pull request). Currently (requests 1.1), the retries count is set to 0. If you really want to set it to a higher value, you'll have to set this globally:

import requests

requests.adapters.DEFAULT_RETRIES = 5

This constant is not documented; use it at your own peril as future releases could change how this is handled.

Update: and this did change; in version 1.2.1 the option to set the max_retries parameter on the HTTPAdapter() class was added, so that now you have to use alternative transport adapters, see above. The monkey-patch approach no longer works, unless you also patch the HTTPAdapter.__init__() defaults (very much not recommended).

Cleanly setting max_retries on Python requests get or post method

A quick search of the python-requests docs will reveal exactly how to set max_retries when using a Session.

To pull the code directly from the documentation:

import requests
s = requests.Session()
a = requests.adapters.HTTPAdapter(max_retries=3)
b = requests.adapters.HTTPAdapter(max_retries=3)
s.mount('http://', a)
s.mount('https://', b)
s.get(url)

---

What you're looking for, however, is not configurable for several reasons:

1. Requests no longer provides a means for configuration

2. The number of retries is specific to the adapter being used, not to the session or the particular request.

3. If one request needs one particular maximum number of requests, that should be sufficient for a different request.

This change was introduced in requests 1.0 over a year ago. We kept it for 2.0 purposefully because it makes the most sense. We also will not be introducing a parameter to configure the maximum number of retries or anything else, in case you were thinking of asking.

---

Edit Using a similar method you can achieve a much finer control over how retries work. You can read this to get a good feel for it. In short, you'll need to import the Retry class from urllib3 (see below) and tell it how to behave. We pass that on to urllib3 and you will have a better set of options to deal with retries.

from requests.packages.urllib3 import Retry
import requests

# Create a session
s = requests.Session()

# Define your retries for http and https urls
http_retries = Retry(...)
https_retries = Retry(...)

# Create adapters with the retry logic for each
http = requests.adapters.HTTPAdapter(max_retries=http_retries)
https = requests.adapters.HTTPAdapter(max_retries=https_retries)

# Replace the session's original adapters
s.mount('http://', http)
s.mount('https://', https)

# Start using the session
s.get(url)

How to implement retry with requests library

It looks like you're using the API incorrectly: https://api.hypixel.net/#section/Authentication/ApiKey

Try setting the key into the requests.Session headers:

statssession = requests.Session()
statssession.headers["API-Key"] = random.choice([key, key2])

To try again after timeout, you can use a try except block:

for _ in range(5):
    try:
        getstats = statssession.get(
            'https://api.hypixel.net/player',
            params = {'uuid': playeruuid},
            timeout=10).json()
        break
    except requests.exceptions.ReadTimeout:
        continue

Or you can set retries within urllib3, used by requests, as outlined here: Can I set max_retries for requests.request?

How to implement retry mechanism into python requests library?

I was able to obtain the desired level of reliability by extending requests.Session class.

Here is the code https://bitbucket.org/bspeakmon/jira-python/src/a7fca855394402f58507ca4056de87ccdbd6a213/jira/resilientsession.py?at=master

EDIT That code was:

from requests import Session
from requests.exceptions import ConnectionError
import logging
import time

class ResilientSession(Session):

    """
    This class is supposed to retry requests that do return temporary errors.

    At this moment it supports: 502, 503, 504
    """

    def __recoverable(self, error, url, request, counter=1):
        if hasattr(error,'status_code'):
            if error.status_code in [502, 503, 504]:
                error = "HTTP %s" % error.status_code
            else:
                return False
        DELAY = 10 * counter
        logging.warn("Got recoverable error [%s] from %s %s, retry #%s in %ss" % (error, request, url, counter, DELAY))
        time.sleep(DELAY)
        return True

    def get(self, url, **kwargs):
        counter = 0
        while True:
            counter += 1
            try:
                r = super(ResilientSession, self).get(url, **kwargs)
            except ConnectionError as e:
                r = e.message
            if self.__recoverable(r, url, 'GET', counter):
                continue
            return r

    def post(self, url, **kwargs):
        counter = 0
        while True:
            counter += 1
            try:
                r = super(ResilientSession, self).post(url, **kwargs)
            except ConnectionError as e:
                r = e.message
            if self.__recoverable(r, url, 'POST', counter):
                continue
            return r

    def delete(self, url, **kwargs):
        counter = 0
        while True:
            counter += 1
            try:
                r = super(ResilientSession, self).delete(url, **kwargs)
            except ConnectionError as e:
                r = e.message
            if self.__recoverable(r, url, 'DELETE', counter):
                continue
            return r

    def put(self, url, **kwargs):
        counter = 0
        while True:
            counter += 1
            try:
                r = super(ResilientSession, self).put(url, **kwargs)
            except ConnectionError as e:
                r = e.message

            if self.__recoverable(r, url, 'PUT', counter):
                continue
            return r

    def head(self, url, **kwargs):
        counter = 0
        while True:
            counter += 1
            try:
                r = super(ResilientSession, self).head(url, **kwargs)
            except ConnectionError as e:
                r = e.message
            if self.__recoverable(r, url, 'HEAD', counter):
                continue
            return r

    def patch(self, url, **kwargs):
        counter = 0
        while True:
            counter += 1
            try:
                r = super(ResilientSession, self).patch(url, **kwargs)
            except ConnectionError as e:
                r = e.message

            if self.__recoverable(r, url, 'PATCH', counter):
                continue
            return r

    def options(self, url, **kwargs):
        counter = 0
        while True:
            counter += 1
            try:
                r = super(ResilientSession, self).options(url, **kwargs)
            except ConnectionError as e:
                r = e.message

            if self.__recoverable(r, url, 'OPTIONS', counter):
                continue
            return r

Implementing retry for requests in Python

you can use urllib3.util.retry module in combination with requests to have something as follow:

from urllib3.util.retry import Retry
import requests
from requests.adapters import HTTPAdapter

def retry_session(retries, session=None, backoff_factor=0.3):
    session = session or requests.Session()
    retry = Retry(
        total=retries,
        read=retries,
        connect=retries,
        backoff_factor=backoff_factor,
        method_whitelist=False,
    )
    adapter = HTTPAdapter(max_retries=retry)
    session.mount('http://', adapter)
    session.mount('https://', adapter)
    return session

Usage:

session = retry_session(retries=5)
session.post(url=endpoint, data=json.dumps(x), headers=headers)

NB: You can also inherit from Retry class and customize the retry behavior and retry intervals.

Get status_code with max_retries setting for requests.head

After a mere two hours of developing the question, the answer took five minutes:

def valid_url(url):
    if (url.lower() == 'none') or (url == ''):
        return False
    try:
        s = requests.Session()
        a = requests.adapters.HTTPAdapter(max_retries=5)
        s.mount(url, a)
        resp = s.head(url)
        return resp.ok
    except requests.exceptions.MissingSchema:
        # If it's missing the schema, run again with schema added
        return valid_url('http://' + url)
    except requests.exceptions.ConnectionError:
        print('Error trying to connect to {}.'.format(url))
        return False

Based on this answer it looks like the head request will be slightly less resource intensive than the get, particularly if the url contains a large amount of data.

The requests.adapters.HTTPAdapter is the built in adaptor for the urllib3 library that underlies the Requests library.

On another note, I'm not sure what the correct term or phrase for what I'm checking here is. A url could still be valid if it returns an error code.

How to configure `requests.get(url)` calls with retries without creating a new session?

This might be a dirty implementation:

import requests
from time import sleep
While True:
    try:
        #requests here
        break
    except:
        sleep(1)

But setting a retry of 3:

import requests
from time import sleep
for i in range(3):
    try:
        #requests here
        break
    except:
        sleep(1)

A good practice:

import requests
from requests.adapters import HTTPAdapter
from requests.packages.urllib3.util.retry import Retry

def requests_retry_session(
    retries=3,
    backoff_factor=0.3,
    status_forcelist=(500, 502, 504),
    session=None,
):
    session = session or requests.Session()
    retry = Retry(
        total=retries,
        read=retries,
        connect=retries,
        backoff_factor=backoff_factor,
        status_forcelist=status_forcelist,
    )
    adapter = HTTPAdapter(max_retries=retry)
    session.mount('http://', adapter)
    session.mount('https://', adapter)
    return session
response = requests_retry_session().get('https://www.example.com/')
print(response.status_code)

s = requests.Session()
s.auth = ('user', 'pass')
s.headers.update({'x-test': 'true'})

response = requests_retry_session(session=s).get(
    'https://www.example.com'
)

• Each retry attempt will create a new Retry object with updated values, so they can be safely reused.
• Documentation is Here.
  
  Note that:

Patching the HTTPAdapter.init() defaults helps you (very much not
recommended).

---

Related Topics