How to Avoid Http Error 429 (Too Many Requests) Python

how to avoid 429 error requests.get() in python?

As per https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/429

you are sending too many requests at once/in short span of time. I recommend using time.sleep(10)

import time

for num in range(len(platform)):
   ....
   ....
   time.sleep(10)

I used 10 seconds, but you have to test it and understand how much time gap is requred. I would also recommend using https://pypi.org/project/retry/ after you figure out the right amount of sleep time.

Python HTTP error 429 (Too Many Requests)

The error is being thrown by the web server that you are making the requests to, almost certainly because you're issuing requests too quickly and they don't like it. It's not because of an error in your code.

Your attempt at fixing it doesn't make much sense -- StringIO allows you to use an in-memory string as if it were a file object. Passing it as a parameter to requests.get isn't really a valid use case -- you should be using requests.get(grab_csv, ... as you were previously, as .get() expects the url parameter to be a string.

I'd consult the documentation for the API your using (if there is any), and slow down your rate of requests to be in line with their limits.

There is a neat Python package (aptly named ratelimit) that lets you decorate your function to enforce the rate limiting: https://pypi.org/project/ratelimit/

Related Topics