How to Throttle My Site's API Users

How do I throttle my site's API users?

Ok, there's no way to do what I asked without any writes to the server, but I can at least eliminate logging every single request. One way is by using the "leaky bucket" throttling method, where it only keeps track of the last request ($last_api_request) and a ratio of the number of requests/limit for the time frame ($minute_throttle). The leaky bucket never resets its counter (unlike the Twitter API's throttle which resets every hour), but if the bucket becomes full (user reached the limit), they must wait n seconds for the bucket to empty a little before they can make another request. In other words it's like a rolling limit: if there are previous requests within the time frame, they are slowly leaking out of the bucket; it only restricts you if you fill the bucket.

This code snippet will calculate a new $minute_throttle value on every request. I specified the minute in $minute_throttle because you can add throttles for any time period, such as hourly, daily, etc... although more than one will quickly start to make it confusing for the users.

$minute = 60;
$minute_limit = 100; # users are limited to 100 requests/minute
$last_api_request = $this->get_last_api_request(); # get from the DB; in epoch seconds
$last_api_diff = time() - $last_api_request; # in seconds
$minute_throttle = $this->get_throttle_minute(); # get from the DB
if ( is_null( $minute_limit ) ) {
    $new_minute_throttle = 0;
} else {
    $new_minute_throttle = $minute_throttle - $last_api_diff;
    $new_minute_throttle = $new_minute_throttle < 0 ? 0 : $new_minute_throttle;
    $new_minute_throttle += $minute / $minute_limit;
    $minute_hits_remaining = floor( ( $minute - $new_minute_throttle ) * $minute_limit / $minute  );
    # can output this value with the request if desired:
    $minute_hits_remaining = $minute_hits_remaining >= 0 ? $minute_hits_remaining : 0;
}

if ( $new_minute_throttle > $minute ) {
    $wait = ceil( $new_minute_throttle - $minute );
    usleep( 250000 );
    throw new My_Exception ( 'The one-minute API limit of ' . $minute_limit 
        . ' requests has been exceeded. Please wait ' . $wait . ' seconds before attempting again.' );
}
# Save the values back to the database.
$this->save_last_api_request( time() );
$this->save_throttle_minute( $new_minute_throttle );

How to check what the throttling limit is for your access to an endpoint with JS

This starts concurency number of workers (I'm using workers as a loose term here; don't @ me). Each one makes as many requests as possible until one of the requests is rate-limited or it runs out of time. It them reports how many of the requests completed successfully inside the given time window.

If you know the rate-limit window (1 minute based on your question), this will find the rate-limit. If you need to discover the window, you would want to intentionally exhaust the limit, then slow down the requests and measure the time until they started going through again. The provided code does not do this.

// call apiCall() a bunch of times, stopping when a apiCall() resolves
// false or when "until" time is reached, whichever comes first. For example
// if your limit is 50 req/min (and you give "until" enough time to
// actuially complete 50+ requests) this will call apiCall() 50 times. Each
// call should return a promise resolving to TRUE, so it will be counted as
// a success. On the 51st call you will presumably hit the limit, the API
// will return an error, apiCall() will detect that, and resolve to false.
// This will cause the worker to stop making requests and return 50.
async function workerThread(apiCall, until) {
    let successfullRequests = 0;
    while(true) {
        const success = await apiCall();
        // only count it if the request was successfull
        // AND finished within the timeframe
        if(success && Date.now() < until) {
            successfullRequests++;
        } else {
            break;
        }
    }
    return successfullRequests;
}

// this just runs a bunch of workerThreads in parallell, since by doing a
// single request at a time, you might not be able to hit the limit
// depending on how slow the API is to return. It returns the sum of each
// workerThread(), AKA the total number of apiCall()s that resolved to TRUE
// across all threads.
async function testLimit(apiCall, concurency, time) {
    const endTime = Date.now() + time;
    
    // launch "concurency" number of requests
    const workers = [];
    while(workers.length < concurency) {
        workers.push(workerThread(apiCall, endTime));
    }
    
    // sum the number of requests that succeded from each worker.
    // this implicitly waits for them to finish.
    let total = 0;
    for(const worker of workers) {
        total += await worker;
    }
    return total;
}

// put in your own code to make a trial API call.
// return true for success or false if you were throttled.
async function yourAPICall() {
    try {
        // this is a really sloppy example API
        // the limit is ROUGHLY 5/min, but because of the sloppy server-side
        // implimentation you might get 4-6.
        const resp = await fetch("https://9072997.com/demos/rate-limit/");
        return resp.ok;
    } catch {
        return false;
    }
}

// this is a demo of how to use the function
(async function() {
    // run 2 requests at a time for 5 seconds
    const limit = await testLimit(yourAPICall, 2, 5*1000);
    console.log("limit is " + limit + " requests in 5 seconds");
})();

Note that this method measures the quota available to itself. If other clients or previous requests have already depleted the quota, it will affect the result.

How to throttle API http request

You need to debounce your method :

let now = Date.now();
const delayQue = () =>{
    if(Date.now() - now > 5000) {
        now = Date.now();
        sendStepData()
    }
}

it will only make the call if 5000ms has passed since the last call,
and it will wait 5000ms before the first call also.

You can also wait for the 5000ms to pass and call the method

let now = Date.now();
let timeoutId = null;
const delayQue = () =>{
    if(Date.now() - now > 5000) {
        now = Date.now();
        sendStepData()
    } else if(!timeoutId) {
        timeoutId = setTimeout(() => {
            now = Date.now();
            sendStepData();
            timeoutId = null,    
        },  5000 - (Date.now() - now))
    }
}

you could look for the lodash debounce method also, if you don't want to do it yourself, and have a more complete solution

Throttle and queue up API requests due to per second cap

I've run into the same issue with various APIs. AWS is famous for throttling as well.

A couple of approaches can be used. You mentioned async.map() function. Have you tried async.queue()? The queue method should allow you to set a solid limit (like 6) and anything over that amount will be placed in the queue.

Another helpful tool is oibackoff. That library will allow you to backoff your request if you get an error back from the server and try again.

It can be useful to wrap the two libraries to make sure both your bases are covered: async.queue to ensure you don't go over the limit, and oibackoff to ensure you get another shot at getting your request in if the server tells you there was an error.

Django rest framework - how can i limit requests to an API endpoint?

To create a custom throttle, override BaseThrottle and implement .allow_request(self, request, view). The method should return True if the request should be allowed, and False otherwise.

Optionally you may also override the .wait() method. If implemented, .wait() should return a recommended number of seconds to wait before attempting the next request, or None. The .wait() method will only be called if .allow_request() has previously returned False.

If the .wait() method is implemented and the request is throttled, then a Retry-After header will be included in the response.

import random
class CustomThrottle(throttling.BaseThrottle):
    def allow_request(self, request, view):
         """
         Return `True` if the request should be allowed, `False` otherwise.
         """
         return random.randint(1, 10) != 1

    def wait(self):
        """
        Optionally, return a recommended number of seconds to wait before
        the next request.
        """
        cu_second = 600
        return cu_second

class UserSecThrottle(CustomThrottle,UserRateThrottle):   # or AnonRateThrottle
    scope = 'user_sec'

class ExampleView(APIView):
    throttle_classes = [UserSecThrottle]
    .......

Reference: https://www.django-rest-framework.org/api-guide/throttling/#custom-throttles

Whose responsibility is it to throttle web requests?

The concept of a library is to give its client code as little to worry about as possible. Therefore I would make it the libraries job to queue requests and return results in a timely manner. In an ideal world you would use a callback or delegate model so that the client code can operate in asynchronously, not blocking the UI. You could also offer the option for skipping the queue, (and failing if it operates too soon) and possibly even offer priorities within the queue model.

I also believe it is the responsibility of the library author to default to being a good citizen, and for the library's default operation to be to comply to the conditions of the data provider.