Throttling Method Calls to M Requests in N Seconds

Throttling method calls to M requests in N seconds

I'd use a ring buffer of timestamps with a fixed size of M. Each time the method is called, you check the oldest entry, and if it's less than N seconds in the past, you execute and add another entry, otherwise you sleep for the time difference.

Throttling method calls using Guava RateLimiter class

You need to call acquire() on the same RateLimiter in every invocation, e.g. by making it available in performOperation():

public class RateLimiterTest {
    public static void main(String[] args) {
        RateLimiter limiter = RateLimiter.create(1.0);
        for (int i = 0; i < 10; i++) {
            performOperation(limiter);
        }
    }

    private static void performOperation(RateLimiter limiter) {
        limiter.acquire();
        System.out.println(new Date() + ": Beep");
    }
}

results in

Fri Aug 07 19:00:10 BST 2015: Beep
Fri Aug 07 19:00:11 BST 2015: Beep

Fri Aug 07 19:00:12 BST 2015: Beep
Fri Aug 07 19:00:13 BST 2015: Beep

Fri Aug 07 19:00:14 BST 2015: Beep
Fri Aug 07 19:00:15 BST 2015: Beep

Fri Aug 07 19:00:16 BST 2015: Beep
Fri Aug 07 19:00:17 BST 2015: Beep

Fri Aug 07 19:00:18 BST 2015: Beep
Fri Aug 07 19:00:19 BST 2015: Beep

Throttle - call a function at most every N milliseconds?

My first reaction would be to use a MailboxProcessor. That's how you generally force all calls to go through a single gateway.

Below is a throttle function that will return an async continuation at most once per timespan. High level, it

1. grabs an invocation requests from the message queue (inbox.Receive). This request contains a channel to return the results.
2. checks if there's any need to delay from the previous run, and sleeps
3. notes the start time of the current invocation (note you could swap this and step 4 if you want to throttle based on the end time of the invocations)
4. triggers the caller (chan.Reply)
5. loops to grab another request

The code is as follows

let createThrottler (delay: TimeSpan) =
  MailboxProcessor.Start(fun inbox ->
    let rec loop (lastCallTime: DateTime option) =
      async {
        let! (chan: AsyncReplyChannel<_>) = inbox.Receive()
        let sleepTime =
          match lastCallTime with
          | None -> 0
          | Some time -> int((time - DateTime.Now + delay).TotalMilliseconds)
        if sleepTime > 0 then
          do! Async.Sleep sleepTime
        let lastCallTime = DateTime.Now
        chan.Reply()
        return! loop(Some lastCallTime)
      }
    loop None)

Then you can use it like this:

[<EntryPoint>]
let main argv =
  // Dummy implementation of callWebServiceAsync
  let sw = Stopwatch.StartNew()
  let callWebServiceAsync i =
    async {
      printfn "Start %d %d" i sw.ElapsedMilliseconds
      do! Async.Sleep(100)
      printfn "End %d %d" i sw.ElapsedMilliseconds
      return i
    }

  // Create a throttler MailboxProcessor and then the throttled function from that.
  let webServiceThrottler = createThrottler (TimeSpan.FromSeconds 1.)
  let callWebServiceAsyncThrottled i =
    async {
       do! webServiceThrottler.PostAndAsyncReply(id)
       return! callWebServiceAsync i
    }

  // Some tests
  Async.Start(async { let! i = callWebServiceAsyncThrottled 0
                      printfn "0 returned %d" i
                      let! i = callWebServiceAsyncThrottled 1
                      printfn "1 returned %d" i
                      let! i = callWebServiceAsyncThrottled 2
                      printfn "2 returned %d" i })
  Async.Start(callWebServiceAsyncThrottled 3 |> Async.Ignore)
  Async.Start(callWebServiceAsyncThrottled 4 |> Async.Ignore)
  Async.Start(callWebServiceAsyncThrottled 5 |> Async.Ignore)
  Async.Start(callWebServiceAsyncThrottled 6 |> Async.Ignore)
  Console.ReadLine() |> ignore
  0

If you run this, you'll see that it throttles your calls to that service as desired, no matter whether you're running in parallel or in series or both.

Throttling an API that 429s after too many requests

This solution is specifically for the BoardGameGeek API.

So instead of doing a bunch of individual requests such as:

/xmlapi2/thing?stats=1&id=188920

/xmlapi2/thing?stats=1&id=174476

You can batch them all into one request like this:

/xmlapi2/thing?stats=1&id=188920,174476

This means that you're only sending off 1 request and will not get rate limited.

I did find out that this still does fail if you attach over ~1200 game ids though. The server responds with 414 Request-URI Too Large.

Here's an example of that error.

If you need more then 1200 then you'll probably have to split the game ids and make multiple requests so that you only request 1200 at a time.

Limit method calls per second(s) (refuse when limit reached)

Have a look at this question. The answer is equally applicable to this problem, even if the question is phrased rather differently.

I confess, I find it slightly mysterious, but it works beautifully.

Any out-of-box methods to implement request throttling in Java?

I can't speak for JBoss but the Oracle Service Bus (optional add on to weblogic) can certainly do throttling if you set up your web service as a proxy/business:

About throttling:

https://blogs.oracle.com/knutvatsendvik/entry/throttling_in_osb

Installing OSB:

http://docs.oracle.com/cd/E14571_01/doc.1111/e15017/apm.htm

Throttle and queue up API requests due to per second cap

I've run into the same issue with various APIs. AWS is famous for throttling as well.

A couple of approaches can be used. You mentioned async.map() function. Have you tried async.queue()? The queue method should allow you to set a solid limit (like 6) and anything over that amount will be placed in the queue.

Another helpful tool is oibackoff. That library will allow you to backoff your request if you get an error back from the server and try again.

It can be useful to wrap the two libraries to make sure both your bases are covered: async.queue to ensure you don't go over the limit, and oibackoff to ensure you get another shot at getting your request in if the server tells you there was an error.

---

Related Topics