Retrying Httpclient Unsuccessful Requests

Retrying HttpClient Unsuccessful Requests

Instead of implementing retry functionality that wraps the HttpClient, consider constructing the HttpClient with a HttpMessageHandler that performs the retry logic internally. For example:

public class RetryHandler : DelegatingHandler
{
    // Strongly consider limiting the number of retries - "retry forever" is
    // probably not the most user friendly way you could respond to "the
    // network cable got pulled out."
    private const int MaxRetries = 3;

    public RetryHandler(HttpMessageHandler innerHandler)
        : base(innerHandler)
    { }

    protected override async Task<HttpResponseMessage> SendAsync(
        HttpRequestMessage request,
        CancellationToken cancellationToken)
    {
        HttpResponseMessage response = null;
        for (int i = 0; i < MaxRetries; i++)
        {
            response = await base.SendAsync(request, cancellationToken);
            if (response.IsSuccessStatusCode) {
                return response;
            }
        }

        return response;
    }
}

public class BusinessLogic
{
    public void FetchSomeThingsSynchronously()
    {
        // ...

        // Consider abstracting this construction to a factory or IoC container
        using (var client = new HttpClient(new RetryHandler(new HttpClientHandler())))
        {
            myResult = client.PostAsync(yourUri, yourHttpContent).Result;
        }

        // ...
    }
}

Retrying C# HttpClient Unsuccessful Requests and Timeouts

You may just want to subclass (or wrap) HttpClient; it seems cleaner to me to retry the requests at the HttpClient level rather than at the handler level. If this is not palatable, then you'll need to split up the "timeout" values.

Since your handler is actually doing multiple results in one, the HttpClient.Timeout applies to the entire process, including retries. You could add another timeout value to your handler which would be the per-request timeout, and use that with a linked cancellation token source:

public class RetryDelegatingHandler : DelegatingHandler {
  public TimmeSpan PerRequestTimeout { get; set; }
  ...
  protected override async Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
  {
    var cts = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken);
    cts.CancelAfter(PerRequestTimeout);
    var token = cts.Token;
    ...
        responseMessage = await base.SendAsync(request, token);
    ...
  }
}

An elegant way to write a retry for http reponse

Polly Example:

var httpClient = new HttpClient();
var maxRetryAttempts = 3;
var pauseBetweenFailures = TimeSpan.FromSeconds(2);

var retryPolicy = Policy
    .Handle<HttpRequestException>()
    .WaitAndRetryAsync(maxRetryAttempts, i => pauseBetweenFailures);

await retryPolicy.ExecuteAsync(async () =>
{
    var response = await httpClient
      .DeleteAsync("https://example.com/api/products/1");
    response.EnsureSuccessStatusCode();
});

Correct way to retry HttpClient requests with Polly

You are getting a bit too much caught up in Polly and how to configure it, and forgetting a few basic aspects. Don't worry, this is too easy to do!

First, you cannot send the same HttpRequestMessage more than once. See this extensive Q&A on the subject. It's also documented officially, though the documentation is a bit opaque on the reason.

Second, as you have it coded, the request you created was captured once by the lambda, and then reused over and over again.

For your particular case, I would move the creation of the request inside the lambda that you are passing to ExecuteAsync. This gives you a new request each time.

Modifying your code,

var jsonContent =  new StringContent(
    JsonSerializer.Serialize(contentObj),
    Encoding.UTF8,
    "application/json");

var retryPolicy = _policyRegistry.Get<IAsyncPolicy<HttpResponseMessage>>PolicyNames.BasicRetry)
    ?? Policy.NoOpAsync<HttpResponseMessage>();

var context = new Context(
    $"GetSomeData-{Guid.NewGuid()}",
    new Dictionary<string, object>
    {
        { PolicyContextItems.Logger, _logger }
    });

var httpClient = _clientFactory.CreateClient("MySender");

var response = await retryPolicy.ExecuteAsync(ctx =>
{
    var requestMessage = new HttpRequestMessage(HttpMethod.Post, "SendEndpoint")
    {
        Content = jsonContent
    };

    requestMessage.Headers.Authorization = new AuthenticationHeaderValue("Bearer", authToken);
    requestMessage.Headers.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
    httpClient.SendAsync(requestMessage), context);
}

The other captures: logger, authToken, might be OK if they don't change request over request, but you may need to move other variables inside the lambda as well.

Not using Polly at all makes most of this thought process unnecessary, but with Polly, you have to remember retries and policy are occurring across time and context.

Retry the same task multiple times in C# when API returns too many requests error

Is this happening because of the fileStream object? It's in the using statement, so it's not being disposed for sure. Can the stream position after first upload attempt be a problem?

Yes, this is one of your problems. Whenever a stream is read its Position is not reset to 0 automatically. If you try to re-read it then it will not read anything since the position is at the end of the stream.

So, you either have to create a new stream each and every time or rewind the stream to the beginning.

Is it normal that the same Task object is being retried? Should I change the way I'm doing it to something better?

Whenever a Task has been finished (either with a particular result or with an exception) then re-await-ing or retrieving its Result will not trigger a re-execution. It will simple return the value or the exception.

So, you have to create a new Task for each and every retry attempt. In order to do so you could anticipate a Func<Task<T>> in your RunWithRetries

public static T RunWithRetries<T>(Func<Task<T>> issueRequest)
{
    ...
    issueRequest().GetAwaiter().GetResult();

}

From the caller side it would look like this:

RunWithRetries(() => externalAPI.UploadAsync(fileRequest, new FileStream(...)));
//or
RunWithRetries(() => { fileStream.Position = 0; externalAPI.UploadAsync(fileRequest, fileStream); });

When sending requests in a loop, httpclient starts to be slowing

List<Task<string>> alltasks = new List<Task<string>>();

for (int page = 0; Position < 1000; page++)
{
    //int totalpages = await frist1(link);
    Task<String> t =  post_all_pages(page, link);
    alltasks.add(t);
    //var jsons = JsonConvert.DeserializeObject<Json5>(full_json);
    // Далее парсинг данных...
}

Tasks.WaitAll(alltasks.ToArray());

You may want to create a async function that will calls post_all_pages, then await result and then does a JsonDeserialize.

[EDIT 07/18/2020]
Using WhenAll is a better solution:

await Task.WhenAll(alltasks.ToArray());

---

Related Topics