Retry a Task Multiple Times Based on User Input in Case of an Exception in Task

Retry a task multiple times based on user input in case of an exception in task

UPDATE 5/2017

C# 6 exception filters make the catch clause a lot simpler :

    private static async Task<T> Retry<T>(Func<T> func, int retryCount)
    {
        while (true)
        {
            try
            {
                var result = await Task.Run(func);
                return result;
            }
            catch when (retryCount-- > 0){}
        }
    }

and a recursive version:

    private static async Task<T> Retry<T>(Func<T> func, int retryCount)
    {
        try
        {
            var result = await Task.Run(func);
            return result;
        }
        catch when (retryCount-- > 0){}
        return await Retry(func, retryCount);
    }

ORIGINAL

There are many ways to code a Retry function: you can use recursion or task iteration. There was a discussion in the Greek .NET User group a while back on the different ways to do exactly this.

If you are using F# you can also use Async constructs. Unfortunately, you can't use the async/await constructs at least in the Async CTP, because the code generated by the compiler doesn't like multiple awaits or possible rethrows in catch blocks.

The recursive version is perhaps the simplest way to build a Retry in C#. The following version doesn't use Unwrap and adds an optional delay before retries :

private static Task<T> Retry<T>(Func<T> func, int retryCount, int delay, TaskCompletionSource<T> tcs = null)
    {
        if (tcs == null)
            tcs = new TaskCompletionSource<T>();
        Task.Factory.StartNew(func).ContinueWith(_original =>
        {
            if (_original.IsFaulted)
            {
                if (retryCount == 0)
                    tcs.SetException(_original.Exception.InnerExceptions);
                else
                    Task.Factory.StartNewDelayed(delay).ContinueWith(t =>
                    {
                        Retry(func, retryCount - 1, delay,tcs);
                    });
            }
            else
                tcs.SetResult(_original.Result);
        });
        return tcs.Task;
    } 

The StartNewDelayed function comes from the ParallelExtensionsExtras samples and uses a timer to trigger a TaskCompletionSource when the timeout occurs.

The F# version is a lot simpler:

let retry (asyncComputation : Async<'T>) (retryCount : int) : Async<'T> = 
let rec retry' retryCount = 
    async {
        try
            let! result = asyncComputation  
            return result
        with exn ->
            if retryCount = 0 then
                return raise exn
            else
                return! retry' (retryCount - 1)
    }
retry' retryCount

Unfortunatley, it isn't possible to write something similar in C# using async/await from the Async CTP because the compiler doesn't like await statements inside a catch block. The following attempt also fails silenty, because the runtime doesn't like encountering an await after an exception:

private static async Task<T> Retry<T>(Func<T> func, int retryCount)
    {
        while (true)
        {
            try
            {
                var result = await TaskEx.Run(func);
                return result;
            }
            catch 
            {
                if (retryCount == 0)
                    throw;
                retryCount--;
            }
        }
    }

As for asking the user, you can modify Retry to call a function that asks the user and returns a task through a TaskCompletionSource to trigger the next step when the user answers, eg:

 private static Task<bool> AskUser()
    {
        var tcs = new TaskCompletionSource<bool>();
        Task.Factory.StartNew(() =>
        {
            Console.WriteLine(@"Error Occured, continue? Y\N");
            var response = Console.ReadKey();
            tcs.SetResult(response.KeyChar=='y');

        });
        return tcs.Task;
    }

    private static Task<T> RetryAsk<T>(Func<T> func, int retryCount,  TaskCompletionSource<T> tcs = null)
    {
        if (tcs == null)
            tcs = new TaskCompletionSource<T>();
        Task.Factory.StartNew(func).ContinueWith(_original =>
        {
            if (_original.IsFaulted)
            {
                if (retryCount == 0)
                    tcs.SetException(_original.Exception.InnerExceptions);
                else
                    AskUser().ContinueWith(t =>
                    {
                        if (t.Result)
                            RetryAsk(func, retryCount - 1, tcs);
                    });
            }
            else
                tcs.SetResult(_original.Result);
        });
        return tcs.Task;
    } 

With all the continuations, you can see why an async version of Retry is so desirable.

UPDATE:

In Visual Studio 2012 Beta the following two versions work:

A version with a while loop:

    private static async Task<T> Retry<T>(Func<T> func, int retryCount)
    {
        while (true)
        {
            try
            {
                var result = await Task.Run(func);
                return result;
            }
            catch
            {
                if (retryCount == 0)
                    throw;
                retryCount--;
            }
        }
    }

and a recursive version:

    private static async Task<T> Retry<T>(Func<T> func, int retryCount)
    {
        try
        {
            var result = await Task.Run(func);
            return result;
        }
        catch
        {
            if (retryCount == 0)
                throw;
        }
        return await Retry(func, --retryCount);
    }

How to retry after exception?

Do a while True inside your for loop, put your try code inside, and break from that while loop only when your code succeeds.

for i in range(0,100):
    while True:
        try:
            # do stuff
        except SomeSpecificException:
            continue
        break

Why is the Exception not being handled in this Asynchronous Retry Wrapper using TPL?

What you want to do is not easy. In fact, it's so challenging to get "just right" that I wrote a library specifically to solve this type of problem (the Rackspace Threading Library, Apache 2.0 License).

Desired Behavior

The desired behavior for your code is most easily described using async/await, even though the final version of the code will not use this functionality. I made a few changes to your Retry method for this.

1. To improve the ability of the method to support truly asynchronous operations, I changed the work parameter from a Func<T> to a Func<Task<T>>.
2. To avoid blocking a thread unnecessarily (even if it's a background thread), I used Task.Delay instead of Thread.Sleep.
3. I made the assumption that onException is never null. While this assumption is not easy to omit for code using async/await, it could if necessary be addressed for the implementation below.
4. I made the assumption if the task returned by work enters the Faulted state, it will have an InnerExceptions property that only contains 1 exception. If it contains more than one exception, all except the first will be ignored. While this assumption is not easy to omit for code using async/await, it could if necessary be addressed for the implementation below.

Here is the resulting implementation. Note that I could have used a for loop instead of a while loop, but as you will see that would complicate the step that follows.

public async Task<T> Retry<T, TException>(Func<Task<T>> work, Action<TException> onException, TimeSpan retryInterval, int maxExecutionCount)
    where TException : Exception
{
    int count = 0;
    while (count < maxExecutionCount)
    {
        if (count > 0)
            await Task.Delay(retryInterval);

        count++;

        try
        {
            return await work();
        }
        catch (TException ex)
        {
            onException(ex);
        }
        catch (Exception ex)
        {
            throw new RetryWrapperException("Unexpected exception occurred", ex);
        }
    }

    string message = string.Format("Retry unsuccessful after: {0} attempt(s)", maxExecutionCount);
    throw new RetryWrapperException(message);
}

Porting to .NET 4.0

Converting this code to .NET 4.0 (and even .NET 3.5) uses the following features of the Rackspace Threading Library:

• TaskBlocks.While: To convert the while loop.
• CoreTaskExtensions.Select: For performing synchronous operations after an antecedent task completes successfully.
• CoreTaskExtensions.Then: For performing asynchronous operations after an antecedent task completes successfully.
• CoreTaskExtensions.Catch (new for V1.1): For exception handling.
• DelayedTask.Delay (new for V1.1): For the behavior of Task.Delay.

There are a few behavioral differences between this implementation and the one above. In particular:

• If work returns null, the Task returned by this method will transition to the canceled state (as shown by this test), where the method above will transition to the faulted state due to a NullReferenceException.
• This implementation behaves as though ConfigureAwait(false) were called before every await in the previous implementation. In my mind at least, this is actually not a bad thing.

• If the onException method throws an exception, this implementation will wrap that exception in a RetryWrapperException. In other words, this implementation actually models this code, rather than the block written in the Desired Behavior section:

  try
  {
      try
      {
          return await work();
      }
      catch (TException ex)
      {
          onException(ex);
      }
  }
  catch (Exception ex)
  {
      throw new RetryWrapperException("Unexpected exception occurred", ex);
  }
  

Here is the resulting implementation:

public static Task<T> Retry<T, TException>(Func<Task<T>> work, Action<TException> onException, TimeSpan retryInterval, int maxExecutionCount)
    where TException : Exception
{
    int count = 0;
    bool haveResult = false;
    T result = default(T);

    Func<bool> condition = () => count < maxExecutionCount;
    Func<Task> body =
        () =>
        {
            Task t1 = count > 0 ? DelayedTask.Delay(retryInterval) : CompletedTask.Default;

            Task t2 =
                t1.Then(
                    _ =>
                    {
                        count++;
                        return work();
                    })
                .Select(
                    task =>
                    {
                        result = task.Result;
                        haveResult = true;
                    });

            Task t3 =
                t2.Catch<TException>(
                    (_, ex) =>
                    {
                        onException(ex);
                    })
                .Catch<Exception>((_, ex) =>
                    {
                        throw new RetryWrapperException("Unexpected exception occurred", ex);
                    });

            return t3;
        };

    Func<Task, T> selector =
        _ =>
        {
            if (haveResult)
                return result;

            string message = string.Format("Retry unsuccessful after: {0} attempt(s)", maxExecutionCount);
            throw new RetryWrapperException(message);
        };

    return
        TaskBlocks.While(condition, body)
        .Select(selector);
}

Sample Test

The following test method demonstrates the code above functions as described.

[TestMethod]
public void Run()
{
    Func<Task<string>> work = GetSampleResultAsync;
    Action<InvalidOperationException> onException = e => Console.WriteLine("Caught the exception: {0}", e);
    TimeSpan retryInterval = TimeSpan.FromSeconds(5);
    int maxRetryCount = 4;

    Task<string> resultTask = Retry(work, onException, retryInterval, maxRetryCount);
    Console.WriteLine("This wrapper doesn't block");

    var result = resultTask.Result;
    Assert.IsFalse(string.IsNullOrWhiteSpace(result));
    Assert.AreEqual("Sample result!", result);
}

private static int _counter;

private static Task<string> GetSampleResultAsync()
{
    if (_counter < 3)
    {
        _counter++;
        throw new InvalidOperationException("Baaah!");
    }

    return CompletedTask.FromResult("Sample result!");
}

Future Considerations

If you really want to have a rock-solid implementation, I recommend you further modify your code in the following ways.

1. Support cancellation.

   1. Add a CancellationToken cancellationToken parameter as the last parameter of the Retry method.
   2. Change the type of work to Func<CancellationToken, Task<T>>.
   3. Pass the cancellationToken argument to work, and to the call to DelayedTask.Delay.

2. Support back-off policies. You could remove the retryInterval and maxExecutionCount parameters and use an IEnumerable<TimeSpan> instead, or you could incorporate and interface like IBackoffPolicy along with a default implementation like BackoffPolicy (both MIT licensed).

How to retry a failed service-task by incrementing the retry count?

yes, it is definitely possible to retry incidents via cockpit. I am doing it all the time.
I suppose your problem is, that you do not use async continuation ...this is required, otherwise, if an exception occurs the transaction is rolled back and you cannot retry ... if you mark the critical parts of your BPMN as async, you will see the incident in the cockpit and can init a retry.

---

Related Topics