Retry and Task.Whenall() - does Retry "do the right thing"?

[stevebus]

Short version: what is the behavior if CallActivityWithRetryAsync is used in concert with Task.WhenAll()? Does the task not return until all retries have been exhausted, even if one of the Activity calls in the middle throws an exception? That's my assumption (and the desired behavior), but want to confirm....

Longer TL:DR version:
I have some code that does some electrical circuit cost calculations and then makes an "offer" to each customer on that circuit to buy or sell power to/from them at that cost (plus a margin, of course). The code is all invoked from an orchestrator with a call to a suborchestrator to get the circuit cost then a fan-out of calls to an activity function function to persist the 'offers' via an API

the code/pseudocode is something like this....

double costForCircuit = //call a suborchestrator to get the circuit cost and add the margin

List<Task> offerWriterTasks = new List<Task>();

foreach (customer on circuit)
{
    // do some other stuff first, and then....
    offerWriterTasks.Add(context.CallActivityAsync(nameof(OfferWriter), "params that don't matter for this question"));
}

try
{
    await Task.WhenAll(offerWriterTasks);
}
catch (Exception e)
{
   log.logerror("one of the tasks failed, but unfortunately I can't get to all of the exceptions because WhenAll is dumb :-) ");
}

foreach (var task in offerWriterTasks)
{
   if (task.Status == TaskStatus.RanToCompletion)
   {
         // do some stuff with the 'successful task'
   }
   else
   {
        //  log some details of the failure
   }
}

With the exception (no pun intended) of the fact that WhenAll is dumb and only gives you the last 'exception' thrown, even if multiple of the tasks throws an Exception, this code works like I want.

The issue is that the API that is called by the "OfferWriter" activity function will sometimes fail with a transient Exception ("Bad gateway" HTTP error). When that happens, I want to automatically retry the call, because it typically works the second time.

If I set some retryOptions and change the call to CallActivityWithRetryAsync, will the combination of CAWRA and Task.Whenall behave as I expect... i.e. the "task" doesn't return if the activity function fails with an exception within the retry count, but only if it exhausts the retries and still fails? And if that happens, will it bubble up that last Exception as it currently does with only one try?

[stevebus]

ugh, sorry for the code formatting, apparently I didn't do that very well... hopefully still understandable

[cgillum]

TL/DR: Task.WhenAll(...) will behave as you expect - it won't fail unless you exhaust the retry count.

Extended answer: The Task that gets returned by context.CallActivityWithRetryAsync(...) will not complete until either the activity function call succeeds or the retry policy expires. This means that the orchestration will not observe any intermittent failures because the tasks it create that have retry policies will just keep running. It also means that Task.WhenAll(...) won't fail unless the retry count is exhausted.

As far as the exception behavior of Task.WhenAll(...), it should behave the same as it did before - the await implementation of C# will expose just the first exception. If you need to see all the exceptions, then you'll need to write your code a little differently, either using .ContinueWith(...) for each task you create (make sure to run the continuations synchronously!) or wrap each CallActivityWithRetry in its own async method, wrap it with try/catch, and then do await Task.WhenAll(...) on the tasks objects returned by your wrapper method.

[stevebus]

Thanks Chris! That’s what i was hoping for. Appreciate the confirmation.