Waiting for Multiple Asynchronous Download Tasks

Waiting for multiple asynchronous download tasks

To expand on Abhinav's answer, you should:

Use dispatch_group_create() to create a group.
Call dispatch_group_enter(group) before starting each download task.
Call dispatch_group_leave(group) inside the task's completion handler.
Then call dispatch_group_notify(group, queue, ^{ ... }) to enqueue a block that will be executed when all the tasks are completed.

You can see an example in this post.

(By the way, it's not true that doing 100 dispatch_asyncs in a row will create 100 threads immediately. The system still retains control over how many threads to use to satisfy the queue. However, your code does not wait for any of the tasks to complete before it returns, nor does it attempt to synchronize between multiple tasks completing.)

Waiting for all async downloads to complete

You should consider using Microsoft's Reactive Framework (aka Rx) - NuGet System.Reactive and add using System.Reactive.Linq; - then you can do this:

var query =
    from x in targets.ToObservable().Select((t, c) => new { t, c })
    where x.t.EndsWith(".vtt", StringComparison.InvariantCultureIgnoreCase)
    let target = $"{base_url}{sub_id}/{x.t}"
    let name = $"{sub_id}.{x.c}.vtt"
    from status in
        Observable
            .Using(
                () => new WebClient(),
                wc =>
                {
                    var progress =
                        Observable
                            .FromEventPattern<DownloadProgressChangedEventHandler, DownloadProgressChangedEventArgs>(
                                h => wc.DownloadProgressChanged += h, h => wc.DownloadProgressChanged -= h)
                            .Select(ep => $"{ep.EventArgs.ProgressPercentage}% downloaded.");

                    var completed =
                        Observable
                            .FromAsync(() => wc.DownloadFileTaskAsync(target, $"{Environment.CurrentDirectory}/Subs/{name}"))
                            .Select(z => $"{target} was downloaded.");

                    return progress.Merge(completed);
                })
    select new { target, status };

That's one asynchronous query that handles all of the parallel calls - disposing the WebClient as it finishes each call.

You can get it to wait for all of the results like this:

query
    .Do(x => Console.WriteLine(x.status))
    .ToArray()
    .Wait();

But the more idiomatic way to handle it is this:

IDisposable subscription =
    query
        .Subscribe(
            x => Console.WriteLine(x.status),
            ex => { /* handle an exception */ },
            () => { /* when everything is done */ });

This processes the results as soon as they are available and gives you a chance to run some code when you're done.

If you need to marshall to a UI thread then you can do this:

IDisposable subscription =
    query
        .ObserveOnDispatcher() // or .ObserveOn(instanceOfForm)
        .Subscribe(
            x => Console.WriteLine(x.status),
            ex => { /* handle an exception */ },
            () => { /* when everything is done */ });

To stop the download in case you need to stop early just do subscription.Dispose();.

How to wait for results from multiple async methods wrapped in a for loop?

I think I know what you try to achieve, and the issue you are having, this.getFinalDisplayResults() is executed before you have the results because the logic inside the for loop is asynchronous, so the fix would be.

async function getDataFromBackend () {
    for(let type in ["player", "team", "event"]) {
        const searchResult = await this.searchService.getSearchResult(type).toPromise()
        if(type === "player")
            this.player_search_results_full = this.getPlayerSearchResults(searchResult, search_string);
           
        if(type === "team")
             this.team_search_results_full = this.getTeamSearchResults(searchResult, search_string);
           
        if(type === "event")
             this.event_search_results_full = this.getEventSearchResults(searchResult, search_string);  
     }
}

  

async function getFinalDisplayResults() {
            await getDataFromBackend(); // this will ensure you have the data, before you do the rest of the process
    //rest of the logic here
}

Secure and effective way for waiting for asynchronous task

I believe replacing of mutex with CountDownLatch in waitingRoom approach prevents deadlock.

CountDownLatch latch = new CountDownLatch(1)
taskProcessor.addToWaitingRoom(uniqueIdentifier, latch)
while (!checkResultIsInDatabase())
  // consider timed version
  latch.await()

//TaskProcessor
... Some complicated calculations
if (uniqueIdentifierExistInWaitingRoom(taskUniqueIdentifier))
  getLatchFromWaitingRoom(taskUniqueIdentifier).countDown()

Wait for all tasks when tasks run async methods

new Task() accepts an Action, so you're passing an async void that does not wait for its result.

You should use Task.Run() instead, which also accepts Func<Task>.

Download multiple files async and wait for all of them to finish before executing the rest of the code

The DownloadFileAsync/DownloadFileCompleted members of WebClient use the Event-based Asynchronous Pattern. If you want to use async and await, you should be using the Task-based Asynchronous Pattern.

In this case, you should use the DownloadFileTaskAsync member, as such:

private async Task DownloadFileAsync(DocumentObject doc)
{
  try
  {
    using (WebClient webClient = new WebClient())
    {
      string downloadToDirectory = @Resources.defaultDirectory + doc.docName;
      webClient.Credentials = System.Net.CredentialCache.DefaultNetworkCredentials;
      await webClient.DownloadFileTaskAsync(new Uri(doc.docUrl), @downloadToDirectory);

      //Add them to the local
      Context.listOfLocalDirectories.Add(downloadToDirectory);
    }         
  }
  catch (Exception)
  {
    Errors.printError("Failed to download File: " + doc.docName);
  }
}

private async Task DownloadMultipleFilesAsync(List<DocumentObject> doclist)
{
  await Task.WhenAll(doclist.Select(doc => DownloadFileAsync(doc)));
}

Please note that your Context.listOfLocalDirectories.Add and Errors.printError methods should be threadsafe.

Wait for async download to be completed

YOu need to use tasks to achieve this

First make your download function async

    private async TaskDownloadFile(string Filename, Uri Uri)
        {

            Console.WriteLine(Filename + " " + Uri);
            _download_completed = false;
            WebClient client = new WebClient();
            client.DownloadFileCompleted += new AsyncCompletedEventHandler(DownloadFileCompleted);
            client.DownloadProgressChanged += new DownloadProgressChangedEventHandler(DownloadProgressCallback);

      await   client.DownloadFileTaskAsync(Uri, Filename);

        }

Make the method async and await the DownloadFileTaskAsync method.

I would suggest to remove the events for DownloadFileCompeted and DownloadProgressChanged if you will do multiple downloads they don't make much sense

Now that you have made the method async do the following

var file1DownloadTask = TaskDownloadFile("filename","url");
var file2DownloadTask = TaskDownloadFile("filename","url");

at this point your have two tasks

You can use the Task.WaitAll(list_of_taks_goes_here)

Task.WaitAll(file1DownloadTask,file2DownloadTask )

The wait all method will wait for all the tasks to finished so the next line of code is called after those two tasks have finished

then you can do change the ui or whatever you need to do after the Task.WaitAll line

You could also do the following

await TaskDownloadFile("filename","url");
await TaskDownloadFile("filename2","url");

which will wait for the downloads to finish before moving on to next line but this way you have to know all the files at the time your write the code, with the first method you could have a list of files from which you could build the taks and pass them to Task.WaitAll

Async/Task/Await: Await does not actually wait

Simple: await it!

 public aysnc Task DownloadFromRepo(String fileName)
 {
    ...
    using (WebClient FileClient = new WebClient())
    {
        ...
        await FileClient.DownloadFileTaskAsync(new System.Uri(RepoDir + fileName), 
curFilePath);
    }
}

Without the await, indeed: the Dispose() happens immediately.

I believe that roslynator now automatically detects this scenario and warns you about it (and has an auto-fix available) - well worth installing.

Likewise:

private async Task RetrieveUpdate()
{
    await UpdateInformationDownload.DownloadFiles();
    AnalyzeFile();
}

and:

public async Task DownloadFiles() {...}

Proper way to wait all tasks while still updating the UI thread

You really should use an asynchronous download method based on HttpClient instead of the synchronous method you are showing. Lacking that, I'll use this one:

private async Task DownloadSourceAsync(string url)
{
    await Task.Run(() => DownloadSource(url));

    listBoxStatus.Items.Add($"Finished Downloading {url} source...");
}

Then, you can make your btnGetPages_Click method something like this:

private async void btnGetPages_Click(object sender, EventArgs e)
{
    var tasks = new List<Task>();

    for (int i = 1; i < 11; i++)
    {
        string url = $"http://someURL/page-{i}.html";
        listBoxStatus.Items.Add($"Downloading source from {url}...");

        tasks.Add(DownloadSourceAsync(url));
    }

    Task.WaitAll(tasks.ToArray());
    listBoxStatus.Items.Add("All Source files have completed...");
}

Waiting for Multiple Asynchronous Download Tasks