What Is the Async/Await Equivalent of a Threadpool Server

What is the async/await equivalent of a ThreadPool server?

I'd let the Framework manage the threading and wouldn't create any extra threads, unless profiling tests suggest I might need to. Especially, if the calls inside HandleConnectionAsync are mostly IO-bound.

Anyway, if you like to release the calling thread (the dispatcher) at the beginning of HandleConnectionAsync, there's a very easy solution. You can jump on a new thread from ThreadPool with await Yield(). That works if you server runs in the execution environment which does not have any synchronization context installed on the initial thread (a console app, a WCF service), which is normally the case for a TCP server.

The following illustrate this (the code is originally from here). Note, the main while loop doesn't create any threads explicitly:

using System;
using System.Collections.Generic;
using System.Net.Sockets;
using System.Text;
using System.Threading.Tasks;

class Program
{
    object _lock = new Object(); // sync lock 
    List<Task> _connections = new List<Task>(); // pending connections

    // The core server task
    private async Task StartListener()
    {
        var tcpListener = TcpListener.Create(8000);
        tcpListener.Start();
        while (true)
        {
            var tcpClient = await tcpListener.AcceptTcpClientAsync();
            Console.WriteLine("[Server] Client has connected");
            var task = StartHandleConnectionAsync(tcpClient);
            // if already faulted, re-throw any error on the calling context
            if (task.IsFaulted)
                await task;
        }
    }

    // Register and handle the connection
    private async Task StartHandleConnectionAsync(TcpClient tcpClient)
    {
        // start the new connection task
        var connectionTask = HandleConnectionAsync(tcpClient);

        // add it to the list of pending task 
        lock (_lock)
            _connections.Add(connectionTask);

        // catch all errors of HandleConnectionAsync
        try
        {
            await connectionTask;
            // we may be on another thread after "await"
        }
        catch (Exception ex)
        {
            // log the error
            Console.WriteLine(ex.ToString());
        }
        finally
        {
            // remove pending task
            lock (_lock)
                _connections.Remove(connectionTask);
        }
    }

    // Handle new connection
    private async Task HandleConnectionAsync(TcpClient tcpClient)
    {
        await Task.Yield();
        // continue asynchronously on another threads

        using (var networkStream = tcpClient.GetStream())
        {
            var buffer = new byte[4096];
            Console.WriteLine("[Server] Reading from client");
            var byteCount = await networkStream.ReadAsync(buffer, 0, buffer.Length);
            var request = Encoding.UTF8.GetString(buffer, 0, byteCount);
            Console.WriteLine("[Server] Client wrote {0}", request);
            var serverResponseBytes = Encoding.UTF8.GetBytes("Hello from server");
            await networkStream.WriteAsync(serverResponseBytes, 0, serverResponseBytes.Length);
            Console.WriteLine("[Server] Response has been written");
        }
    }

    // The entry point of the console app
    static async Task Main(string[] args)
    {
        Console.WriteLine("Hit Ctrl-C to exit.");
        await new Program().StartListener();
    }
}

Alternatively, the code might look like below, without await Task.Yield(). Note, I pass an async lambda to Task.Run, because I still want to benefit from async APIs inside HandleConnectionAsync and use await in there:

// Handle new connection
private static Task HandleConnectionAsync(TcpClient tcpClient)
{
    return Task.Run(async () =>
    {
        using (var networkStream = tcpClient.GetStream())
        {
            var buffer = new byte[4096];
            Console.WriteLine("[Server] Reading from client");
            var byteCount = await networkStream.ReadAsync(buffer, 0, buffer.Length);
            var request = Encoding.UTF8.GetString(buffer, 0, byteCount);
            Console.WriteLine("[Server] Client wrote {0}", request);
            var serverResponseBytes = Encoding.UTF8.GetBytes("Hello from server");
            await networkStream.WriteAsync(serverResponseBytes, 0, serverResponseBytes.Length);
            Console.WriteLine("[Server] Response has been written");
        }
    });
}

Updated, based upon the comment: if this is going to be a library code, the execution environment is indeed unknown, and may have a non-default synchronization context. In this case, I'd rather run the main server loop on a pool thread (which is free of any synchronization context):

private static Task StartListener()
{
    return Task.Run(async () => 
    {
        var tcpListener = TcpListener.Create(8000);
        tcpListener.Start();
        while (true)
        {
            var tcpClient = await tcpListener.AcceptTcpClientAsync();
            Console.WriteLine("[Server] Client has connected");
            var task = StartHandleConnectionAsync(tcpClient);
            if (task.IsFaulted)
                await task;
        }
    });
}

This way, all child tasks created inside StartListener wouldn't be affected by the synchronization context of the client code. So, I wouldn't have to call Task.ConfigureAwait(false) anywhere explicitly.

Updated in 2020, someone just asked a good question off-site:

I was wondering what is the reason for using a lock here? This is not
necessary for exception handling. My understanding is that a lock is
used because List is not thread safe, therefore the real question
is why add the tasks to a list (and incur the cost of a lock under
load).

Since Task.Run is perfectly able to keep track of the tasks it
started, my thinking is that in this specific example the lock is
useless, however you put it there because in a real program, having
the tasks in a list allows us to for example, iterate currently
running tasks and terminate the tasks cleanly if the program receives
a termination signal from the operating system.

Indeed, in a real-life scenario we almost always want to keep track of the tasks we start with Task.Run (or any other Task objects which are "in-flight"), for a few reasons:

• To track task exceptions, which otherwise might be silently swallowed if go unobserved elsewhere.
• To be able to wait asynchronously for completion of all the pending tasks (e.g., consider a Start/Stop UI button or handling a request to start/stop a inside a headless Windows service).
• To be able to control (and throttle/limit) the number of tasks we allow to be in-flight simultaneously.

There are better mechanisms to handle a real-life concurrency workflows (e.g., TPL Dataflow Library), but I did include the tasks list and the lock on purpose here, even in this simple example. It might be tempting to use a fire-and-forget approach, but it's almost never is a good idea. In my own experience, when I did want a fire-and-forget, I used async void methods for that (check this).

Async-Await vs ThreadPool vs MultiThreading on High-Performance Sockets (C10k Solutions?)

async/await is roughly analogous to the "Serve many clients with each thread, and use asynchronous I/O and completion notification" approach in your referenced article.

While async and await by themselves do not cause any additional threads, they will make use of thread pool threads if an async method resumes on a thread pool context. Note that the async interaction with ThreadPool is highly optimized; it is very doubtful that you can use Thread or ThreadPool to get the same performance (with a reasonable time for development).

If you can, I'd recommend using an existing protocol - e.g., SignalR. This will greatly simplify your code, since there are many (many) pitfalls to writing your own TCP/IP protocol. SignalR can be self-hosted or hosted on ASP.NET.

When using Async Methods are there Threads of the Threadpool blocked?

It is not a good idea to block a thread of the threadpool for a long time (because they are limited).

The thread pool will respond to a "loss" of a thread by injecting a new one, so blocking a thread pool thread for a long time is not particularly problematic. (Of course, it would be better not to block a thread pool thread in the first place).

Tasks use threads of the threadpool and are therefore for short term, not long blocking actions.

This is only true of Delegate Tasks that are scheduled to the thread pool. A lot of the old (.NET 4-era) documentation around tasks assume that you're using Delegate Tasks scheduled on the thread pool, so this is a common misunderstanding of tasks in general.

In the modern world, Delegate Tasks are much more rare; Promise Tasks are more common these days. Promise Tasks just act as a "signal" meaning "something completed." For more about Delegate Tasks and Promise Tasks, see my blog posts on Task Overview and Task status.

(Also, Delegate Tasks can be scheduled to any kind of TaskScheduler; they're not just for thread pool tasks).

Async methods (like FindAsync in Entity Framework) return a task you can wait (or await) for to receive the result.

There's a distinction here that needs to be made between methods that use async for their implementation, and TAP methods that end in *Async and are intended for use with await. Those two commonly go together, but not always.

async always returns a Promise Task, never a Delegate Task. Also, it's generally expected that an *Async method will return a Promise Task, but some methods return Delegate Tasks instead. When methods do this, I call it "fake asynchrony", because it's just synchronously blocking another thread.

If I call e.g. FindAsync is there simply a task created that runs on a ThreadPool thread and invokes the non async find method (blocking the ThreadPool Thread until the find returns)?

Not in the general case. Pretty much all *Async methods provided by .NET return Promise Tasks.

Or are there deeper operating system mechanisms involved and a ThreadPool Thread is used not before the Find method Returns?

The Promise Tasks returned by most .NET *Async methods use I/O Completion Ports under the hood, as I describe in my article There Is No Thread.

If variant 1 holds true, there is no difference between calling the FindAsync method or starting a task myself and calling the Find method in it.

If variant 2 holds true, there is a difference, because starting a task calling the Find method will long term block a ThreadPool Thread, while calling FindAsync will not.

In the general case, you want to use the *Async methods, which avoid blocking any threads at all.

However, as others have noted in the comments, this particular example of Entity Framework is a bit more complex. Entity Framework itself is async-agnostic; it builds on top of "providers", which may or may not support asynchrony. Microsoft's SqlClient provider does support asynchrony, so FindAsync talking to SQL Server will properly work asynchronously. However, other providers may not (as of this writing, SQLite is a common provider that does not support asynchrony), and for those providers, "asynchronous" APIs like FindAsync are actually implemented by blocking a thread pool thread.

So, in the general case, "variant 2" would be true; but for your particular example of FindAsync, they are both true.

Threads, Task, async/await, Threadpool

What you are missing is that not every asynchronous operation is done on another thread. Waiting on an IO operation or a web service call does not require the creation of a thread. On Windows this is done by using the OS I/O Completion Ports.

What happens when you call something like Stream.ReadAsync is that the OS will issue a read command to the disk and then return to the caller. Once the disk completes the read the notifies the OS kernel which will then trigger a call back to your processes. So there is no need to create a new threadpool thread that will just sit and block.

How async / await can help in ASP.Net application?

I think you are confusing pool queues. There are 5 places where ASP.NET requests can become queued on an IIS server.

1. Application pool queue
2. IIS worker process
3. CLR threadpool queue
4. Integrated mode global queue
5. Classic mode application queue

The Queue Length you are setting to 10 is HTTP.SYS: Application pool queue.

When you use async/awat you are using ASP.NET: CLR threadpool queue.

That is the reason why you get 503 error even with async/await.

On the other hand, here there is a wonderful article about scalling web app with async/await that can help you.

[EDIT] I had just found this article about request queuing that can help too.

Why async/await performs better than threads if it is just a wrapper around them?

There is no reason to expect Node to be faster than a server written in Java. Why do you think it might be?

It seems the other answers here (so far) are explaining the benefits of asynchronous programming in JS compared to single-threaded synchronous operations -- that's obvious, but not the question.

The key point everyone agrees on is that certain operations are inherently slow (e.g.: waiting for network requests, waiting for disk/database access), and it's efficient to let the CPU do something else while such operations are in flight. Using several threads in your application is one well-established way to do that; but of course that's only possible in languages that give you threads. Many traditional server implementations (in Java, C, C++, ...) use one thread per request (or, equivalently, a thread pool to distribute incoming requests over). These threads can block waiting for, say, the database -- that's okay, the operating system will put the waiting thread to sleep and let the CPU work on another thread (handling another request) in the meantime. The end result is fairly similar to what you get with Node.

JavaScript, of course, doesn't make threads available to the programmer. But instead, it has this concept of scheduling requests with the JavaScript engine and providing a callback to be invoked upon completion of the request. That's how the overall system behaves similarly to a traditional threaded programming language: user code can say "do this stuff, then schedule a database access, and when the result is available, continue with this [callback] code here", and while waiting for the database request, the CPU gets to execute some other code. What you want to avoid is the CPU sitting around idly waiting while there is other work waiting for the CPU to have time for it, and both approaches (Java threads and JavaScript callbacks) accomplish that.

Finally, async/await (just like Promises) are indeed just syntactic sugar that make it easier to write callback-based code. Code using async/await isn't any faster than old-style code using callbacks directly, just prettier and less error-prone. It also isn't any faster than a (well-written) Java-based server.

Node.js is popular because it's convenient to use the same language for the client and server parts of an app. From a performance point of view, it's not better than traditional alternatives, it's just also not worse (or at least not much; in practice how efficiently you design your app matters more than whether you implement it in Java or JavaScript). Don't fall for the hype :-)

If async-await doesn't create any additional threads, then how does it make applications responsive?

Actually, async/await is not that magical. The full topic is quite broad but for a quick yet complete enough answer to your question I think we can manage.

Let's tackle a simple button click event in a Windows Forms application:

public async void button1_Click(object sender, EventArgs e)
{
    Console.WriteLine("before awaiting");
    await GetSomethingAsync();
    Console.WriteLine("after awaiting");
}

I'm going to explicitly not talk about whatever it is GetSomethingAsync is returning for now. Let's just say this is something that will complete after, say, 2 seconds.

In a traditional, non-asynchronous, world, your button click event handler would look something like this:

public void button1_Click(object sender, EventArgs e)
{
    Console.WriteLine("before waiting");
    DoSomethingThatTakes2Seconds();
    Console.WriteLine("after waiting");
}

When you click the button in the form, the application will appear to freeze for around 2 seconds, while we wait for this method to complete. What happens is that the "message pump", basically a loop, is blocked.

This loop continuously asks windows "Has anyone done something, like moved the mouse, clicked on something? Do I need to repaint something? If so, tell me!" and then processes that "something". This loop got a message that the user clicked on "button1" (or the equivalent type of message from Windows), and ended up calling our button1_Click method above. Until this method returns, this loop is now stuck waiting. This takes 2 seconds and during this, no messages are being processed.

Most things that deal with windows are done using messages, which means that if the message loop stops pumping messages, even for just a second, it is quickly noticeable by the user. For instance, if you move notepad or any other program on top of your own program, and then away again, a flurry of paint messages are sent to your program indicating which region of the window that now suddenly became visible again. If the message loop that processes these messages is waiting for something, blocked, then no painting is done.

So, if in the first example, async/await doesn't create new threads, how does it do it?

Well, what happens is that your method is split into two. This is one of those broad topic type of things so I won't go into too much detail but suffice to say the method is split into these two things:

1. All the code leading up to await, including the call to GetSomethingAsync
2. All the code following await

Illustration:

code... code... code... await X(); ... code... code... code...

Rearranged:

code... code... code... var x = X(); await X; code... code... code...
^                                  ^          ^                     ^
+---- portion 1 -------------------+          +---- portion 2 ------+

Basically the method executes like this:

1. It executes everything up to await

2. It calls the GetSomethingAsync method, which does its thing, and returns something that will complete 2 seconds in the future

   So far we're still inside the original call to button1_Click, happening on the main thread, called from the message loop. If the code leading up to await takes a lot of time, the UI will still freeze. In our example, not so much

3. What the await keyword, together with some clever compiler magic, does is that it basically something like "Ok, you know what, I'm going to simply return from the button click event handler here. When you (as in, the thing we're waiting for) get around to completing, let me know because I still have some code left to execute".

   Actually it will let the SynchronizationContext class know that it is done, which, depending on the actual synchronization context that is in play right now, will queue up for execution. The context class used in a Windows Forms program will queue it using the queue that the message loop is pumping.

4. So it returns back to the message loop, which is now free to continue pumping messages, like moving the window, resizing it, or clicking other buttons.

   For the user, the UI is now responsive again, processing other button clicks, resizing and most importantly, redrawing, so it doesn't appear to freeze.

5. 2 seconds later, the thing we're waiting for completes and what happens now is that it (well, the synchronization context) places a message into the queue that the message loop is looking at, saying "Hey, I got some more code for you to execute", and this code is all the code after the await.

6. When the message loop gets to that message, it will basically "re-enter" that method where it left off, just after await and continue executing the rest of the method. Note that this code is again called from the message loop so if this code happens to do something lengthy without using async/await properly, it will again block the message loop

There are many moving parts under the hood here so here are some links to more information, I was going to say "should you need it", but this topic is quite broad and it is fairly important to know some of those moving parts. Invariably you're going to understand that async/await is still a leaky concept. Some of the underlying limitations and problems still leak up into the surrounding code, and if they don't, you usually end up having to debug an application that breaks randomly for seemingly no good reason.

• Asynchronous Programming with Async and Await (C# and Visual Basic)
• SynchronizationContext Class
• Stephen Cleary - There is no thread well worth a read!
• Channel 9 - Mads Torgersen: Inside C# Async well worth a watch!

---

OK, so what if GetSomethingAsync spins up a thread that will complete in 2 seconds? Yes, then obviously there is a new thread in play. This thread, however, is not because of the async-ness of this method, it is because the programmer of this method chose a thread to implement asynchronous code. Almost all asynchronous I/O don't use a thread, they use different things. async/await by themselves do not spin up new threads but obviously the "things we wait for" may be implemented using threads.

There are many things in .NET that do not necessarily spin up a thread on their own but are still asynchronous:

• Web requests (and many other network related things that takes time)
• Asynchronous file reading and writing
• and many more, a good sign is if the class/interface in question has methods named SomethingSomethingAsync or BeginSomething and EndSomething and there's an IAsyncResult involved.

Usually these things do not use a thread under the hood.

---

OK, so you want some of that "broad topic stuff"?

Well, let's ask Try Roslyn about our button click:

Try Roslyn

I'm not going to link in the full generated class here but it's pretty gory stuff.

Why does a blocking thread consume more then async/await?

Well, let's imagine a web-server. Most of his time, all he does is wait. it doesn't really CPU-bound usually, but more of I/O bound. It waits for network I/O, disk I/O etc. After every time he waits, he has something (usually very short to do) and then all he does is waiting again. Now, the interesting part is what happend while he waits. In the most "trivial" case (that of course is absolutely not production), you would create a thread to deal with every socket you have.

Now, each of those threads has it's own cost. Some handles, 1MB of stack space... And of course, not all those threads can run in the same time - so the OS scheduler need to deal with that and choose the right thread to run each time (which means A LOT of context switching). It will work for 1 clients. It'll work for 10 clients. But, let's imagine 10,000 clients at the same time. 10,000 threads means 10GB of memory. That's more than the average web server in the world.

All of these resources, is because you dedicated a thread for a user. BUT, most of this threads does nothing! they just wait for something to happen. and the OS has API for async IO that allows you to just queue an operation that will be done once the IO operation completed, without having dedicated thread waiting for it.

If you use async/await, you can write application that will easily use less threads, and each of the thread will be utilized much more - less "doing nothing" time.

async/await is not the only way of doing that. You could have done this before async/await was introduced. BUT, async/await allows you to write code that's very readable and very easy to write that does that, and look almost as it runs just on a single thread (not a lot of callbacks and delegates moving around like before).

By combining the easy syntax of async/await and some features of the OS like async I/O (by using IO completion port), you can write much more scalable code, without losing readability.

Another famous sample is WPF/WinForms. You have the UI thread, that all he does is to process events, and usually has nothing special to do. But, you can't block it or the GUI will hang and the user won't like it. By using async/await and splitting each "hard" work to short operations, you can achieve responsible UI and readable code. If you have to access the DB to execute a query, you'll start the async operation from the UI thread, and then you'll "await" it until it ends and you have results that you can process in the UI thread (because you need to show them to the user, for example). You could have done it before, but using async/await makes it much more readable.

Hope it helps.

Why does my TPL program use more ThreadPool resources than its async/await counterpart?

In this case, the classic TPL version uses more threads than the async/await version because each ContinueWith continuation is executed on a separate pool thread.

Fix that with TaskContinuationsOptions.ExecuteSynchronously:

static Task<TimeSpan> TaskAsyncCalling(TimeSpan time)
{
    SleepService.SleepServiceClient client = new SleepService.SleepServiceClient();

    return client.SleepAsync(time)
        .ContinueWith(t =>
        {
            TimeSpan result = t.Result;
            Console.WriteLine(Thread.CurrentThread.ManagedThreadId);

            return client.SleepAsync(TimeSpan.FromTicks(time.Ticks / 2))
                .ContinueWith(t1 =>
                {
                    result += t1.Result;
                    Console.WriteLine(Thread.CurrentThread.ManagedThreadId);
                    (client as IDisposable).Dispose();

                    return result;
                }, TaskContinuationsOptions.ExecuteSynchronously);
         }, TaskContinuationsOptions.ExecuteSynchronously)
         .Unwrap();
 }

OTOH, the await continuations are normally executed synchronously (if the operation completed on the same synchronization context it was started on, or if there was no synchronization at both points of execution). So it's expected to acquire two less threads.

A good related read: "Why is TaskContinuationsOptions.ExecuteSynchronously opt-in?"

---

Related Topics