Why the Default Synchronizationcontext Is Not Captured in a Console App

Why the default SynchronizationContext is not captured in a Console App?

The word "capture" is too opaque, it sounds too much like that is something that the framework is supposed to. Misleading, since it normally does in a program that uses one of the default SynchronizationContext implementations. Like the one you get in a Winforms app. But when you write your own then the framework no longer helps and it becomes your job to do it.

The async/await plumbing gives the context an opportunity to run the continuation (the code after the await) on a specific thread. That sounds like a trivial thing to do, since you've done it so often before, but it is in fact quite difficult. It is not possible to arbitrarily interrupt the code that this thread is executing, that would cause horrible re-entrancy bugs. The thread has to help, it needs to solve the standard producer-consumer problem. Takes a thread-safe queue and a loop that empties that queue, handling invoke requests. The job of the overridden Post and Send methods is to add requests to the queue, the job of the thread is to use a loop to empty it and execute the requests.

The main thread of a Winforms, WPF or UWP app has such a loop, it is executed by Application.Run(). With a corresponding SynchronizationContext that knows how to feed it with invoke requests, respectively WindowsFormsSynchronizationContext, DispatcherSynchronizationContext and WinRTSynchronizationContext. ASP.NET can do it too, uses AspNetSynchronizationContext. All provided by the framework and automagically installed by the class library plumbing. They capture the sync context in their constructor and use Begin/Invoke in their Post and Send methods.

When you write your own SynchronizationContext then you must now take care of these details. In your snippet you did not override Post and Send but inherited the base methods. They know nothing and can only execute the request on an arbitrary threadpool thread. So SynchronizationContext.Current is now null on that thread, a threadpool thread does not know where the request came from.

Creating your own isn't that difficult, ConcurrentQueue and delegates help a lot of cut down on the code. Lots of programmers have done so, this library is often quoted. But there is a severe price to pay, that dispatcher loop fundamentally alters the way a console mode app behaves. It blocks the thread until the loop ends. Just like Application.Run() does.

You need a very different programming style, the kind that you'd be familiar with from a GUI app. Code cannot take too long since it gums up the dispatcher loop, preventing invoke requests from getting dispatched. In a GUI app pretty noticeable by the UI becoming unresponsive, in your sample code you'll notice that your method is slow to complete since the continuation can't run for a while. You need a worker thread to spin-off slow code, there is no free lunch.

Worthwhile to note why this stuff exists. GUI apps have a severe problem, their class libraries are never thread-safe and can't be made safe by using lock either. The only way to use them correctly is to make all the calls from the same thread. InvalidOperationException when you don't. Their dispatcher loop help you do this, powering Begin/Invoke and async/await. A console does not have this problem, any thread can write something to the console and lock can help to prevent their output from getting intermingled. So a console app shouldn't need a custom SynchronizationContext. YMMV.

Utilizing Async/Await in .NET Console applications breaks when calling Application.Run() or instantiating a WinForms UserControl object

In the absence of synchronization context (or when the default SyncrhonizationContext is used), it's often possible for an await continuation to run synchronously, i.e., on the same thread where its antecedent task has ended. That can lead to obscure deadlocks, and it was one of the reasons TaskContinuationOptions.RunContinuationsAsynchronously was introduced in .NET Framework 4.6. For some more details and examples, check out this blog post: The danger of TaskCompletionSource class.

The fact that AsyncPump stops your code from hanging indicates you may have a similar situation somewhere inside mcc.Run(). As AsyncPump imposes true asynchrony for await continuations (albeit on the same thread), it reduces the chance for deadlocks.

That said, I'm not suggesting using AsyncPump or WindowsFormsSynchronizationContext as a workaround. Rather, you should try to find what exactly causes your code to hang (and where), and solve it locally, e.g. simply by wrapping the offending call with Task.Run.

One other issue I can spot in your code is that you don't wait or await the task returned by MainAsync. Because of that, at least for the console branch of your logic (especially without using AsyncPump), your program may be ending prematurely, depending on what's going in inside mcc.Run(), and you may be letting some exceptions go unobserved.

Why the code after awating is running on different thread (even with SynchronizationContext set)?

I would like to show some code according to my understanding, hopefully it can help somebody.

As aepot, dymanoid and Hans Passant (thanks to them) said, using the default SynchronizationContext will do nothing more than Posting the rest of the code after awaiting to the SynchronizationContext.

I created a very very basic and NOT optimal SynchronizationContext to demonstrate how the basic implementation should look like. My implementation will create a new Thread and run some Tasks in a specific context inside the same newly created Thread.

Better implementation (but much complex) may be found here in Stephen Cleary's GitHub repository.

My implementation looks basically like following (from my GitHub repository, the code in the repository may look different in the future):

/// <summary>
/// This <see cref="SynchronizationContext"/> will call all posted callbacks in a single new thread.
/// </summary>
public class SingleNewThreadSynchronizationContext : SynchronizationContext
{
    readonly Thread _workerThread;
    readonly BlockingCollection<KeyValuePair<SendOrPostCallback, object>> _actionStatePairs = new BlockingCollection<KeyValuePair<SendOrPostCallback, object>>();

    /// <summary>
    /// Returns the Id of the worker <see cref="Thread"/> created by this <see cref="SynchronizationContext"/>.
    /// </summary>
    public int ManagedThreadId => _workerThread.ManagedThreadId;

    public SingleNewThreadSynchronizationContext()
    {
        // Creates a new thread to run the posted calls.
        _workerThread = new Thread(() =>
        {
            try
            {
                while (true)
                {
                    var actionStatePair = _actionStatePairs.Take();
                    SetSynchronizationContext(this);
                    actionStatePair.Key?.Invoke(actionStatePair.Value);
                }
            }
            catch (ThreadAbortException)
            {
                Console.WriteLine($"The thread {_workerThread.ManagedThreadId} of {nameof(SingleNewThreadSynchronizationContext)} was aborted.");
            }
        });

        _workerThread.IsBackground = true;
        _workerThread.Start();
    }

    public override void Post(SendOrPostCallback d, object state)
    {
        // Queues the posted callbacks to be called in this SynchronizationContext.
        _actionStatePairs.Add(new KeyValuePair<SendOrPostCallback, object>(d, state));
    }

    public override void Send(SendOrPostCallback d, object state)
    {
        throw new NotSupportedException();
    }

    public override void OperationCompleted()
    {
        _actionStatePairs.Add(new KeyValuePair<SendOrPostCallback, object>(new SendOrPostCallback(_ => _workerThread.Abort()), null));
        _actionStatePairs.CompleteAdding();
    }
}

and here is a Demo to use it:

static void SingleNewThreadSynchronizationContextDemo()
{
    var synchronizationContext = new SingleNewThreadSynchronizationContext();

    // Creates some tasks to test that the whole calls in the tasks (before and after awaiting) will be called in the same thread.
    for (int i = 0; i < 20; i++)
        Task.Run(async () =>
        {
            SynchronizationContext.SetSynchronizationContext(synchronizationContext);
            // Before yielding, the task will be started in some thread-pool thread.
            var threadIdBeforeYield = Thread.CurrentThread.ManagedThreadId;
            // We yield to post the rest of the task after await to the SynchronizationContext.
            // Other possiblity here is maybe to start the whole Task using a different TaskScheduler.
            await Task.Yield();

            var threadIdBeforeAwait1 = Thread.CurrentThread.ManagedThreadId;
            await Task.Delay(100);
            var threadIdBeforeAwait2 = Thread.CurrentThread.ManagedThreadId;
            await Task.Delay(100);

            Console.WriteLine($"SynchronizationContext: thread Id '{synchronizationContext.ManagedThreadId}' | type '{SynchronizationContext.Current?.GetType()}.'");
            Console.WriteLine($"Thread Ids: Before yield '{threadIdBeforeYield}' | Before await1 '{threadIdBeforeAwait1}' | Before await2 '{threadIdBeforeAwait2}' | After last await '{Thread.CurrentThread.ManagedThreadId}'.{Environment.NewLine}");
        });
}

static void Main(string[] args)
{
    Console.WriteLine($"Entry thread {Thread.CurrentThread.ManagedThreadId}");
    SingleNewThreadSynchronizationContextDemo();
    Console.WriteLine($"Exit thread {Thread.CurrentThread.ManagedThreadId}");

    Console.ReadLine();
}

Output:

Entry thread 1   
Exit thread 1  
SynchronizationContext: thread Id '5' | type 'SynchronizationContexts.SingleNewThreadSynchronizationContext.'  
Thread Ids: Before yield '11' | Before await1 '5' | Before await2 '5' | After last await '5'.

SynchronizationContext: thread Id '5' | type 'SynchronizationContexts.SingleNewThreadSynchronizationContext.'  
Thread Ids: Before yield '4' | Before await1 '5' | Before await2 '5' | After last await '5'.

SynchronizationContext: thread Id '5' | type 'SynchronizationContexts.SingleNewThreadSynchronizationContext.'  
Thread Ids: Before yield '12' | Before await1 '5' | Before await2 '5' | After last await '5'.

SynchronizationContext: thread Id '5' | type 'SynchronizationContexts.SingleNewThreadSynchronizationContext.'  
Thread Ids: Before yield '6' | Before await1 '5' | Before await2 '5' | After last await '5'.

SynchronizationContext: thread Id '5' | type 'SynchronizationContexts.SingleNewThreadSynchronizationContext.'  
Thread Ids: Before yield '10' | Before await1 '5' | Before await2 '5' | After last await '5'.

SynchronizationContext: thread Id '5' | type 'SynchronizationContexts.SingleNewThreadSynchronizationContext.'  
Thread Ids: Before yield '7' | Before await1 '5' | Before await2 '5' | After last await '5'.

Default SynchronizationContext vs Default TaskScheduler

When you start diving this deep into the implementation details, it's important to differentiate between documented/reliable behavior and undocumented behavior. Also, it's not really considered proper to have SynchronizationContext.Current set to new SynchronizationContext(); some types in .NET treat null as the default scheduler, and other types treat null or new SynchronizationContext() as the default scheduler.

When you await an incomplete Task, the TaskAwaiter by default captures the current SynchronizationContext - unless it is null (or its GetType returns typeof(SynchronizationContext)), in which case the TaskAwaiter captures the current TaskScheduler. This behavior is mostly documented (the GetType clause is not AFAIK). However, please note that this describes the behavior of TaskAwaiter, not TaskScheduler.Default or TaskFactory.StartNew.

After the context (if any) is captured, then await schedules a continuation. This continuation is scheduled using ExecuteSynchronously, as described on my blog (this behavior is undocumented). However, do note that ExecuteSynchronously does not always execute synchronously; in particular, if a continuation has a task scheduler, it will only request to execute synchronously on the current thread, and the task scheduler has the option to refuse to execute it synchronously (also undocumented).

Finally, note that a TaskScheduler can be requested to execute a task synchronously, but a SynchronizationContext cannot. So, if the await captures a custom SynchronizationContext, then it must always execute the continuation asynchronously.

So, in your original Test #1:

StartNew starts a new task with the default task scheduler (on thread 10).
SetResult synchronously executes the continuation set by await tcs.Task.
At the end of the StartNew task, it synchronously executes the continuation set by await task.

In your original Test #2:

StartNew starts a new task with a task scheduler wrapper for a default-constructed synchronization context (on thread 10). Note that the task on thread 10 has TaskScheduler.Current set to a SynchronizationContextTaskScheduler whose m_synchronizationContext is the instance created by new SynchronizationContext(); however, that thread's SynchronizationContext.Current is null.
SetResult attempts to execute the await tcs.Task continuation synchronously on the current task scheduler; however, it cannot because SynchronizationContextTaskScheduler sees that thread 10 has a SynchronizationContext.Current of null while it is requiring a new SynchronizationContext(). Thus, it schedules the continuation asynchronously (on thread 11).
A similar situation happens at the end of the StartNew task; in this case, I believe it's coincidental that the await task continues on the same thread.

In conclusion, I must emphasize that depending on undocumented implementation details is not wise. If you want to have your async method continue on a thread pool thread, then wrap it in a Task.Run. That will make the intent of your code much clearer, and also make your code more resilient to future framework updates. Also, don't set SynchronizationContext.Current to new SynchronizationContext(), since the handling of that scenario is inconsistent.

In console app, why does synchronous blocking code (Thread.Sleep(..)) when used in an awaited async task behave like multi threading?

await Task.Delay(5000); does quite a good job of simultaing IO-bound operations. And though IO-bound operations do not use threads to wait for completion (see there is no thread article by Stephen Cleary, also docs can shed some light), the continuations will be run on thread pool. So downloadTasksQuery.ToList() will start all your await Task.Delay's in parallel then (depending on number of tasks and thread pool and SynchronizationContext) settings some or all of them can be continued on separate threads.

So why is each Task not blocking for 10 seconds? And what would be the way to make each Task block for 10 seconds?

It blocks but it blocks a separate thread in your case.

I have been advised that this behaviour is related to synchronization context.

Yes, this behaviour can be affected by synchronization context. For example in desctop apps the continuations which are not marked with ConfigureAwait(false) will run on the single UI threads and since you don't have ConfigureAwait(false) configured for await Task.Delay(5000) you effectively end up making UI unresponsive.

async/await deadlock when using WindowsFormsSynchronizationContext in a console app

This happens because the WindowsFormsSynchronizationContext depends on the existence of a standard Windows message loop. A console application does not start such a loop, so the messages posted to the WindowsFormsSynchronizationContext are not processed, the task continuations are not invoked, and so the program hangs on the first await. You can confirm the non-existence of a message loop by querying the boolean property Application.MessageLoop.

Gets a value indicating whether a message loop exists on this thread.

To make the WindowsFormsSynchronizationContext functional you must start a message loop. It can be done like this:

static void Main(string[] args)
{
    EventHandler idleHandler = null;
    idleHandler = async (sender, e) =>
    {
        Application.Idle -= idleHandler;
        await MyMain(args);
        Application.ExitThread();
    };
    Application.Idle += idleHandler;
    Application.Run();
}

The MyMain method is your current Main method, renamed.

Update: Actually the Application.Run method installs automatically a WindowsFormsSynchronizationContext in the current thread, so you don't have to do it explicitly. If you want you can prevent this automatic installation, be configuring the property WindowsFormsSynchronizationContext.AutoInstall before calling Application.Run.

The AutoInstall property determines whether the WindowsFormsSynchronizationContext is installed when a control is created, or when a message loop is started.

Why the Default Synchronizationcontext Is Not Captured in a Console App