Which Blocking Operations Cause an Sta Thread to Pump Com Messages

Which blocking operations cause an STA thread to pump COM messages?

BlockingCollection will indeed pump while blocking. I've learnt that while answering the following question, which has some interesting details about STA pumping:

StaTaskScheduler and STA thread message pumping

However, it will pump a very limited undisclosed set of COM-specific messages, same as the other APIs you listed. It won't pump general purpose Win32 messages (a special case is WM_TIMER, which won't be dispatched either). This might be a problem for some STA COM objects which expect a full-featured message loop.

If you like to experiment with this, create your own version of SynchronizationContext, override SynchronizationContext.Wait, call SetWaitNotificationRequired and install your custom synchronization context object on an STA thread. Then set a breakpoint inside Wait and see what APIs will make it get called.

To what extent the standard pumping behavior of WaitOne is actually limited? Below is a typical example causing a deadlock on the UI thread. I use WinForms here, but the same concern applies to WPF:

public partial class MainForm : Form
{
    public MainForm()
    {
        InitializeComponent();

        this.Load += (s, e) =>
        {
            Func<Task> doAsync = async () =>
            {
                await Task.Delay(2000);
            };

            var task = doAsync();
            var handle = ((IAsyncResult)task).AsyncWaitHandle;

            var startTick = Environment.TickCount;
            handle.WaitOne(4000);
            MessageBox.Show("Lapse: " + (Environment.TickCount - startTick));
        };
    }
}

The message box will show the time lapse of ~ 4000 ms, although the task takes only 2000 ms to complete.

That happens because the await continuation callback is scheduled via WindowsFormsSynchronizationContext.Post, which uses Control.BeginInvoke, which in turn uses PostMessage, posting a regular Windows message registered with RegisterWindowMessage. This message doesn't get pumped and handle.WaitOne times out.

If we used handle.WaitOne(Timeout.Infinite), we'd have a classic deadlock.

Now let's implement a version of WaitOne with explicit pumping (and call it WaitOneAndPump):

public static bool WaitOneAndPump(
    this WaitHandle handle, int millisecondsTimeout)
{
    var startTick = Environment.TickCount;
    var handles = new[] { handle.SafeWaitHandle.DangerousGetHandle() };

    while (true)
    {
        // wait for the handle or a message
        var timeout = (uint)(Timeout.Infinite == millisecondsTimeout ?
                Timeout.Infinite :
                Math.Max(0, millisecondsTimeout + 
                    startTick - Environment.TickCount));

        var result = MsgWaitForMultipleObjectsEx(
            1, handles,
            timeout,
            QS_ALLINPUT,
            MWMO_INPUTAVAILABLE);

        if (result == WAIT_OBJECT_0)
            return true; // handle signalled
        else if (result == WAIT_TIMEOUT)
            return false; // timed-out
        else if (result == WAIT_ABANDONED_0)
            throw new AbandonedMutexException(-1, handle);
        else if (result != WAIT_OBJECT_0 + 1)
            throw new InvalidOperationException();
        else
        {
            // a message is pending 
            if (timeout == 0)
                return false; // timed-out
            else
            {
                // do the pumping
                Application.DoEvents();
                // no more messages, raise Idle event
                Application.RaiseIdle(EventArgs.Empty);
            }
        }
    }
}

And change the original code like this:

var startTick = Environment.TickCount;
handle.WaitOneAndPump(4000);
MessageBox.Show("Lapse: " + (Environment.TickCount - startTick));

The time lapse now will be ~2000 ms, because the await continuation message gets pumped by Application.DoEvents(), the task completes and its handle is signaled.

That said, I'd never recommend using something like WaitOneAndPump for production code (besides for very few specific cases). It's a source of various problems like UI re-entrancy. Those problems are the reason Microsoft has limited the standard pumping behavior to only certain COM-specific messages, vital for COM marshaling.

Which blocking threading operations in .NET will handle COM messages when blocked?

This can be reverse-engineered somewhat from the SSCLI20 source, albeit that it is dated. The core CLR function that implements MsgWaitForMultipleHandles is DoAppropriateWait. From what I see, it is used by AutoResetEvent, ManualResetEvent and Semaphore but not by Mutex. One quirk is that WaitHandle.WaitAny is okay from an STA thread but WaitAll() generates an exception. Yeah, this isn't documented anywhere other then a few hints in Chris Brumme's blog. You can't assume too much from what I found, test this to be sure.

How to post messages to an STA thread running a message pump?

Keep in mind that the message queue that Windows creates for an STA thread is already an implementation of a thread-safe queue. So just use it for your own purposes. Here's a base class that you can use, derive your own to include your COM object. Override the Initialize() method, it will be called as soon as the thread is ready to start executing code. Don't forget to call base.Initialize() in your override.

It you want to run code on that thread then use the BeginInvoke or Invoke methods, just like you would for the Control.Begin/Invoke or Dispatcher.Begin/Invoke methods. Call its Dispose() method to shut down the thread, it is optional. Beware that this is only safe to do when you are 100% sure that all COM objects are finalized. Since you don't usually have that guarantee, it is better that you don't.

using System;
using System.Threading;
using System.Windows.Forms;

class STAThread : IDisposable {
    public STAThread() {
        using (mre = new ManualResetEvent(false)) {
            thread = new Thread(() => {
                Application.Idle += Initialize;
                Application.Run();
            });
            thread.IsBackground = true;
            thread.SetApartmentState(ApartmentState.STA);
            thread.Start();
            mre.WaitOne();
        }
    }
    public void BeginInvoke(Delegate dlg, params Object[] args) {
        if (ctx == null) throw new ObjectDisposedException("STAThread");
        ctx.Post((_) => dlg.DynamicInvoke(args), null);
    }
    public object Invoke(Delegate dlg, params Object[] args) {
        if (ctx == null) throw new ObjectDisposedException("STAThread");
        object result = null;
        ctx.Send((_) => result = dlg.DynamicInvoke(args), null);
        return result;
    }
    protected virtual void Initialize(object sender, EventArgs e) {
        ctx = SynchronizationContext.Current;
        mre.Set();
        Application.Idle -= Initialize;
    }
    public void Dispose() {
        if (ctx != null) {
            ctx.Send((_) => Application.ExitThread(), null);
            ctx = null;
        }
    }
    private Thread thread;
    private SynchronizationContext ctx;
    private ManualResetEvent mre;
}

blocking in STA COM object

You need to allow the STA thread to pump messages because otherwise, calls cannot be dispatched to the STA thread: Since there is already a thread in the apartment (and it's single threaded, duh) no other thread can enter the apartment, so posting a message to the threads (hidden) window is the "only" way to communicate with the STA thread.

This is used to process ORPC requests, but also to process messages not necessarily related to COM (i.e. GUI messages). Interrupting message processing by making a blocking call thus disrupts ORPC functionality and can potentially also block the user interface.

See Don Box "Essential COM" for thorough information on COM.

How to pump COM messages?

The short and long of it is that you have to pump ALL messages normally, you can't just single out COM messages by themselves (and besides, there is no documented messages that you can peek/pump by themselves, they are known only to COM's internals).

How to make WebBrower.Navigate2 synchronous?

You can't. But you don't have to wait for the OnDocumentComplete event, either. You can busy-loop inside of NavigateToEmpty() itself until the WebBrowser's ReadyState property is READYSTATE_COMPLETE, pumping the message queue when messages are waiting to be processed:

procedure TContoso.NavigateToEmpty(WebBrowser: IWebBrowser2);
begin
  WebBrowser.Navigate2('about:blank');
  while (WebBrowser.ReadyState <> READYSTATE_COMPLETE) and (not Application.Terminated) do
  begin
    // if MsgWaitForMultipleObjects(0, Pointer(nil)^, False, 5000, QS_ALLINPUT) = WAIT_OBJECT_0 then
    // if GetQueueStatus(QS_ALLINPUT) <> 0 then
      Application.ProcessMessages;
  end;
end;

How to pump COM messages?

You can't, not by themselves anyway. Pump everything, and be prepared to handle any reentry issues that result from that.

Does pumping COM messages cause COM events to callback?

Yes.

How to use CoWaitForMultipleHandles

Try something like this:

procedure TContoso.NavigateToEmpty(WebBrowser: IWebBrowser2);
var
  hEvent: THandle;
  dwIndex: DWORD;
  hr: HRESULT;
begin
  // when UseCOMWait() is true, TEvent.WaitFor() does not wait for, or
  // notify, when messages are pending in the queue, so use
  // CoWaitForMultipleHandles() directly instead.  But you have to still
  // use a waitable object, just don't signal it...
  hEvent := CreateEvent(nil, True, False, nil);
  if hEvent = 0 then RaiseLastOSError;
  try
    WebBrowser.Navigate2('about:blank');
    while (WebBrowser.ReadyState <> READYSTATE_COMPLETE) and (not Application.Terminated) do
    begin
      hr := CoWaitForMultipleHandles(COWAIT_INPUTAVAILABLE, 5000, 1, hEvent, dwIndex);
      case hr of
        S_OK: Application.ProcessMessages;
        RPC_S_CALLPENDING, RPC_E_TIMEOUT: begin end;
      else
        RaiseLastOSError(hr);
      end;
    end;
  finally
    CloseHandle(hEvent);
  end;
end;

StaTaskScheduler and STA thread message pumping

My understanding of your problem: you are using StaTaskScheduler only to organize the classic COM STA apartment for your legacy COM objects. You're not running a WinForms or WPF core message loop on the STA thread of StaTaskScheduler. That is, you're not using anything like Application.Run, Application.DoEvents or Dispatcher.PushFrame inside that thread. Correct me if this is a wrong assumption.

By itself, StaTaskScheduler doesn't install any synchronization context on the STA threads it creates. Thus, you're relying upon the CLR to pump messages for you. I've only found an implicit confirmation that the CLR pumps on STA threads, in Apartments and Pumping in the CLR by Chris Brumme:

I keep saying that managed blocking will perform “some pumping” when
called on an STA thread. Wouldn’t it be great to know exactly what
will get pumped? Unfortunately, pumping is a black art which is
beyond mortal comprehension. On Win2000 and up, we simply delegate to
OLE32’s CoWaitForMultipleHandles service.

This indicates the CLR uses CoWaitForMultipleHandles internally for STA threads. Further, the MSDN docs for COWAIT_DISPATCH_WINDOW_MESSAGES flag mention this:

... in STA is only a small set of special-cased messages dispatched.

I did some research on that, but could not get to pump the WM_TEST from your sample code with CoWaitForMultipleHandles, we discussed that in the comments to your question. My understanding is, the aforementioned small set of special-cased messages is really limited to some COM marshaller-specific messages, and doesn't include any regular general-purpose messages like your WM_TEST.

So, to answer your question:

... Should I implemented a custom synchronization context, which would
explicitly pump messages with CoWaitForMultipleHandles, and install it
on each STA thread started by StaTaskScheduler?

Yes, I believe that creating a custom synchronization context and overriding SynchronizationContext.Wait is indeed the right solution.

However, you should avoid using CoWaitForMultipleHandles, and use MsgWaitForMultipleObjectsEx instead. If MsgWaitForMultipleObjectsEx indicates there's a pending message in the queue, you should manually pump it with PeekMessage(PM_REMOVE) and DispatchMessage. Then you should continue waiting for the handles, all inside the same SynchronizationContext.Wait call.

Note there's a subtle but important difference between MsgWaitForMultipleObjectsEx and MsgWaitForMultipleObjects. The latter doesn't return and keeps blocking, if there's a message already seen in the queue (e.g., with PeekMessage(PM_NOREMOVE) or GetQueueStatus), but not removed. That's not good for pumping, because your COM objects might be using something like PeekMessage to inspect the message queue. That might later cause MsgWaitForMultipleObjects to block when not expected.

OTOH, MsgWaitForMultipleObjectsEx with MWMO_INPUTAVAILABLE flag doesn't have such shortcoming, and would return in this case.

A while ago I created a custom version of StaTaskScheduler (available here as ThreadAffinityTaskScheduler) in attempt to solve a different problem: maintaining a pool of threads with thread affinity for subsequent await continuations. The thread affinity is vital if you use STA COM objects across multiple awaits. The original StaTaskScheduler exhibits this behavior only when its pool is limited to 1 thread.

So I went ahead and did some more experimenting with your WM_TEST case. Originally, I installed an instance of the standard SynchronizationContext class on the STA thread. The WM_TEST message didn't get pumped, which was expected.

Then I overridden SynchronizationContext.Wait to just forward it to SynchronizationContext.WaitHelper. It did get called, but still didn't pump.

Finally, I implemented a full-featured message pump loop, here's the core part of it:

// the core loop
var msg = new NativeMethods.MSG();
while (true)
{
    // MsgWaitForMultipleObjectsEx with MWMO_INPUTAVAILABLE returns,
    // even if there's a message already seen but not removed in the message queue
    nativeResult = NativeMethods.MsgWaitForMultipleObjectsEx(
        count, waitHandles,
        (uint)remainingTimeout,
        QS_MASK,
        NativeMethods.MWMO_INPUTAVAILABLE);

    if (IsNativeWaitSuccessful(count, nativeResult, out managedResult) || WaitHandle.WaitTimeout == managedResult)
        return managedResult;

    // there is a message, pump and dispatch it
    if (NativeMethods.PeekMessage(out msg, IntPtr.Zero, 0, 0, NativeMethods.PM_REMOVE))
    {
        NativeMethods.TranslateMessage(ref msg);
        NativeMethods.DispatchMessage(ref msg);
    }
    if (hasTimedOut())
        return WaitHandle.WaitTimeout;
}

This does work, WM_TEST gets pumped. Below is an adapted version of your test:

public static async Task RunAsync()
{
    using (var staThread = new Noseratio.ThreadAffinity.ThreadWithAffinityContext(staThread: true, pumpMessages: true))
    {
        Console.WriteLine("Initial thread #" + Thread.CurrentThread.ManagedThreadId);
        await staThread.Run(async () =>
        {
            Console.WriteLine("On STA thread #" + Thread.CurrentThread.ManagedThreadId);
            // create a simple Win32 window
            IntPtr hwnd = CreateTestWindow();

            // Post some WM_TEST messages
            Console.WriteLine("Post some WM_TEST messages...");
            NativeMethods.PostMessage(hwnd, NativeMethods.WM_TEST, new IntPtr(1), IntPtr.Zero);
            NativeMethods.PostMessage(hwnd, NativeMethods.WM_TEST, new IntPtr(2), IntPtr.Zero);
            NativeMethods.PostMessage(hwnd, NativeMethods.WM_TEST, new IntPtr(3), IntPtr.Zero);
            Console.WriteLine("Press Enter to continue...");
            await ReadLineAsync();

            Console.WriteLine("After await, thread #" + Thread.CurrentThread.ManagedThreadId);
            Console.WriteLine("Pending messages in the queue: " + (NativeMethods.GetQueueStatus(0x1FF) >> 16 != 0));

            Console.WriteLine("Exiting STA thread #" + Thread.CurrentThread.ManagedThreadId);
        }, CancellationToken.None);
    }
    Console.WriteLine("Current thread #" + Thread.CurrentThread.ManagedThreadId);
}

The output:

Initial thread #9
On STA thread #10
Post some WM_TEST messages...
Press Enter to continue...
WM_TEST processed: 1
WM_TEST processed: 2
WM_TEST processed: 3

After await, thread #10
Pending messages in the queue: False
Exiting STA thread #10
Current thread #12
Press any key to exit

Note this implementation supports both the thread affinity (it stays on the thread #10 after await) and the message pumping. The full source code contains re-usable parts (ThreadAffinityTaskScheduler and ThreadWithAffinityContext) and is available here as self-contained console app. It hasn't been thoroughly tested, so use it at your own risk.

Does WaitForMultipleObjects cause problems in a thread creating STA COM objects?

Yes, problems are possible - see this KB article. Basically if your thread is an STA thread it should not call functions that can block for long periods of time since while the thread is blocked it doesn't pump and dispatch messages and this can prevent proper marshalling functioning.

Blocking method of an STA COM object is a design defect?

Yes, objects on single-threaded apartments are synchronized via messages and all calls to them are serialized in such way that no more than one method can be called on any such object at any moment of time (also no method can be called more than once at any moment of time). This is by design and is done to achieve a certain degree of thread-safety. Having a long running method by itself is not a big problem unless you introduce a deadlock. Yes, the callers will wait until their calls are run in turn.

Objects in the multi-threaded apartment are not synchronized via messages - all calls to such objects are done directly without synchronization so more than one or more methods can be called at any object at any moment of time in parallel and it's up to the object to ensure thread-safety. Methods of course can block inside, but care should be taken to not introduce a deadlock.

---

Related Topics