How to prevent re-entrancy of WPF event handler during ActiveX method call?

We're calling methods on an ActiveX component from within a WPF and STA application. This calling is late-bound performed via:

res = ocx.GetType().InvokeMember(methodName, flags, null, ocx, args);

...where ocx is the ActiveX object retrieved with the System.Windows.Forms.AxHost.GetOcx() method.

This call is performed from within a WPF event handler, say 'mouse clicked'.

Now the problem. If we double-click the 'mouse clicked' event will trigger, running InvokeMember(). However, during this call, we see that the 'mouse clicked' event is re-entered. So in the same thread, we see the event handler twice on the call stack. This is very unexpected, and we're trying to prevent that. How can we prevent this from happening?

The only reason we can think of why it happens is:

• The COM object is created in another STA, so we're performing a cross-STA call which needs to be marshalled
• cross-thread STA calls uses a Windows Message to send an RPC request to the COM component
• cross-thread STA calls use the Windows Message pump to receive RPC answer
• During the waiting another type of event comes in (like 'mouse clicked'), and this gets handled before the RPC answer gets handled.
• This RPC answer gets handled

Things we tried to fix the problem:

• use lock() in all event handlers. This does not work since lock() will lock a thread, and in this case it is the same thread which re-enters the event handler.
• use custom locking like 'bool locked = false; if (!locked) { locked = true; InvokeMethod(); ...; locked = false; }'. This works partially: it throws away the events instead of queuing them for later, and need an extensive change to all our event handlers, which is not nice to do.
• use Dispatcher.DisableProcessing to stop (other) messages from being processed. This does not help: it throws an exception because of messages being processed anyways.
• create a second dispatcher in a new thread, and run ocx.InvokeMehod() via Dispatcher.Invoke() to have it handled by another thread. This gives 'An event was unable to invoke any of the subscribers (Exception from HRESULT: 0x80040201)' (Yes, we're also subscribed to COM events of the ActiveX object).
• use Dispatcher.PushFrame() to stop event handling from happening. This also fails.

A wild idea which might work, but don't know how to implement this would be creating a new message pump as the WPF message pump which can be configured to temporarily handle only RPC calls. This is along the lines of http://jmorrill.hjtcentral.com/Home/tabid/428/EntryId/430/WPF-MediaKit-Updates.aspx , but still somewhat different from this situation.

So the question boils down to how can we make ActiveX call synchronously like we expected it already to be instead of async?

Update

To make it more clear that the mechanism involved is not only about mouse-events, but the more generic problem of 'a new event is handled while the old is being executed', I'll give another example with a stack trace:

Context: we've got a WPF Grid on which we get a mouseclick (Grid_MouseDown), we've got an ActiveX object on which we perform the method 'CloseShelf'. Opening the shelf will take time, so we are subscribed to the event 'EventShelfClosed', which in the event handler of EventShelfClosed will call 'ListShelf' to know which shelfs are left.

This is how the managed stack trace looks like (Hans asked for an unmanaged stacktrace, but I don't know how to get one):

MyAxWrapper.dll!MyAxWrapper.MyAxWrapper.InvokeMember(string methodName, System.Reflection.BindingFlags flags, object[] args, int refArgIdx) Line 53 C#
MyAxWrapper.dll!MyAxWrapper.LoggingMyAxWrapper.InvokeMember(string methodName, System.Reflection.BindingFlags flags, object[] args, int refArgIdx) Line 151 + 0x14 bytes    C#
MyAxWrapper.dll!MyAxWrapper.MyAxWrapper.InvokeMethod(string methodName, object[] args) Line 92 + 0x18 bytes C#
MyAxWrapper.dll!MyAxWrapper.MyAxAppWrapper.ListShelfs(string CanvasPageId) Line 300 + 0x42 bytes    C#
PACS.dll!PACS.MyAxDatabase.GetShelfIdsOn(string canvasPageId) Line 223 + 0xf bytes  C#
MyAxCanvas.dll!MyAxCanvas.MyAxCanvasPlugin.UpdateTimeLineSelection(string canvasPageId) Line 123 + 0x10 bytes   C#
MyAxCanvas.dll!MyAxCanvas.MyAxCanvasPlugin.EventShelfClosed(string canvasPageId, string shelfId) Line 180 + 0xb bytes   C#
MyAxWrapper.dll!MyAxWrapper.MyAxAppWrapper.FireEvent(string eventName, object[] args) Line 21 + 0x73 bytes  C#
MyAxWrapper.dll!MyAxWrapper.MyAxEventForwarder.EventShelfClosed(string CanvasPageID, string ShelfID) Line 177 + 0x58 bytes  C#
[Native to Managed Transition]  
[Native to Managed Transition]  
MyAxWrapper.dll!MyAxWrapper.MyAxWrapper.InvokeMember(string methodName, System.Reflection.BindingFlags flags, object[] args, int refArgIdx) Line 75 + 0x2b bytes    C#
MyAxWrapper.dll!MyAxWrapper.LoggingMyAxWrapper.InvokeMember(string methodName, System.Reflection.BindingFlags flags, object[] args, int refArgIdx) Line 151 + 0x14 bytes    C#
MyAxWrapper.dll!MyAxWrapper.MyAxWrapper.InvokeMethod(string methodName, object[] args) Line 92 + 0x18 bytes C#
MyAxWrapper.dll!MyAxWrapper.MyAxAppWrapper.CloseShelf(string a) Line 218 + 0x42 bytes   C#
MyAxCanvas.dll!MyAxCanvas.MyAxCanvasPlugin.EventCanvasPageCreated.AnonymousMethod__0(DataModel.Item exam) Line 110 + 0x1d bytes C#
ItemPresenter.dll!ItemPresenter.ItemPresenter.OnItemClicked(DataModel.Item study) Line 36 + 0x14 bytes  C#
ItemPresenter.dll!ItemPresenter.ItemPresenter.ItemPresenterPerYearControls_Click(object sender, System.Windows.RoutedEventArgs e) Line 215 + 0x1e bytes C#
PresentationCore.dll!System.Windows.RoutedEventHandlerInfo.InvokeHandler(object target, System.Windows.RoutedEventArgs routedEventArgs) + 0x78 bytes    
PresentationCore.dll!System.Windows.EventRoute.InvokeHandlersImpl(object source, System.Windows.RoutedEventArgs args, bool reRaised) + 0x1ae bytes  
PresentationCore.dll!System.Windows.UIElement.RaiseEventImpl(System.Windows.DependencyObject sender, System.Windows.RoutedEventArgs args) + 0x79 bytes  
PresentationCore.dll!System.Windows.UIElement.RaiseEvent(System.Windows.RoutedEventArgs e) + 0x17 bytes 
ItemPresenter.dll!ItemPresenter.ItemPresenterControl.Grid_MouseDown(object sender, System.Windows.Input.MouseButtonEventArgs e) Line 47 + 0x29 bytes    C#
PresentationCore.dll!System.Windows.Input.MouseButtonEventArgs.InvokeEventHandler(System.Delegate genericHandler, object genericTarget) + 0x31 bytes    
PresentationCore.dll!System.Windows.RoutedEventArgs.InvokeHandler(System.Delegate handler, object target) + 0x29 bytes  
PresentationCore.dll!System.Windows.RoutedEventHandlerInfo.InvokeHandler(object target, System.Windows.RoutedEventArgs routedEventArgs) + 0x3e bytes    
PresentationCore.dll!System.Windows.EventRoute.InvokeHandlersImpl(object source, System.Windows.RoutedEventArgs args, bool reRaised) + 0x1ae bytes  
PresentationCore.dll!System.Windows.UIElement.RaiseEventImpl(System.Windows.DependencyObject sender, System.Windows.RoutedEventArgs args) + 0x79 bytes  
PresentationCore.dll!System.Windows.UIElement.RaiseTrustedEvent(System.Windows.RoutedEventArgs args) + 0x41 bytes   
PresentationCore.dll!System.Windows.UIElement.RaiseEvent(System.Windows.RoutedEventArgs args, bool trusted) + 0x2c bytes    
PresentationCore.dll!System.Windows.Input.InputManager.ProcessStagingArea() + 0x1ff bytes   
PresentationCore.dll!System.Windows.Input.InputManager.ProcessInput(System.Windows.Input.InputEventArgs input) + 0x45 bytes 
PresentationCore.dll!System.Windows.Input.InputProviderSite.ReportInput(System.Windows.Input.InputReport inputReport) + 0x62 bytes  
PresentationCore.dll!System.Windows.Interop.HwndMouseInputProvider.ReportInput(System.IntPtr hwnd, System.Windows.Input.InputMode mode, int timestamp, System.Windows.Input.RawMouseActions actions, int x, int y, int wheel) + 0x263 bytes 
PresentationCore.dll!System.Windows.Interop.HwndMouseInputProvider.FilterMessage(System.IntPtr hwnd, MS.Internal.Interop.WindowMessage msg, System.IntPtr wParam, System.IntPtr lParam, ref bool handled) + 0x46d bytes 
PresentationCore.dll!System.Windows.Interop.HwndSource.InputFilterMessage(System.IntPtr hwnd, int msg, System.IntPtr wParam, System.IntPtr lParam, ref bool handled) + 0x75 bytes   
WindowsBase.dll!MS.Win32.HwndWrapper.WndProc(System.IntPtr hwnd, int msg, System.IntPtr wParam, System.IntPtr lParam, ref bool handled) + 0xbe bytes    
WindowsBase.dll!MS.Win32.HwndSubclass.DispatcherCallbackOperation(object o) + 0x7d bytes    
WindowsBase.dll!System.Windows.Threading.ExceptionWrapper.InternalRealCall(System.Delegate callback, object args, int numArgs) + 0x53 bytes 
WindowsBase.dll!MS.Internal.Threading.ExceptionFilterHelper.TryCatchWhen(object source, System.Delegate method, object args, int numArgs, System.Delegate catchHandler) + 0x42 bytes    
WindowsBase.dll!System.Windows.Threading.Dispatcher.InvokeImpl(System.Windows.Threading.DispatcherPriority priority, System.TimeSpan timeout, System.Delegate method, object args, int numArgs) + 0xb4 bytes    
WindowsBase.dll!MS.Win32.HwndSubclass.SubclassWndProc(System.IntPtr hwnd, int msg, System.IntPtr wParam, System.IntPtr lParam) + 0x104 bytes    
[Native to Managed Transition]  
[Managed to Native Transition]  
WindowsBase.dll!System.Windows.Threading.Dispatcher.PushFrameImpl(System.Windows.Threading.DispatcherFrame frame) + 0xc1 bytes  
WindowsBase.dll!System.Windows.Threading.Dispatcher.PushFrame(System.Windows.Threading.DispatcherFrame frame) + 0x49 bytes  
WindowsBase.dll!System.Windows.Threading.Dispatcher.Run() + 0x4c bytes  
PresentationFramework.dll!System.Windows.Application.RunDispatcher(object ignore) + 0x17 bytes  
PresentationFramework.dll!System.Windows.Application.RunInternal(System.Windows.Window window) + 0x6f bytes 
PresentationFramework.dll!System.Windows.Application.Run(System.Windows.Window window) + 0x26 bytes 
PresentationFramework.dll!System.Windows.Application.Run() + 0x1b bytes 
MyAxCanvasStandalone.exe!MyAxCanvasStandalone.App.Main(string[] args) Line 37 + 0xa bytes   C#
[Native to Managed Transition]  
[Managed to Native Transition]  
mscorlib.dll!System.AppDomain.ExecuteAssembly(string assemblyFile, System.Security.Policy.Evidence assemblySecurity, string[] args) + 0x6d bytes    
Microsoft.VisualStudio.HostingProcess.Utilities.dll!Microsoft.VisualStudio.HostingProcess.HostProc.RunUsersAssembly() + 0x2a bytes  
mscorlib.dll!System.Threading.ThreadHelper.ThreadStart_Context(object state) + 0x63 bytes   
mscorlib.dll!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool ignoreSyncCtx) + 0xb0 bytes    
mscorlib.dll!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state) + 0x2c bytes    
mscorlib.dll!System.Threading.ThreadHelper.ThreadStart() + 0x44 bytes   
[Native to Managed Transition]  

What happens is the the method 'CloseShelf' will close the shelf, but in this case 'CloseShelf' is so fast that the event 'EventShelfClosed' is emitted and handled during the call to CloseShelf. Now CloseShelf will call ListShelfs, but ListShelfs will fail and return null because the ActiveX component is locked by the 'CloseShelf' call which is still active.

Why is this a problem? Becasue the programmer does not expect a method call to be async. This hit us after creating a large program, which now means auditting all calls for unexpected behaviour.

What would we like to see in this case? We would like to see that 'CloseShelf' returns without handling other events during the call. The method should be synchronous, and any pending events handled during the main (non-recursive) message loop.

Having a 'lock' kind of boolean won't help here, since we would be missing events here, which locks up the application.

<tldr> try the code below in Attempt #3. There was nothing on TV when I sat down to write this.</tldr>

Thanks for the clarification! I see that there is only one thread here; and also, since the CloseShelf frame is still on the stack, it looks like the COM call is actually blocking.

From the stack trace, it looks like the com object is calling GetMessage or PeekMessage API (or if it's VB6, DoEvents or similar) which will check the message queue and PROCESS messages on it - irregaurdless of if it will cause re-entrancy. AKA 'pumping the message queue' - but if it uses peekmessage, it won't block if there are no messages, but will still execute the messages. In your stacktrack these calls could be hidden in the invisible native part. The stack trace actually proves that somehow, the COM call is calling back into your code! It looks like you are aware of this. If it's a bit foggy for some reader here are a couple links:

http://discuss.joelonsoftware.com/default.asp?joel.3.456478.15
http://en.wikipedia.org/wiki/Event_loop

Cooperative multitasking (one message loop for the whole OS like win3.0): http://en.wikipedia.org/wiki/Computer_multitasking#Cooperative_multitasking.2Ftime-sharing

The quote 'voluntarily ceded time...' is actually done by these calls. And it STILL happens all the time on the GUI thread of every windows app!

Since the actual call is in the COM, if you can't edit it, you still have to code around it. It is probably just this one method (hopefully).

Sometimes programs and components check the message loop on purpose, to times to allow things to respond, or repaint the screen. Just like this poster is trying to do:

Is there a function like PeekMessage that doesn't process messages?

The quote about 'The system may also process internal events' is buring you here.

note where you say 'the programmer does not expect a method call to be async' - it isn't async, or the stack trace would look different. it's 'recursing' back into your code. As an old Win3.1 programmer this was the hell we dealt with for EVERY app in the system!

I found that link googling for 'check the message queue peekmessage' while trying to see if Get/Peekmessage, etc. could be prevented from processing messages. You can see from this documention...

http://msdn.microsoft.com/en-us/library/windows/desktop/ms644943(v=vs.85).aspx

...that sending something like PM_QS_INPUT | PM_QS_PAINT would prevent the problem. Unfortunately, since you aren't calling it, you can't change it! And I didn't see any way to set a flag to control subsequent message processing from later calls.

If a reader is still confused (if not skip this code) ... For a simple example, make a VB Winforms App and make one button and double click it and paste this code in - (I say VB because application.doevents is the handiest way to invoke this nasty message queue check):

    For i As Integer = 0 To 20
        Text = i.ToString
        System.Threading.Thread.Sleep(100)
        Application.DoEvents()
    Next

Now click the button. Note you can enlarge the window and the background repaints - as the doevents allows these events to happen by checking the message queue (REM out the doevents and it will 'wait' untill the count is done; also try clicking 3x and you will get 3 counts in a row).

NOW... the kicker. Click the button w/ Doevents NOT commented out. Click the button 3 times - the countdowns interrupt each other, then resume as the previous one finishes. You can Pause the IDE and see 3 click callstacks.

Yummy!

I also see that you tried a couple things to stop messages or events from being processed. If the code that fires the EventShelfClosed is getting called from the re-entrancy caused by a PeekMessage or 'Doevents', however, that may not effect it.

Note this practice has it's detractors: http://www.codinghorror.com/blog/2005/08/is-doevents-evil-revisited.html

Best would be to change the COM so it doesn't make any API calls that are checking the message loop.

Good luck with that!

Another way to change it would be to move from events controlling EventShelfClosed, and call it explicity after the call to CloseShelf is exited (since the com call is really happening). Unfortunately your program's architechture probably won't allow that without major changes and/or increased cohesion and other stuff that dirties pretty models (not fashion models mind you :-).

A different way would be to make a new thread object pointed to a function that makes the com invoke, then start it, then join it, in hopes that anything like PeekMessage would not find a message pump on the new thread and therefore not interfere with things. It looks like a couple of you attempts involved this type of thing. Unfortunately if the COM Peeks for messages, and there is no message pump on the thread, kaboom. It will probably blow up instead of just ignoring things. Sounds like that's what happened. Further more, if the COM relies on other items that should only be accessed from the GUI/messagepump thread, you're in trouble with cross-thread calls (which would certainly be the case if the COM interacts with the UI or any UI objects).

If you can't stop the message queue from being checked, or prevent EventShelfClosed from firing until later, you have no choice but to let the EventShelfClosed get called. But what you can do is cause it to wait, and then fall thru when the CloseShelf is finished.

So you still have to have a class-level boolean field set/unset by CloseShelf so EventShelfClosed will know that it is running.

Unfortunately just checking this in a while loop, even with a sleep, will block the single thread you have, and freeze the app. You may just try to have EventShelfClosed re-raise itself and exit the function as long as the bool is set; but since RaiseEvent stays inside of managed, runs the code immediately, and does not check the message queue it will in crash with a stackoverflow. If you can figure out how to re-raise EventShelfClosed by calling the PostMessage API (not SendMessage, that runs it right away) - that will keep putting on the GUI thread's message queue as many times as the COM call will make windows check it. Unless said COM waits for the queue to be empty for some stupid reason - another lockup. Not recommending.

Soo... You can fight fire with fire. Here I am implementing another message-checking loop to allow your events to happen while you wait for the bool to clear.

Note this is only going to fix this one problem in this one case. Auditing all calls ... this isn't a magic bullet. My guess is there is none. Very messy and this is a total hack.

Attempt #3

It's not really attempt #3, it's more like possibility #8. But I referenced this in my old answer and am too lazy to edit that.

Boolean InCloseShelf
function CloseShelf(...)
    InCloseShelf=True;
    try
    {
         com call and all else
     }
     finally
         InCloseShelf=False

function EventShelfClosed(...
    while (InCloseShelf)
    {
         DoEvents
     }

Now of course there is no DoEvents in WPF, it's in winforms' 'application'. This blog has an implementation

http://dedjo.blogspot.com/2007/08/how-to-doevents-in-wpf.html

void DoEvents(){ 
DispatcherFrame f = new DispatcherFrame(); 
Dispatcher.CurrentDispatcher.BeginInvoke(DispatcherPriority.Background,  
(SendOrPostCallback)delegate(object arg) { 
    DispatcherFrame fr =  arg as DispatcherFrame; 
    fr.Continue=True; 
}, f); 
Dispatcher.PushFrame(frame); 
}

Untested, of course! Note this is from the correction in the comments:

static void DoEvents()
{
DispatcherFrame frame = new DispatcherFrame(true);
Dispatcher.CurrentDispatcher.BeginInvoke
(
DispatcherPriority.Background, 
(SendOrPostCallback) delegate(object arg)
{
var f = arg as DispatcherFrame; 
f.Continue = false;
}, 
frame
);
Dispatcher.PushFrame(frame);
} 

Or you could always reference WinForms and call Application.DoEvents.

I'm guessing you already know this, but you're in a bad spot right now. If this doesn't do it, Good luck! If you find a better way please update the post, because, well, evil hacks like this suck, but now you can see why people say to check the message queue sparingly!