How to handle 'Not enough quota is available to process this command' exception while closing an App after an extended period of time

I have a software that opens a 3rd party application. For them to communicate with each other I have a bridge solution (just a fake wpf window which is never displayed) which sends messages between each other and manages XML files with simple data. Everything works fine until the application is opened for more than 10-15 minutes (it can even be on idle) and then I try to close it - then my software crashes with the 'not enough quota exception'. The 3rd party app sometimes remains opened and still working while my software crashes.

I've been looking up for the solution for the past 3 days. Nothing's been helpful. The 'catch Win32Exception' doesn't catch it. Changing paging file in virtual memory doesn't help. It's reproducible on multiple stations with various specs - every station has more than enough resources. The memory/disk usage isn't high at all. The 3rd party app is not flooding my software with messages, it's sending 1-2 messages tops while closing. The communication is synchronous. I've tried lookign at BaseCompatibilityPreferences.HandleDispatcherRequestProcessingFailure which was posted here: https://github.com/dotnet/wpf/issues/137 - to no avail. The pointers to the wpf window are not null. I also checked the threading.

Exception details:

System.ComponentModel.Win32Exception
  HResult=0x80004005
  Message=Not enough quota is available to process this command
  Source=WindowsBase
  StackTrace:
   at MS.Win32.UnsafeNativeMethods.PostMessage(HandleRef hwnd, WindowMessage msg, IntPtr wparam, IntPtr lparam)
   at System.Windows.Interop.HwndTarget.UpdateWindowSettings(Boolean enableRenderTarget, Nullable`1 channelSet)
   at System.Windows.Interop.HwndTarget.UpdateWindowPos(IntPtr lParam)
   at System.Windows.Interop.HwndTarget.OnWindowPosChanged(IntPtr lParam)
   at System.Windows.Interop.HwndTarget.HandleMessage(WindowMessage msg, IntPtr wparam, IntPtr lparam)
   at System.Windows.Interop.HwndSource.HwndTargetFilterMessage(IntPtr hwnd, Int32 msg, IntPtr wParam, IntPtr lParam, Boolean& handled)
   at MS.Win32.HwndWrapper.WndProc(IntPtr hwnd, Int32 msg, IntPtr wParam, IntPtr lParam, Boolean& handled)
   at MS.Win32.HwndSubclass.DispatcherCallbackOperation(Object o)
   at System.Windows.Threading.ExceptionWrapper.InternalRealCall(Delegate callback, Object args, Int32 numArgs)
   at System.Windows.Threading.ExceptionWrapper.TryCatchWhen(Object source, Delegate callback, Object args, Int32 numArgs, Delegate catchHandler)
   at System.Windows.Threading.Dispatcher.LegacyInvokeImpl(DispatcherPriority priority, TimeSpan timeout, Delegate method, Object args, Int32 numArgs)
   at MS.Win32.HwndSubclass.SubclassWndProc(IntPtr hwnd, Int32 msg, IntPtr wParam, IntPtr lParam)

My app only crashes after the 3rd party app is opened for more than 10-15 minutes and it's almost always reproducible. I can do a ton of operations within 10 minutes, close the app and everything works fine but I can just open the app, do nothing for 15mins and it crashes my software while closing the app.

Any idea what debugging strategy I could use to diagnose the problem? Any additional data I can provide?

EDIT:

  private bool StartApplication()
  {

     SharedMemory sharedMemory = SharedMemory.GetInstance();
     if (sharedMemory == null)
     {
        Int32 memorySize = Convert.ToInt32(Properties.Resources.SharedMemoryInitialSize);
        SharedMemory.CreateSharedMemory(memorySize);
        sharedMemory = SharedMemory.GetInstance();
     }

     if (sharedMemory == null)
     {
        SeeLogWrapper.Log(Panel3DStarterCommonUtils.Properties.Messages.MemoryManagerNotFound);
        Debug.Assert(false, Panel3DStarterCommonUtils.Properties.Messages.MemoryManagerNotFound);
        return false;
     }

     string initData = CreateInitDataForProcess();

     if (string.IsNullOrEmpty(initData))
        return false;

     Process proc = new Process();
     proc.StartInfo.FileName = StarterExePath;
     proc.StartInfo.Arguments = initData;

     StarterProc = proc;

     try
     {
        proc.Start();
        proc.WaitForExit();

        if (MessageWindow != null)
        {
           MessageWindow.Dispatcher.Invoke(() =>
           {
              MessageWindow.Close(); //This is usually where I get the not enough quota exception
           });
        }

        MessageWindow = null;
        proc.Close();

     }
     //catch phrases + return
  }


  private void Initialize(string A_starterPath, Iproc A_starterInterface, string A_loggerId, IntPtr A_parentHandle)
  {
     LoggerID = "";
     if (string.IsNullOrEmpty(A_loggerId) == false)
     {
        LoggerID = A_loggerId;
        Logger.Initialize(A_loggerId);
     }

     StarterExePath = A_starterPath;
     Panel3DStarterInterface = A_starterInterface;

     MainInstance = this;

     MessageWindow = new Window()
     {
        Width = 0,
        Height = 0,
        WindowStyle = WindowStyle.None,
        ShowInTaskbar = false,
        ShowActivated = false,
        WindowStartupLocation = WindowStartupLocation.Manual,
        Left = -2000,
        Top = -2000
     };

     MessageWindow.Closing += MessageWindowClosing;

     WindowHelper = new WindowInteropHelper(MessageWindow);
     WindowHelper.Owner = A_parentHandle;

     MessageWindow.Show();

     HwndSource messageWindowSource = HwndSource.FromVisual(MessageWindow) as HwndSource;
     messageWindowSource.AddHook(WndProc);
  }

We just had the same sort of issue with one of our applications. After much research and trial and error, the problem boiled down to blocking the main thread with Process.WaitForExit(). We made a simple change to open a new thread and have it WaitForExit() instead. The main thread then enters a loop to wait for that thread to close:

/* ----- In the Main Thread ----- */
Thread t = new Thread(() => waitForProcessClose(newProcess));
t.Start();
while (t.IsAlive)
{
    Thread.Sleep(500);                         
}
/* ----- In the Main Thread ----- */


public static bool waitForProcessClose(Process handleToProcess)
{
    handleToProcess.WaitForExit();

    return true;
}

Doing it this way seems to give the main thread a chance to clear it's message queue every interval, so it doesn't stack up and crash anymore.

Maybe your issue doesn't involve the main thread, but perhaps one of your threads is blocking a message queue in a similar fashion.