How to prevent [STAThread] from blocking the change to SetThreadDesktop()

I have an EXE that I want to be called from a service. This EXE has been developed in C#, but using COM interop, it calls into DLLs we have developed. Some of these DLLs use a lot of desktop heap because they have windows/menus/etc that are created but never seen. It is too much work to prevent that, believe me.

We have developed ATL EXE servers before that get called from a service. For the ATL service, we can create a new desktop with a larger heap using calls to CreateDesktopEx(). After the desktop is created we can then call SetThreadDesktop() to change the desktop. The ATL EXE stuff has been working for years and years.

Now comes a different C# EXE. We want to be able to do the same thing with the desktop heap. I wrote PInvoke functions and verified that they work to be able to create desktops and set them--as long as the thread is not an STA.

Now, the stuff I need to call needs to run in an Single Threaded Apartment (STA). So, for the Main() function of the EXE, I add a [STAThread] attribute. Things have been working fine for a year, but now we want to be able change the desktop/desktop heap.

In my STAThread, I can sucessfully create the Desktop using CreateDesktopEx(), but when I try to set the desktop using SetThreadDesktop(), then it fails with Win32 error code 170 which is documented in the SetThreadDesktop() documentation.

If any hooks or windows have been created on the thread, then the desktop cannot be changed. I assume that the effect of the STAThread attribute is to call CoInitialize/Ex() under the hood, and that CoInitialize/Ex() creates an invisible window to handle incoming calls or sets up some kind of hook.

I've already tried creating a new thread calling Thread.SetApartmentState() with the STA value and running the thread, but I still can't change the desktop/desktop heap.

I want to know if there is anything I can do to delay this, or undo this so that I can set the desktop heap?

Ok... I added a minimal example. For references I have Serilog.dll and Serilog.Sinks.ColoredConsole.dll. From what I have been able to determine, if I comment out the code where I initialize the Serilog.Log.Logger, then the thread will allow the desktop heap to change. However, if the code is possibly in the call stack, it causes the worker thread to come up already initialized somehow and I cannot change the heap. For example, if there is a line like "if (SomeFalseConditional) InitializeLoger();" such that SomeFalseConditional is false and so InitializeLogger() never gets called, the worker thread is blocked from changing the desktop heap size. I will see if I can manage a workaround. It may or not be sufficient to put the initialization in the main thread. IDK. But, it seems strange that the worker thread gets initialized such that calls to SetThreadDesktop() fail.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using System.Runtime.InteropServices;
using System.Security.Principal;
using System.Reflection;

using System.ComponentModel;
using static System.Console;
using System.Threading;
using Serilog;


namespace DesktopHeapDemo
{
    [StructLayout(LayoutKind.Sequential)]
    public struct SECURITY_ATTRIBUTES
    {
        public int nLength;
        public IntPtr lpSecurityDescriptor;
        public int bInheritHandle;
    }

    [Flags]
    public enum ACCESS_MASK : uint
    {
        GENERIC_ALL = 0x10000000
    }

    public enum UserObjectInformationType
    {
        UOI_FLAGS = 1,
        UOI_NAME = 2,
        UOI_TYPE = 3,
        UOI_USER_SID = 4,
        UOI_HEAPSIZE = 5,
        UOI_IO = 6,
    }

    class DesktopHeap
    {
        const uint SDF_ALLOWOTHERACCOUNTHOOK = 1;

        [DllImport("user32.dll", SetLastError = true, CharSet = CharSet.Auto)]
        public static extern IntPtr CreateDesktopEx(string lpszDesktop, [Optional] string lpszDevice, [Optional] IntPtr pDevmode, [Optional] uint dwFlags, ACCESS_MASK dwDesiredAccess, [Optional] IntPtr lpSecurityAttributes, uint ulHeapSize, [Optional] IntPtr pvoid);

        [DllImport("user32.dll", SetLastError = true)]
        [return: MarshalAs(UnmanagedType.Bool)]
        public static extern bool CloseDesktop(IntPtr hDesktop);

        [DllImport("user32.dll", SetLastError = true, ExactSpelling = true)]
        [return: MarshalAs(UnmanagedType.Bool)]
        public static extern bool SetThreadDesktop(IntPtr hDesktop);

        [DllImport("user32.dll", SetLastError = true, ExactSpelling = true)]
        public static extern IntPtr GetThreadDesktop(uint dwThreadId);

        [DllImport("user32.dll", SetLastError = true, CharSet = CharSet.Auto)]
        [return: MarshalAs(UnmanagedType.Bool)]
        public static extern bool GetUserObjectInformation(IntPtr hObj, UserObjectInformationType nIndex, IntPtr pvInfo, uint nLength, out uint lpnLengthNeeded);

        [DllImport("kernel32.dll")]
        public static extern uint GetCurrentThreadId();

        [DllImport("user32.dll", SetLastError = true)]
        public static extern IntPtr GetProcessWindowStation();

        public static IntPtr GetCurrentDesktopHandle() { return GetThreadDesktop(GetCurrentThreadId()); }

        public static int GetCurrentDesktopHeapSize()
        {
            int lHeapSize = 0;
            IntPtr hGlobal = Marshal.AllocHGlobal(4);

            uint nNeeded;
            if (GetUserObjectInformation(GetCurrentDesktopHandle(), UserObjectInformationType.UOI_HEAPSIZE, hGlobal, 4, out nNeeded))
            {
                lHeapSize = Marshal.ReadInt32(hGlobal);
            }

            Marshal.FreeHGlobal(hGlobal);
            return lHeapSize;
        }

        public static string GetCurrentDesktop()
        {
            string sRet = string.Empty;
            uint nNeeded;
            IntPtr hDesktop = GetCurrentDesktopHandle();
            GetUserObjectInformation(hDesktop, UserObjectInformationType.UOI_NAME, IntPtr.Zero, 0, out nNeeded);

            IntPtr hGlobal = Marshal.AllocHGlobal((int)nNeeded);
            if (GetUserObjectInformation(hDesktop, UserObjectInformationType.UOI_NAME, hGlobal, nNeeded, out nNeeded))
            {
                sRet = Marshal.PtrToStringAuto(hGlobal);
            }
            Marshal.FreeHGlobal(hGlobal);

            return sRet;
        }

        public static string GetCurrentWindowStation()
        {
            string sRet = string.Empty;
            uint nNeeded;
            IntPtr hWindowStation = GetProcessWindowStation();
            GetUserObjectInformation(hWindowStation, UserObjectInformationType.UOI_NAME, IntPtr.Zero, 0, out nNeeded);
            IntPtr hGlobal = Marshal.AllocHGlobal((int)nNeeded);
            if (GetUserObjectInformation(hWindowStation, UserObjectInformationType.UOI_NAME, hGlobal, nNeeded, out nNeeded))
            {
                sRet = Marshal.PtrToStringAuto(hGlobal);
            }
            Marshal.FreeHGlobal(hGlobal);

            return sRet;
        }


        public static IntPtr CreateDesktop(uint nKiloBytes, ACCESS_MASK dwDesiredAccess = ACCESS_MASK.GENERIC_ALL)
        {
            string sName = string.Format($"StaCLI-{System.Diagnostics.Process.GetCurrentProcess().Id}-{GetCurrentThreadId()}");
            WriteLine($"CreateDesktopEx : {sName}");
            IntPtr hNewDesktop = CreateDesktopEx(lpszDesktop: sName, ulHeapSize: nKiloBytes, dwDesiredAccess: ACCESS_MASK.GENERIC_ALL, lpSecurityAttributes: IntPtr.Zero);
            if (hNewDesktop == IntPtr.Zero)
            {
                var win32error = Marshal.GetLastWin32Error();
                var msg = new Win32Exception(win32error).Message;
                WriteLine($"Win32 error calling CreateDesktopEx ({win32error}): { msg }");
            }

            return hNewDesktop;
        }

        public static IntPtr CreateAndSelectDesktop(uint nKiloBytes)
        {
            IntPtr hNewDesktop = CreateDesktop(nKiloBytes);

            if (hNewDesktop != IntPtr.Zero)
            {
                if (!SetThreadDesktop(hNewDesktop))
                {
                    var win32error = Marshal.GetLastWin32Error();
                    var msg = new Win32Exception(win32error).Message;
                    WriteLine($"Win32 error calling SetThreadDesktop ({win32error}): { msg }");
                    CloseDesktop(hNewDesktop);
                    hNewDesktop = IntPtr.Zero;
                }
                else
                {
                    WriteLine("Succeeded: SetThreadDesktop");
                }
            }

            return hNewDesktop;
        }
    }

    class Program
    {
        static void AdjustDesktopHeap(string sThreadName, int nDesiredHeapSize = 5120)
        {
            int nCurrentHeap = DesktopHeap.GetCurrentDesktopHeapSize();
            IntPtr hDesktop = DesktopHeap.GetCurrentDesktopHandle();
            IntPtr hNewDesktop = IntPtr.Zero;

            if (nCurrentHeap != nDesiredHeapSize)
            {
                WriteLine("Going to create and select desktop");
                hNewDesktop = DesktopHeap.CreateAndSelectDesktop((uint)5120);
                WriteLine($"Current desktop name: {DesktopHeap.GetCurrentDesktop()}");
                WriteLine($"Current desktop heap size: {DesktopHeap.GetCurrentDesktopHeapSize()}");
            }

            if (hNewDesktop != IntPtr.Zero)
            {
                DesktopHeap.SetThreadDesktop(hDesktop);
                DesktopHeap.CloseDesktop(hNewDesktop);
            }
            else
            {
                WriteLine($"Desktop heap was not changed in thread {sThreadName}");
            }
        }

        [STAThread]
        static void Main(string[] args)
        {
            int nCurrentHeap = DesktopHeap.GetCurrentDesktopHeapSize();
            WriteLine($"Current default desktop heap size: {nCurrentHeap}");

            string sCurrentDesktopName = DesktopHeap.GetCurrentDesktop();
            WriteLine($"Current desktop name: {sCurrentDesktopName}");
            WriteLine($"Current window station: {DesktopHeap.GetCurrentWindowStation()}");

            AdjustDesktopHeap("Main"); // works as long as main isn't [STAThread]

            Thread thread = new Thread(WorkerProc);
            thread.SetApartmentState(ApartmentState.STA);
            thread.Start();
            thread.Join();
        }



        static void WorkerProc()
        {
            InitializeLogger();
            WriteLine();
            WriteLine("In WorkerProc");
            AdjustDesktopHeap("WorkerProc");

            CreateFromProgId();

        }

        static string ProgId = "Word.Document";

        static void CreateFromProgId()
        {
            Type t = Type.GetTypeFromProgID(ProgId);
            var obj = Activator.CreateInstance(t);

            var name = t.InvokeMember("Name", BindingFlags.GetProperty, null, obj, null).ToString();
            WriteLine($"obj.Name = {name}");
        }

        static void InitializeLogger()
        {
            Log.Logger = new LoggerConfiguration().MinimumLevel.Error().WriteTo.ColoredConsole().CreateLogger();
        }

    }
}