CoInitializeSecurity throws RPC_E_TOO_LATE in Visual Studio 2017

▲
3
▼

I'm trying to run an application making a call to CoInitializeSecurity at startup. This works in Visual Studio 2013, but does not work in Visual Studio 2017 - and I'm curious to why this is.

When calling CoInitializeSecurity at startup in Visual Studio 2017 I get a COMException with the error code RPC_E_TOO_LATE (0x80010119) which indicates a call has already been made to CoInitialize, this does not happen in Visual Studio 2013.

I have seen this behaviour before in Visual Studio 2013 when the Visual Studio hosting process is enabled or when an assembly using COM is loaded before CoInitializeSecurity has been called.

The loaded assemblies differ between Visual Studio 2013 and 2017 (snapshot taken when entering the App constructor), differences highlighted:

2013:

'WPFTestVS2017.exe' (CLR v4.0.30319: DefaultDomain): Loaded 'C:\windows\Microsoft.Net\assembly\GAC_64\mscorlib\v4.0_4.0.0.0__b77a5c561934e089\mscorlib.dll'.

'WPFTestVS2017.exe' (CLR v4.0.30319: DefaultDomain): Loaded 'C:\WPFTestVS2017\bin\Debug\WPFTestVS2017.exe'.

'WPFTestVS2017.exe' (CLR v4.0.30319: WPFTestVS2017.exe): Loaded 'C:\windows\Microsoft.Net\assembly\GAC_MSIL\PresentationFramework\v4.0_4.0.0.0__31bf3856ad364e35\PresentationFramework.dll'.

'WPFTestVS2017.exe' (CLR v4.0.30319: WPFTestVS2017.exe): Loaded 'C:\windows\Microsoft.Net\assembly\GAC_MSIL\WindowsBase\v4.0_4.0.0.0__31bf3856ad364e35\WindowsBase.dll'.

'WPFTestVS2017.exe' (CLR v4.0.30319: WPFTestVS2017.exe): Loaded 'C:\windows\Microsoft.Net\assembly\GAC_MSIL\System\v4.0_4.0.0.0__b77a5c561934e089\System.dll'.

'WPFTestVS2017.exe' (CLR v4.0.30319: WPFTestVS2017.exe): Loaded 'C:\windows\Microsoft.Net\assembly\GAC_64\PresentationCore\v4.0_4.0.0.0__31bf3856ad364e35\PresentationCore.dll'.

'WPFTestVS2017.exe' (CLR v4.0.30319: WPFTestVS2017.exe): Loaded 'C:\windows\Microsoft.Net\assembly\GAC_MSIL\System.Xaml\v4.0_4.0.0.0__b77a5c561934e089\System.Xaml.dll'.

'WPFTestVS2017.exe' (CLR v4.0.30319: WPFTestVS2017.exe): Loaded 'C:\windows\Microsoft.Net\assembly\GAC_MSIL\System.Configuration\v4.0_4.0.0.0__b03f5f7f11d50a3a\System.Configuration.dll'.

'WPFTestVS2017.exe' (CLR v4.0.30319: WPFTestVS2017.exe): Loaded 'C:\windows\Microsoft.Net\assembly\GAC_MSIL\System.Xml\v4.0_4.0.0.0__b77a5c561934e089\System.Xml.dll'.

2017:

'WPFTestVS2017.exe' (CLR v4.0.30319: DefaultDomain): Loaded 'C:\windows\Microsoft.Net\assembly\GAC_64\mscorlib\v4.0_4.0.0.0__b77a5c561934e089\mscorlib.dll'.

'WPFTestVS2017.exe' (CLR v4.0.30319: DefaultDomain): Loaded 'C:\WPFTestVS2017\bin\Debug\WPFTestVS2017.exe'.

'WPFTestVS2017.exe' (CLR v4.0.30319: WPFTestVS2017.exe): Loaded 'C:\windows\Microsoft.Net\assembly\GAC_MSIL\PresentationFramework\v4.0_4.0.0.0__31bf3856ad364e35\PresentationFramework.dll'.

'WPFTestVS2017.exe' (CLR v4.0.30319: WPFTestVS2017.exe): Loaded 'C:\windows\Microsoft.Net\assembly\GAC_MSIL\WindowsBase\v4.0_4.0.0.0__31bf3856ad364e35\WindowsBase.dll'.

'WPFTestVS2017.exe' (CLR v4.0.30319: WPFTestVS2017.exe): Loaded 'C:\windows\Microsoft.Net\assembly\GAC_MSIL\System.Core\v4.0_4.0.0.0__b77a5c561934e089\System.Core.dll'.

'WPFTestVS2017.exe' (CLR v4.0.30319: WPFTestVS2017.exe): Loaded 'C:\windows\Microsoft.Net\assembly\GAC_MSIL\System\v4.0_4.0.0.0__b77a5c561934e089\System.dll'.

'WPFTestVS2017.exe' (CLR v4.0.30319: WPFTestVS2017.exe): Loaded 'C:\windows\Microsoft.Net\assembly\GAC_64\PresentationCore\v4.0_4.0.0.0__31bf3856ad364e35\PresentationCore.dll'.

'WPFTestVS2017.exe' (CLR v4.0.30319: WPFTestVS2017.exe): Loaded 'C:\windows\Microsoft.Net\assembly\GAC_MSIL\System.Xaml\v4.0_4.0.0.0__b77a5c561934e089\System.Xaml.dll'.

'WPFTestVS2017.exe' (CLR v4.0.30319: WPFTestVS2017.exe): Loaded 'C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\Remote Debugger\x64\Runtime\Microsoft.VisualStudio.Debugger.Runtime.dll'.

'WPFTestVS2017.exe' (CLR v4.0.30319: WPFTestVS2017.exe): Loaded 'C:\windows\Microsoft.Net\assembly\GAC_MSIL\System.Configuration\v4.0_4.0.0.0__b03f5f7f11d50a3a\System.Configuration.dll'.

'WPFTestVS2017.exe' (CLR v4.0.30319: WPFTestVS2017.exe): Loaded 'C:\windows\Microsoft.Net\assembly\GAC_MSIL\System.Xml\v4.0_4.0.0.0__b77a5c561934e089\System.Xml.dll'.

The remote debugger makes me suspicious since it reminds me of the Visual Studio hosting process. The other line that differs is the System.Core.dll which doesn't appear in the loaded assemblies in VS2013.

Code:

App.xaml

<Application x:Class="WPFTestVS2017.App"
            xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
            xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
            StartupUri="MainWindow.xaml">
</Application>

App.xaml.cs

using System;
using System.Runtime.InteropServices;
using System.Windows;

namespace WPFTestVS2017
{
    internal static class NativeMethods
    {
        private enum RpcAuthnLevel
        {
            Default = 0,
            None = 1,
            Connect = 2,
            Call = 3,
            Pkt = 4,
            PktIntegrity = 5,
            PktPrivacy = 6
        }

        private enum RpcImpLevel
        {
            Default = 0,
            Anonymous = 1,
            Identify = 2,
            Impersonate = 3,
            Delegate = 4
        }

        private enum EoAuthnCap
        {
            None = 0x0000,
            MutualAuth = 0x0001,
            StaticCloaking = 0x0020,
            DynamicCloaking = 0x0040,
            AnyAuthority = 0x0080,
            MakeFullSIC = 0x0100,
            Default = 0x0800,
            SecureRefs = 0x0002,
            AccessControl = 0x0004,
            AppID = 0x0008,
            Dynamic = 0x0010,
            RequireFullSIC = 0x0200,
            AutoImpersonate = 0x0400,
            NoCustomMarshal = 0x2000,
            DisableAAA = 0x1000
        }

        [DllImport("Ole32.dll",
            ExactSpelling = true,
            EntryPoint = "CoInitializeSecurity",
            CallingConvention = CallingConvention.StdCall,
            SetLastError = false,
            PreserveSig = false)]
        private static extern void CoInitializeSecurity(
            IntPtr pVoid,
            int cAuthSvc,
            IntPtr asAuthSvc,
            IntPtr pReserved1,
            uint dwAuthnLevel,
            uint dwImpLevel,
            IntPtr pAuthList,
            uint dwCapabilities,
            IntPtr pReserved3);

        public static void Initialize()
        {
            CoInitializeSecurity(IntPtr.Zero,
                -1,
                IntPtr.Zero,
                IntPtr.Zero,
                (uint)RpcAuthnLevel.PktPrivacy,
                (uint)RpcImpLevel.Impersonate,
                IntPtr.Zero,
                (uint)EoAuthnCap.DynamicCloaking,
                IntPtr.Zero);
        }
    }

    public partial class App : Application
    {
        public App()
        {
            NativeMethods.Initialize();
        }
    }
}

MainWindow.xaml

<Window x:Class="WPFTestVS2017.MainWindow"
        xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
        xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"/>

MainWindow.xaml.cs

using System.Windows;

namespace WPFTestVS2017
{
    public partial class MainWindow : Window
    {
        public MainWindow()
        {
            InitializeComponent();
        }
    }
}

Edit:

I made the following modifications to App.xaml.cs:

public App()
{
    try
    {
        NativeMethods.Initialize();
    }
    catch (Exception e)
    {
        MessageBox.Show(e.ToString());
    }
}

The messagebox appears when debugging in Visual Studio 2017, however, it does not appear when running the same executable outside of Visual Studio.

c#
.net
visual-studio
com
visual-studio-2017
asked on Stack Overflow Jan 30, 2019 by sweerpotato • edited Jun 20, 2020 by Community

2 Answers

▲
5
▼

You are doing battle with the changes in the managed debugging engine in VS2017. It doesn't have anything to do with what you guessed at, I deem it somewhat likely that the removal of the Visual Studio Hosting Process option is related. Blind guess, this is a black box that's very hard to penetrate without assistance from somebody in the Microsoft debugger team.

You have several possible workarounds available, sorted by practicality:

  1. Tools > Options > Debugging > General, tick the "Use Managed Compatibility Mode" checkbox. This replaces the new debugging engine with the old one, last used in VS2010. You'll miss out on some recent debugger features (new PDB format, return value inspection, 64-bit Edit+Continue), little that should stop you from debugging a WPF app.

  2. If undesired, you can stop the function from throwing an exception. Change the [DllImport]'s PreserveSig property to true, change the return type from void to int. It will still fail, indicated by the negative return value, but you can keep motoring debugging the rest of your code. Perhaps you want to use the return value to set a global variable that you'd use to bypass the tricky COM code.

  3. If undesired, you can delay initializing the debugging engine until after the CoInitializeSecurity call. Append System.Diagnostics.Debugger.Launch();, wrapped with #if DEBUG. Now you can press Ctrl+F5 to start debugging, select the running instance of VS as the desired debugger when you get the prompt. Using Debug > Attach to Process is a similar workaround.

answered on Stack Overflow Feb 1, 2019 by Hans Passant • edited Sep 9, 2019 by ogggre
▲
0
▼

I finally managed to find a solution to this problem - the problem seems to originate from STAThread.

Switching the build action from ApplicationDefinition to Page in App.xaml's properties allows us to define our own Main method, instead of using the compiler-generated one in App.g.cs.

Still using the NativeMethods class from the question for reference:

internal static class NativeMethods
{
    private enum RpcAuthnLevel
    {
         Default = 0,
         None = 1,
         Connect = 2,
         Call = 3,
         Pkt = 4,
         PktIntegrity = 5,
         PktPrivacy = 6
    }

    private enum RpcImpLevel
    {
         Default = 0,
         Anonymous = 1,
         Identify = 2,
         Impersonate = 3,
         Delegate = 4
     }

     private enum EoAuthnCap
     {
         None = 0x0000,
         MutualAuth = 0x0001,
         StaticCloaking = 0x0020,
         DynamicCloaking = 0x0040,
         AnyAuthority = 0x0080,
         MakeFullSIC = 0x0100,
         Default = 0x0800,
         SecureRefs = 0x0002,
         AccessControl = 0x0004,
         AppID = 0x0008,
         Dynamic = 0x0010,
         RequireFullSIC = 0x0200,
         AutoImpersonate = 0x0400,
         NoCustomMarshal = 0x2000,
         DisableAAA = 0x1000
     }

     [DllImport("Ole32.dll",
         ExactSpelling = true,
         EntryPoint = "CoInitializeSecurity",
         CallingConvention = CallingConvention.StdCall,
         SetLastError = false,
         PreserveSig = false)]
     private static extern void CoInitializeSecurity(
         IntPtr pVoid,
         int cAuthSvc,
         IntPtr asAuthSvc,
         IntPtr pReserved1,
         uint dwAuthnLevel,
         uint dwImpLevel,
         IntPtr pAuthList,
         uint dwCapabilities,
         IntPtr pReserved3);

    public static void Initialize()
    {
        CoInitializeSecurity(IntPtr.Zero,
            -1,
            IntPtr.Zero,
            IntPtr.Zero,
            (uint)RpcAuthnLevel.PktPrivacy,
            (uint)RpcImpLevel.Impersonate,
            IntPtr.Zero,
            (uint)EoAuthnCap.DynamicCloaking,
            IntPtr.Zero);
    }
}

The new Main method needs to make the call to CoInitializeSecurity the first thing that happens, like so:

internal static void Main()
{
    NativeMethods.Initialize();
}

There are a few things lacking here, like the logic that was previously run in the compiler-generated Main. We need to run the App constructor in a thread with the STA threading model now when we have made our call to CoInitializeSecurity, like so:

[DebuggerNonUserCode]
internal static void STAMain()
{
    //This is what the compiler-generated Main method executes by default
    App app = new App();
    app.InitializeComponent();
    app.Run();
}

//Marking this as [STAThread] will cause RPC_E_TOO_LATE
internal static void Main()
{
    //This call won't throw an RPC_E_TOO_LATE COMException anymore
    NativeMethods.Initialize();
    
    /*
      We will have to create a GUI thread manually here 
      since the COM threading model isn't STA for this thread
    */
    Thread guiThread = new Thread(STAMain);
    guiThread.SetApartmentState(ApartmentState.STA);
    guiThread.Start();
}

Disclaimer: I'm not completely sure of how sensible or sane this solution is, but it seems to allow the application to work normally, and without managed compatibility mode.

answered on Stack Overflow Oct 14, 2020 by sweerpotato • edited Oct 14, 2020 by sweerpotato

User contributions licensed under CC BY-SA 3.0