Citrix Server with Server 2008 R2 svchost.exe Occasionally Crashes On Startup

Issue occurs across multiple Citrix severs running Windows 2008 R2. After svchost crashes, the services start fine. It is not the same service responsible every time. I am looking for ways to further investigate the root cause.

Application Event log shows error like below, with exception code 0xc0000022 (Access Denied) or 0xc0000024 (Application Fault) :

Faulting application name: svchost.exe, version: 6.1.7601.22137, time stamp: 0x5080442a
Faulting module name: ntdll.dll, version: 6.1.7601.23864, time stamp: 0x595fa942
Exception code: 0xc0000022
Fault offset: 0x00000000000c8078
Faulting process id: 0x61c
Faulting application start time: 0x01d3e3042b78dd80
Faulting application path: C:\Windows\system32\svchost.exe
Faulting module path: C:\Windows\SYSTEM32\ntdll.dll
Report Id: 721e7740-4ef7-11e8-ad7f-a204493e038b

I enabled user mode crash dumps as per MSDN article here

Based on analysis the following services have crashed at startup, on each search only one of these crashes, but it is a different each time it occurs, and it doesn't occur after every reboot:

• Network List Service (netprofm)
• Distributed Link Tracking Client (trkwks)
• Remote Desktop Services (termservice)
• World Wide Publishing Service (W3SVC)
• Windows Process Activation Service (WAS)
• Application Host Helper Service (apphostsvc)
• Windows Font Cache Service (fontcache)
• NET Driver HPZ12 (NET Driver HPZ12) Windows Font

The "Failure bucket" ID from several crashes from WinDbg:

• SVCHOSTGROUP_LocalService_ACCESS_DENIED_c0000022_svchost.exe!OpenServiceParametersKey
• SVCHOSTGROUP_LocalSystemNetworkRestricted_ACCESS_DENIED_c0000022_svchost.exe!OpenServiceParametersKey
• SVCHOSTGROUP_termsvcs_APPLICATION_FAULT_c0000024_svchost.exe!InitializeSecurity
• SVCHOSTGROUP_iissvcs_INVALID_HANDLE_c0000008_svchost.exe!BuildServiceArray
• SVCHOSTGROUP_apphost_APPLICATION_FAULT_c0000024_svchost.exe!CallPerInstanceInitFunctions
• SVCHOSTGROUP_termsvcs_ACCESS_DENIED_c0000022_svchost.exe!InitializeSecurity
• SVCHOSTGROUP_apphost_ACCESS_DENIED_c0000022_svchost.exe!CallPerInstanceInitFunctions

• SVCHOSTGROUP_LocalService_ACCESS_DENIED_c0000022_Sfrhook64.dll!InitSfrDataRegistry

  Checking several crash dumps, crash often occurs after registry access

Example stack:

 0:000> ~*k

.  0  Id: 4c8.4ec Suspend: 0 Teb: 000007ff`fffde000 Unfrozen
 # Child-SP          RetAddr           Call Site
00 00000000`001ffb68 000007fe`fd9d10ac ntdll!ZwWaitForSingleObject+0xa
01 00000000`001ffb70 000007fe`fdd1affb KERNELBASE!WaitForSingleObjectEx+0x79
02 00000000`001ffc10 000007fe`fdd19d61 sechost!ScSendResponseReceiveControls+0x13b
03 00000000`001ffd00 000007fe`fdd19c16 sechost!ScDispatcherLoop+0x121
04 00000000`001ffe10 00000000`ff9c1d3a sechost!StartServiceCtrlDispatcherW+0x14e
05 00000000`001ffe60 00000000`ff9c257a svchost!wmain+0x110
06 00000000`001ffe90 00000000`77ab59cd svchost!ScCreateWellKnownSids+0x2fd
07 00000000`001ffed0 00000000`77bea561 kernel32!BaseThreadInitThunk+0xd
08 00000000`001fff00 00000000`00000000 ntdll!RtlUserThreadStart+0x1d

   1  Id: 4c8.808 Suspend: 0 Teb: 000007ff`fffda000 Unfrozen
 # Child-SP          RetAddr           Call Site
00 00000000`00d6fc88 00000000`77bda3c7 ntdll!NtWaitForMultipleObjects+0xa
01 00000000`00d6fc90 00000000`77ab59cd ntdll!TppWaiterpThread+0x14d
02 00000000`00d6ff30 00000000`77bea561 kernel32!BaseThreadInitThunk+0xd
03 00000000`00d6ff60 00000000`00000000 ntdll!RtlUserThreadStart+0x1d

   2  Id: 4c8.80c Suspend: 0 Teb: 000007ff`fffd6000 Unfrozen
 # Child-SP          RetAddr           Call Site
00 00000000`00b7fa98 00000000`77bded15 ntdll!NtWaitForWorkViaWorkerFactory+0xa
01 00000000`00b7faa0 00000000`77ab59cd ntdll!TppWorkerThread+0x304
02 00000000`00b7fd30 00000000`77bea561 kernel32!BaseThreadInitThunk+0xd
03 00000000`00b7fd60 00000000`00000000 ntdll!RtlUserThreadStart+0x1d

   3  Id: 4c8.810 Suspend: 0 Teb: 000007ff`fffd4000 Unfrozen
 # Child-SP          RetAddr           Call Site
00 00000000`0126f578 00000000`77bded15 ntdll!NtWaitForWorkViaWorkerFactory+0xa
01 00000000`0126f580 00000000`77ab59cd ntdll!TppWorkerThread+0x304
02 00000000`0126f810 00000000`77bea561 kernel32!BaseThreadInitThunk+0xd
03 00000000`0126f840 00000000`00000000 ntdll!RtlUserThreadStart+0x1d

   4  Id: 4c8.818 Suspend: 0 Teb: 000007ff`fffac000 Unfrozen
 # Child-SP          RetAddr           Call Site
00 00000000`00def708 000007fe`f0091076 ntdll!NtReplyWaitReceivePort+0xa
01 00000000`00def710 000007fe`f0093325 uxsms!CPortBase::PortThreadInternal+0xbf
02 00000000`00def770 00000000`77ab59cd uxsms!CPortBase::PortThread+0x9
03 00000000`00def7a0 00000000`77bea561 kernel32!BaseThreadInitThunk+0xd
04 00000000`00def7d0 00000000`00000000 ntdll!RtlUserThreadStart+0x1d

   5  Id: 4c8.8e8 Suspend: 0 Teb: 000007ff`fffae000 Unfrozen
 # Child-SP          RetAddr           Call Site
00 00000000`0149f9c8 000007fe`fd9d16ad ntdll!NtRemoveIoCompletion+0xa
01 00000000`0149f9d0 00000000`77aa9991 KERNELBASE!GetQueuedCompletionStatus+0x39
02 00000000`0149fa30 000007fe`efc1f352 kernel32!GetQueuedCompletionStatusStub+0x11
03 00000000`0149fa70 00000000`77ab59cd audiosrv!EventWorkerThread+0xb2
04 00000000`0149fab0 00000000`77bea561 kernel32!BaseThreadInitThunk+0xd
05 00000000`0149fae0 00000000`00000000 ntdll!RtlUserThreadStart+0x1d

#  6  Id: 4c8.1144 Suspend: 0 Teb: 000007ff`fffdc000 Unfrozen
 # Child-SP          RetAddr           Call Site
00 00000000`011ce7d8 000007fe`fd9d1430 ntdll!NtWaitForMultipleObjects+0xa
01 00000000`011ce7e0 00000000`77ac16e3 KERNELBASE!WaitForMultipleObjectsEx+0xe8
02 00000000`011ce8e0 00000000`77b3b8b5 kernel32!WaitForMultipleObjectsExImplementation+0xb3
03 00000000`011ce970 00000000`77b3ba37 kernel32!WerpReportFaultInternal+0x215
04 00000000`011cea10 00000000`77b3ba8f kernel32!WerpReportFault+0x77
05 00000000`011cea40 00000000`77b3bcac kernel32!BasepReportFault+0x1f
06 00000000`011cea70 00000000`77c40108 kernel32!UnhandledExceptionFilter+0x1fc
07 00000000`011ceb50 00000000`77bd7958 ntdll! ?? ::FNODOBFM::`string'+0x2025
08 00000000`011ceb80 00000000`77be812d ntdll!_C_specific_handler+0x8c
09 00000000`011cebf0 00000000`77bd855f ntdll!RtlpExecuteHandlerForException+0xd
0a 00000000`011cec20 00000000`77c880c0 ntdll!RtlDispatchException+0x45a
0b 00000000`011cf300 00000000`77c47c7d ntdll!RtlRaiseStatus+0x60
0c 00000000`011cf8a0 00000000`77c08e54 ntdll! ?? ::FNODOBFM::`string'+0xa6bb
0d 00000000`011cf950 00000000`77ab3b40 ntdll!RtlEnterCriticalSection+0xd1
0e 00000000`011cf980 00000000`77ab38fc kernel32!MapPredefinedHandleInternal+0xb4
0f 00000000`011cf9d0 00000000`77ab3a1d kernel32!RegOpenKeyExInternalW+0xca
10 00000000`011cfa60 00000000`ff9c1055 kernel32!RegOpenKeyExW+0x1d
11 00000000`011cfaa0 00000000`ff9c1129 svchost!OpenServiceParametersKey+0x45
12 00000000`011cfae0 00000000`ff9c135b svchost!UnloadServiceDll+0x39
13 00000000`011cfb30 000007fe`fdd1a82d svchost!ServiceStarter+0x1ff
14 00000000`011cfbc0 00000000`77ab59cd sechost!ScSvcctrlThreadA+0x25
15 00000000`011cfbf0 00000000`77bea561 kernel32!BaseThreadInitThunk+0xd
16 00000000`011cfc20 00000000`00000000 ntdll!RtlUserThreadStart+0x1d

   7  Id: 4c8.115c Suspend: 0 Teb: 000007ff`fffaa000 Unfrozen
 # Child-SP          RetAddr           Call Site
00 00000000`00c0e588 000007fe`fd9d1203 ntdll!ZwDelayExecution+0xa
01 00000000`00c0e590 00000000`77b3ba05 KERNELBASE!SleepEx+0xab
02 00000000`00c0e630 00000000`77b3ba8f kernel32!WerpReportFault+0x45
03 00000000`00c0e660 00000000`77b3bcac kernel32!BasepReportFault+0x1f
04 00000000`00c0e690 00000000`77c40108 kernel32!UnhandledExceptionFilter+0x1fc
05 00000000`00c0e770 00000000`77bd7958 ntdll! ?? ::FNODOBFM::`string'+0x2025
06 00000000`00c0e7a0 00000000`77be812d ntdll!_C_specific_handler+0x8c
07 00000000`00c0e810 00000000`77bd855f ntdll!RtlpExecuteHandlerForException+0xd
08 00000000`00c0e840 00000000`77bd8b58 ntdll!RtlDispatchException+0x45a
09 00000000`00c0ef20 000007fe`fd9ea06d ntdll!RtlRaiseException+0x22f
0a 00000000`00c0f8d0 000007fe`e83f9586 KERNELBASE!RaiseException+0x39
0b 00000000`00c0f9a0 000007fe`e83f4e36 trkwks+0x9586
0c 00000000`00c0fa00 00000000`77ab59cd trkwks+0x4e36
0d 00000000`00c0fa30 00000000`77bea561 kernel32!BaseThreadInitThunk+0xd
0e 00000000`00c0fa60 00000000`00000000 ntdll!RtlUserThreadStart+0x1d

   8  Id: 4c8.e00 Suspend: 0 Teb: 000007ff`fffa8000 Unfrozen
 # Child-SP          RetAddr           Call Site
00 00000000`0155fa08 00000000`77c08f58 ntdll!ZwWaitForSingleObject+0xa
01 00000000`0155fa10 00000000`77c08e54 ntdll!RtlpWaitOnCriticalSection+0xe8
02 00000000`0155fac0 00000000`ff9c1795 ntdll!RtlEnterCriticalSection+0xd1
03 00000000`0155faf0 000007fe`fdd1a82d svchost!ServiceStarter+0x45
04 00000000`0155fb80 00000000`77ab59cd sechost!ScSvcctrlThreadA+0x25
05 00000000`0155fbb0 00000000`77bea561 kernel32!BaseThreadInitThunk+0xd
06 00000000`0155fbe0 00000000`00000000 ntdll!RtlUserThreadStart+0x1d

   9  Id: 4c8.1468 Suspend: 0 Teb: 000007ff`fffa6000 Unfrozen
 # Child-SP          RetAddr           Call Site
00 00000000`012ff918 00000000`77c08f58 ntdll!ZwWaitForSingleObject+0xa
01 00000000`012ff920 00000000`77c08e54 ntdll!RtlpWaitOnCriticalSection+0xe8
02 00000000`012ff9d0 00000000`ff9c1795 ntdll!RtlEnterCriticalSection+0xd1
03 00000000`012ffa00 000007fe`fdd1a82d svchost!ServiceStarter+0x45
04 00000000`012ffa90 00000000`77ab59cd sechost!ScSvcctrlThreadA+0x25
05 00000000`012ffac0 00000000`77bea561 kernel32!BaseThreadInitThunk+0xd
06 00000000`012ffaf0 00000000`00000000 ntdll!RtlUserThreadStart+0x1d

In this case, in the thread that crashed trkwks is first argument passed to svchost!OpenServiceParametersKey and the first argument passed to kernel32!RegOpenKeyExW is System\CurrentControlSet\Services

Process environment block:

   0:006> !peb
    PEB at 000007fffffd8000
        InheritedAddressSpace:    No
        ReadImageFileExecOptions: Yes
        BeingDebugged:            No
        ImageBaseAddress:         00000000ff9c0000
        Ldr                       0000000077ced640
        Ldr.Initialized:          Yes
        Ldr.InInitializationOrderModuleList: 0000000000082c70 . 00000000000d0240
        Ldr.InLoadOrderModuleList:           0000000000082b60 . 00000000000d0220
        Ldr.InMemoryOrderModuleList:         0000000000082b70 . 00000000000d0230
                Base TimeStamp                     Module
            ff9c0000 4a5bc3c1 Jul 14 09:31:13 2009 C:\Windows\System32\svchost.exe
            77bc0000 59b94ee4 Sep 14 01:29:40 2017 C:\Windows\SYSTEM32\ntdll.dll
            77aa0000 59b94f29 Sep 14 01:30:49 2017 C:\Windows\system32\kernel32.dll
         7fefd9d0000 59b94f2a Sep 14 01:30:50 

2017 C:\Windows\system32\KERNELBASE.dll
     7fefe6d0000 4eeb033f Dec 16 19:37:19 2011 C:\Windows\system32\msvcrt.dll
     7fefdd10000 55636728 May 26 04:17:12 2015 C:\Windows\SYSTEM32\sechost.dll
     7feffb60000 59b94e91 Sep 14 01:28:17 2017 C:\Windows\system32\RPCRT4.dll
     7fefd7c0000 5a04bbf7 Nov 10 07:35:03 2017 C:\Program Files (x86)\Citrix\system32\MfApHook64.dll
     7feffa60000 59b94e85 Sep 14 01:28:05 2017 C:\Windows\system32\ADVAPI32.dll
        779a0000 5824a140 Nov 11 03:33:04 2016 C:\Windows\system32\user32.dll
     7fefe380000 59b2b7a8 Sep 09 01:30:48 2017 C:\Windows\system32\GDI32.dll
     7fefe770000 59debda5 Oct 12 11:56:05 2017 C:\Windows\system32\LPK.dll
     7fefe2b0000 599464f6 Aug 17 01:29:58 2017 C:\Windows\system32\USP10.dll
     7feffc90000 4a5bdf40 Jul 14 11:28:32 2009 C:\Windows\system32\IMM32.DLL
     7fefdd30000 59b94ec5 Sep 14 01:29:09 2017 C:\Windows\system32\MSCTF.dll
        75590000 5a4bcf7b Jan 03 05:29:15 2018 C:\Windows\System32\lsihok64.dll
     7fefd680000 5ac7b4ef Apr 07 03:57:03 2018 C:\Program Files (x86)\Citrix\system32\CtxMFPlugin64.dll
     7feff590000 598d5026 Aug 11 16:35:18 2017 C:\Windows\system32\ole32.dll
     7fefe1d0000 58f4e14e Apr 18 01:37:50 2017 C:\Windows\system32\OLEAUT32.dll
     7fefd240000 59985949 Aug 20 01:29:13 2017 C:\Windows\System32\MF.dll
     7fefd220000 4a5bde70 Jul 14 11:25:04 2009 C:\Windows\System32\ATL.DLL
     7fefd1b0000 57603c1b Jun 15 03:17:15 2016 C:\Windows\System32\MFPlat.DLL
     7feff790000 573365bb May 12 03:02:51 2016 C:\Windows\system32\WS2_32.dll
     7fefe3f0000 598d5072 Aug 11 16:36:34 2017 C:\Windows\system32\NSI.dll
     7fefe780000 4ce7c9ab Nov 21 00:14:19 2010 C:\Windows\system32\SHLWAPI.dll
     7fefd1a0000 4a5bde96 Jul 14 11:25:42 2009 C:\Windows\System32\AVRT.dll
     7fefd190000 4a5be082 Jul 14 11:33:54 2009 C:\Windows\System32\VERSION.dll
        75580000 56672a67 Dec 09 06:07:19 2015 C:\Windows\System32\ksuser.dll
     7fefd150000 5ac7b321 Apr 07 03:49:21 2018 C:\Program Files (x86)\Citrix\system32\CtxGraphicsHelper64.dll
     7feff880000 4ce7c9a2 Nov 21 00:14:10 2010 C:\Windows\system32\SETUPAPI.dll
     7fefda80000 4ce7c55c Nov 20 23:55:56 2010 C:\Windows\system32\CFGMGR32.dll
     7fefda60000 4a5bdee1 Jul 14 11:26:57 2009 C:\Windows\system32\DEVOBJ.dll
     7fefd040000 5ac7b1f9 Apr 07 03:44:25 2018 C:\Program Files (x86)\Citrix\system32\mmhook64.dll
     7fefce60000 5ac7b1fc Apr 07 03:44:28 2018 C:\Program Files (x86)\Citrix\system32\Sfrhook64.dll
     7fefd100000 4a5be09c Jul 14 11:34:20 2009 C:\Windows\System32\WTSAPI32.dll
     7fefcec0000 5ac7a2cd Apr 07 02:39:41 2018 C:\Program Files (x86)\Citrix\system32\scardhook64.dll
     7fefcdd0000 5ac7b1ac Apr 07 03:43:08 2018 C:\Program Files (x86)\Citrix\system32\cxinjime64.dll
     7fefcda0000 59b94f68 Sep 14 01:31:52 2017 C:\Windows\System32\CRYPTBASE.dll
     7fef0090000 4a5be092 Jul 14 11:34:10 2009 c:\windows\system32\uxsms.dll
     7fefd640000 53c72fe8 Jul 17 12:07:36 2014 C:\Windows\System32\WINSTA.dll
     7feefc10000 57603bf5 Jun 15 03:16:37 2016 c:\windows\system32\audiosrv.dll
     7fef2440000 4a5be062 Jul 14 11:33:22 2009 c:\windows\system32\POWRPROF.dll
     7fefb280000 4a5bdf68 Jul 14 11:29:12 2009 c:\windows\system32\MMDevAPI.DLL
     7fefb150000 4ce7c94a Nov 21 00:12:42 2010 c:\windows\system32\PROPSYS.dll
     7fefdc70000 4a5bdeba Jul 14 11:26:18 2009 C:\Windows\system32\CLBCatQ.DLL
     7fee83f0000 4a5be082 Jul 14 11:33:54 2009 c:\windows\system32\trkwks.dll
    SubSystemData:     0000000000000000
    ProcessHeap:       0000000000080000
    ProcessParameters: 0000000000082050
    CurrentDirectory:  'C:\Windows\system32\'
    WindowTitle:  'C:\Windows\System32\svchost.exe'
    ImageFile:    'C:\Windows\System32\svchost.exe'
    CommandLine:  'C:\Windows\System32\svchost.exe -k LocalSystemNetworkRestricted'
    DllPath:      'C:\Windows\System32;C:\Windows\system32;C:\Windows\system;C:\Windows;.;C:\ProgramData\Oracle\Java\javapath;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Program Files (x86)\Citrix\System32\;C:\Program Files (x86)\Enterprise Vault\EVClient\;C:\Program Files (x86)\Citrix\system32;C:\Program Files\Citrix\Virtual Desktop Agent\;C:\Program Files (x86)\Norskale\Norskale Agent Host\'
    Environment:  0000000000081320
        ALLUSERSPROFILE=C:\ProgramData
        APPDATA=C:\Windows\system32\config\systemprofile\AppData\Roaming
        CommonProgramFiles=C:\Program Files\Common Files
        CommonProgramFiles(x86)=C:\Program Files (x86)\Common Files
        CommonProgramW6432=C:\Program Files\Common Files
        COMPUTERNAME=SERVER01
        ComSpec=C:\Windows\system32\cmd.exe
        FP_NO_HOST_CHECK=NO
        lib=C:\Program Files (x86)\SQLXML 3.0\bin\
        LOCALAPPDATA=C:\Windows\system32\config\systemprofile\AppData\Local
        NUMBER_OF_PROCESSORS=4
        OS=Windows_NT
        Path=C:\ProgramData\Oracle\Java\javapath;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Program Files (x86)\Citrix\System32\;C:\Program Files (x86)\Enterprise Vault\EVClient\;C:\Program Files (x86)\Citrix\system32;C:\Program Files\Citrix\Virtual Desktop Agent\;C:\Program Files (x86)\Norskale\Norskale Agent Host\
        PATHEXT=.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC
        PROCESSOR_ARCHITECTURE=AMD64
        PROCESSOR_IDENTIFIER=Intel64 Family 6 Model 45 Stepping 7, GenuineIntel
        PROCESSOR_LEVEL=6
        PROCESSOR_REVISION=2d07
        ProgramData=C:\ProgramData
        ProgramFiles=C:\Program Files
        ProgramFiles(x86)=C:\Program Files (x86)
        ProgramW6432=C:\Program Files
        PSModulePath=C:\Windows\system32\WindowsPowerShell\v1.0\Modules\;C:\Program Files\Citrix\Telemetry Service\
        PUBLIC=C:\Users\Public
        SystemDrive=C:
        SystemRoot=C:\Windows
        TEMP=C:\Windows\TEMP
        TMP=C:\Windows\TEMP
        UATDATA=C:\Windows\CCM\UATData\D9F8C395-CAB8-491d-B8AC-179A1FE1BE77
        USERDOMAIN=DOMAIN
        USERNAME=SERVER01$
        USERPROFILE=C:\Windows\system32\config\systemprofile
        windir=C:\Windows
        windows_tracing_flags=3
       windows_tracing_logfile=C:\BVTBin\Tests\installpackage\csilogfile.log