Distributed Transaction Coordinator stops working properly after few hours

We noticed that DTC service stopped working properly. Log from ASP.NET application:

System.UnauthorizedAccessException: Access is denied. (Exception from HRESULT: 0x80070005 (E_ACCESSDENIED))
   at System.Transactions.Oletx.IDtcProxyShimFactory.ConnectToProxy(String nodeName, Guid resourceManagerIdentifier, IntPtr managedIdentifier, Boolean& nodeNameMatches, UInt32& whereaboutsSize, CoTaskMemHandle& whereaboutsBuffer, IResourceManagerShim& resourceManagerShim)
   at System.Transactions.Oletx.DtcTransactionManager.Initialize()
   at System.Transactions.Oletx.DtcTransactionManager.get_ProxyShimFactory()

DTC cannot be restarted while in this state. We noticed some event in Windows log so we tried different solutions. Here is the one that actually helps but only for few hours: https://technet.microsoft.com/en-us/library/cc774114(v=ws.10).aspx. DTC restarts but then following event appears and the problem comes back:

Source: Microsoft-Windows-MSDTC Client
Date: 2017-03-16 09:25:19
Event ID: 4356
Task Category: MSDTC Proxy
Description:
Failed to initialize the MS DTC Communication Manager.  Error Specifics: hr = 0x80070005, com\complus\dtc\dtc\cm\src\ccm.cpp:2118, CmdLine: C:\Windows\system32\msdtc.exe, Pid: 4252

Restarting DTC once again ends with error so following events appear:

Source: Microsoft-Windows-MSDTC
Date: 2017-03-16 10:20:41
Event ID: 4441
Task Category: SVC
Description:
A MS DTC component has encountered an internal error. The process is being terminated. Error Specifics: DtcSystemShutdown (com\complus\dtc\dtc\msdtc\src\msdtc.cpp@2485): Shutting down with an error

Source: Microsoft-Windows-MSDTC
Date: 2017-03-16 10:20:41
Event ID: 4439
Task Category: Unknown
Description:
Failed to verify MS DTC service account information. Internal Information : msdtc_trace : File: com\complus\dtc\shared\util\security.cpp, Line: 818, VerifyAccountInfo: CService::Create failed, hr=0x80070005.

Source: Microsoft-Windows-MSDTC Client
Date: 2017-03-16 10:20:41
Event ID: 4356
Task Category: MSDTC Proxy
Description:
Failed to initialize the MS DTC Communication Manager.  Error Specifics: hr = 0x80070005, com\complus\dtc\dtc\cm\src\ccm.cpp:2118, CmdLine: C:\Windows\system32\msdtc.exe, Pid: 4252

DTC reconfiguration helps again but not for long. More noticed events:

Log Name:      Application
Source: Microsoft-Windows-MSDTC Client
Date: 2017-03-14 12:03:27
Event ID: 4156
Task Category: CM
Description:
String message: INFORMATIONAL: Negotiate Resources returned OUT OF RESOURCES IN SSTATE_UP.

Source: Microsoft-Windows-Perflib
Date: 2017-03-15 13:58:59
Event ID: 1008
Task Category: None
Description:
The Open Procedure for service "MSDTC" in DLL "C:\Windows\system32\msdtcuiu.DLL" failed. Performance data for this service will not be available. The first four bytes (DWORD) of the Data section contains the error code.

Thanks to Martin Nolte and Feroz Khan from Microsoft support we managed to resolve the issue.

Verify that the problem is not connected with the "red arrow issue". Run dcomcnfg open Component Services and see if “Red Arrow” is on “My Computer”.

1. If you see a MSDTC red arrow on the database server. To rectify the issue, please follow the steps as mentioned below (or from https://blogs.msdn.microsoft.com/asiatech/2010/02/21/how-to-troubleshoot-the-red-arrow-issue-in-component-services-i/)

2. Make sure the MSDTC service is started:

3. Verify that Users group has permission to read subkeys under HKEY_CLASSES_ROOT\CLSID. If the Users group has no permission to read the subkeys, the COM+ System Application service may have difficulties to start and cause the same “Red Arrow” problem. To grand the Read permission to the Users group, we can follow this steps (at first we should make backup of HKEY_CLASSES_ROOT\CLSID):

   • Open Regedt32, locate HKEY_CLASSES_ROOT\CLSID
   • Select the CLSID, click Security -> Permission in the menu bar
   • In the Security tab, add USERS in the permission list, give it Read permission.
   • Click the Advanced button, select the “Replace permissions entries on all child objects with entries shown here that apply to child objects” option. Click Apply.

4. Check if everyone has Read permission on C:\Windows\Registration and its sub objects.

5. Ensure that the COM+ System Application service is in started status:

6. The MSDTC service should allow Authenticated Users to query service status. To check this, we can run this command in the Command window:

   sc sdshow msdtc
   

7. If the Authenticated Users group doesn’t have query permission on the MSDTC service object, this will result in most users having no permission to get the MSDTC service status, for example:

   (A;;CR;;;AU)
   

8. We need to run this command to grant enough permission for the Authenticated user, and then restart DLLHOST.exe (before doing this, please backup the output information of “sc sdshow msdtc”):

   sc sdset msdtc D:(A;;CCLCSWRPLOCRRC;;;S-1-2-0)
   (A;;CCLCSWRPWPDTLOCRRC;;;SY)(A;;CCDCLCSWRPWPDTLOCRSDRCWDWO;;;BA)
   (A;;CCLCSWLOCRRC;;;IU)(A;;CCLCSWLOCRRC;;;SU)(A;;CCLCSWLOCRRC;;;AU)        
   (A;;CCLCSWRPWPDTLOCRRC;;;PU)(A;;CCLCSWRPLORC;;;NS)S:
   (AU;FA;CCDCLCSWRPWPDTLOCRSDRCWDWO;;;WD)
   

9. If the above steps will not work, you have to follow other steps like rebuilding COM+ Catalog and eliminating registry keys corruption.

10. Restart server

Verify the problem with MSDTC service still exist. If so, make sure you do not have group policies set for MSDTC.

1. Restore the default account: https://technet.microsoft.com/en-us/library/cc774114(v=ws.10).aspx

2. If you set the account to “Network Service” in dcomcnfg then the following registry value is set: AccountName under HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\MSDTC\Security.

3. To review if this registry value is affected by your group policy you can perform the following steps:

   • Open rsop.msc on the affected machine which will display the applied group policies
   • Check “Computer Configuration” and go to “Windows Settings”/”Security Settings” and check the “Registry”
   • Can you see any definitions of HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\MSDTC\Security?
   • Check ”System Services” which is also below “Security Settings”. Can you see any modifications for “Distributed Transaction Coordinator”
   • Go to “User Configuration” and follow the same steps from step 2, 3, and 4.

4. To get all the applied group policy you would need to download

   • Remote Server Administration Tools for Windows 10
   • https://www.microsoft.com/en-us/download/details.aspx?id=45520

5. Open Powershell command prompt and enter these commands:

   Import-Module GroupPolicy Get-GPOReport -All -ReportType Html -Path AllGPOsReport.htm
   

6. Set default group policy for MSDTC

7. Run iisreset /restart
8. Again restore the default account: https://technet.microsoft.com/en-us/library/cc774114(v=ws.10).aspx
9. MSDTC service should work properly