MS DTC Cancelling Transaction with Entity Framework Core

I'm receiving a

System.Data.SqlClient.SqlException (0x80131904): The Microsoft Distributed Transaction Coordinator (MS DTC) has cancelled the distributed transaction.

error when using EF Core 2.1. The full stack trace is at the bottom of this message.

Here's the basic process:

1. Read request message from Kafka
2. Write message to a set of "raw" tables on SQL Server
3. Call a stored procedure that reads message from raw tables, validates the incoming data, transforms a few fields in the data, and writes to a set of "production" tables.
4. Retrieve transformed request from production tables.
5. Process the request through an orchestration of mostly microservices, saving updates back to the production tables as progress is made through the orchestration.

The MSDTC error occurs intermittently, but frequently, at step 3. The raw and production tables in steps 2 and 3 are in separate databases on the same SQL Server. There is also a call to a separate SQL Server as part of step 5, but the only error I'm getting is the MSDTC cancellation during step 3.

I know that .NET Core doesn't support MSDTC, but I'm not sure why this any part of this process is being promoted to MSDTC. I don't have any explicit transactions and the connection strings are standardized for each database, via ConnectionStrings in appsettings.

Any insight appreciated.

authAPI.usp_AuthTransform @CorrelationID, @AuthStateID OUT, @CorrelationSeqID OUT

System.Data.SqlClient.SqlException (0x80131904): The Microsoft Distributed Transaction Coordinator (MS DTC) has cancelled the distributed transaction.
   at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
   at System.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
   at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
   at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
   at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString)
   at System.Data.SqlClient.SqlCommand.CompleteAsyncExecuteReader()
   at System.Data.SqlClient.SqlCommand.EndExecuteNonQueryInternal(IAsyncResult asyncResult)
   at System.Data.SqlClient.SqlCommand.EndExecuteNonQuery(IAsyncResult asyncResult)
   at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.EntityFrameworkCore.Storage.Internal.RelationalCommand.ExecuteAsync(IRelationalConnection connection, DbCommandMethod executeMethod, IReadOnlyDictionary`2 parameterValues, CancellationToken cancellationToken)
ClientConnectionId:24a868f2-3f00-4e07-92fd-7537beb23d33
Error Number:1206,State:118,Class:18
Exception thrown: 'System.Data.SqlClient.SqlException' in System.Private.CoreLib.dll
Exception thrown: 'System.Data.SqlClient.SqlException' in System.Private.CoreLib.dll
'dotnet.exe' (CoreCLR: clrhost): Loaded 'C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.1.15\System.IO.MemoryMappedFiles.dll'. Skipped loading symbols. Module is optimized and the debugger option 'Just My Code' is enabled.
nH.AuthRequestIntake.nHDischarge.ReceiveHandler:Error: 353c5648-5ef0-44b9-bd5b-ac1395243ac6-System.Data.SqlClient.SqlException (0x80131904): The Microsoft Distributed Transaction Coordinator (MS DTC) has cancelled the distributed transaction.
   at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
   at System.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
   at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
   at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
   at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString)
   at System.Data.SqlClient.SqlCommand.CompleteAsyncExecuteReader()
   at System.Data.SqlClient.SqlCommand.EndExecuteNonQueryInternal(IAsyncResult asyncResult)
   at System.Data.SqlClient.SqlCommand.EndExecuteNonQuery(IAsyncResult asyncResult)
   at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.EntityFrameworkCore.Storage.Internal.RelationalCommand.ExecuteAsync(IRelationalConnection connection, DbCommandMethod executeMethod, IReadOnlyDictionary`2 parameterValues, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.RelationalDatabaseFacadeExtensions.ExecuteSqlCommandAsync(DatabaseFacade databaseFacade, RawSqlString sql, IEnumerable`1 parameters, CancellationToken cancellationToken)
   at nH.AuthRequestIntake.Domain.AuthRequestReceivedEventHandler`2.TransformToAuthRequestAsync(AuthRequest rawRequest, IIncomingDbContext dbContext) in C:\Users\scott.carter\source\repos\authrequestintake\nH.AuthRequestIntake.Domain\AuthRequestReceivedEventHandler.cs:line 170
   at nH.AuthRequestIntake.Domain.AuthRequestReceivedEventHandler`2.HandleAsync(IDomainEvent domainEvent) in C:\Users\scott.carter\source\repos\authrequestintake\nH.AuthRequestIntake.Domain\AuthRequestReceivedEventHandler.cs:line 68
ClientConnectionId:24a868f2-3f00-4e07-92fd-7537beb23d33
Error Number:1206,State:118,Class:18

System.Data.SqlClient.SqlException (0x80131904): The Microsoft Distributed Transaction Coordinator (MS DTC) has cancelled the distributed transaction.
   at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
   at System.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
   at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
   at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
   at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString)
   at System.Data.SqlClient.SqlCommand.CompleteAsyncExecuteReader()
   at System.Data.SqlClient.SqlCommand.EndExecuteNonQueryInternal(IAsyncResult asyncResult)
   at System.Data.SqlClient.SqlCommand.EndExecuteNonQuery(IAsyncResult asyncResult)
   at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.EntityFrameworkCore.Storage.Internal.RelationalCommand.ExecuteAsync(IRelationalConnection connection, DbCommandMethod executeMethod, IReadOnlyDictionary`2 parameterValues, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.RelationalDatabaseFacadeExtensions.ExecuteSqlCommandAsync(DatabaseFacade databaseFacade, RawSqlString sql, IEnumerable`1 parameters, CancellationToken cancellationToken)
   at nH.AuthRequestIntake.Domain.AuthRequestReceivedEventHandler`2.TransformToAuthRequestAsync(AuthRequest rawRequest, IIncomingDbContext dbContext) in C:\Users\scott.carter\source\repos\authrequestintake\nH.AuthRequestIntake.Domain\AuthRequestReceivedEventHandler.cs:line 170
   at nH.AuthRequestIntake.Domain.AuthRequestReceivedEventHandler`2.HandleAsync(IDomainEvent domainEvent) in C:\Users\scott.carter\source\repos\authrequestintake\nH.AuthRequestIntake.Domain\AuthRequestReceivedEventHandler.cs:line 68
ClientConnectionId:24a868f2-3f00-4e07-92fd-7537beb23d33
Error Number:1206,State:118,Class:18```

This turned out to be a red herring. I read somewhere that MSDTC issues sometimes mask other issues and recommended doing a database trace. Sure enough, the the trace showed that a temp table was getting deleted across connections when under a load so we're now working to resolve this issue.

Thanks to all that read and considered the issue!