How to find the source of a "Exception encountered unloading application domain" with the Nunit runner?

We have a big bunch of tests(6000+) in differents assemblies. We run them on a local Azure Agent, with the Nunit Test adapter.

It was taking a very long time, so we activated the runInParallel.

The yaml step is now the following:

  - task: VSTest@2
    timeoutInMinutes: 300
    inputs:
      testSelector: 'testAssemblies'
      testAssemblyVer2: '*.Test*.dll'
      searchFolder: '$(System.DefaultWorkingDirectory)/$(buildConfiguration)'
      codeCoverageEnabled: false
      platform: 'Any CPU'
      uiTests: false
      configuration: '$(buildConfiguration)'
      rerunFailedTests: false
      pathtoCustomTestAdapters: 'Solution/packages/NUnit3TestAdapter.3.12.0/build/net35'
      minimumExpectedTests: 1000
      runInParallel: true
      distributionBatchType: basedOnAssembly
      failOnMinTestsNotRun: true
      resultsFolder: 'testResults'
      runSettingsFile: '$(Build.SourcesDirectory)\azure-tests.runsettings'
    #continueOnError: true

Now, we get some tests that doesn't pass. I've analyzed one of them, and basically, it fails because at some point, it does a OfType<SomeInterface>. The objects in the collection does implement this interface, so I guess I must have an issue with this assembly loaded 2 time and this interface not being considered the same as the created objects.

When I check the logs, I now see this, which seems to confirm this theory:

##[warning]RunMessage : Exception NUnit.Engine.NUnitEngineUnloadException,    Exception thrown unloading tests from C:\DevOp\2\s\debug\Acceptance.Tests.dll
##[warning]RunMessage : Exception encountered unloading application domain
##[warning]RunMessage :    at NUnit.Engine.Services.DomainManager.DomainUnloader.Unload()
   at NUnit.Engine.Runners.TestDomainRunner.UnloadPackage()
   at NUnit.Engine.Runners.AbstractTestRunner.Dispose(Boolean disposing)
   at NUnit.Engine.Runners.AbstractTestRunner.Dispose()
   at NUnit.Engine.Runners.MasterTestRunner.Dispose(Boolean disposing)
   at NUnit.Engine.Runners.MasterTestRunner.Dispose()
   at NUnit.VisualStudio.TestAdapter.NUnit3TestExecutor.RunAssembly(String assemblyPath, TestFilter filter) in D:\repos\nunit\nunit3-vs-adapter\src\NUnitTestAdapter\NUnit3TestExecutor.cs:line 321
##[warning]RunMessage : Innerexception: NUnit.Engine.NUnitEngineException: Exception encountered unloading application domain: Error while unloading appdomain. (Exception from HRESULT: 0x80131015)
Application domain name: domain-dcec2bc8-Acceptance.Tests.dll
Application domain BaseDirectory: C:\DevOp\2\s\debug\

I've searched, but I didn't found how to solve this issue. I can understand, that I probably have something that prevent the assembly to properly unload, but we have more than 6000 tests, 15 tests assemblies which references some production code and some test code. The execution of all tests takes ~2h45.

How do I know what cannot be unloaded(and why?) How would you proceed to solve this error?