Exception on opening DataSet Visualizer due to the AssemblyResolve event

When trying to use the magnifing glasson one of my DataSet or DataTable in my .Net Core 3.1 WPF Project I get a System.IO.FileLoadException with following text:

Could not load file or assembly 'DataSetVisualizer.DebuggeeSide, Version=16.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'. General Exception (0x80131500)

Stack trace:

at System.Reflection.RuntimeAssembly.nLoad(AssemblyName fileName, String codeBase, RuntimeAssembly assemblyContext, StackCrawlMark& stackMark, Boolean throwOnFileNotFound, AssemblyLoadContext assemblyLoadContext)
at System.Reflection.RuntimeAssembly.InternalLoadAssemblyName(AssemblyName assemblyRef, StackCrawlMark& stackMark, AssemblyLoadContext assemblyLoadContext)
at System.Reflection.Assembly.Load(AssemblyName assemblyRef, StackCrawlMark& stackMark, AssemblyLoadContext assemblyLoadContext)
at System.Reflection.Assembly.Load(AssemblyName assemblyRef)
at Microsoft.VisualStudio.DebuggerVisualizers.DebuggeeSide.Impl.ClrCustomVisualizerDebuggeeHost..ctor(String debuggeeSideVisualizerTypeName, String debuggeeSideVisualizerAssemblyName, String[] probePaths)
at Microsoft.VisualStudio.DebuggerVisualizers.DebuggeeSide.Impl.ClrCustomVisualizerDebuggeeHost.Create(String debuggeeSideVisualizerTypeName, String debuggeeSideVisualizerAssemblyName, String[] probePaths)

I narrowed the cause for this problem down to my recently implemented method to load assemblies from subfolders at runtime, which I wrote based on Reza Aghaei's answer on my last question.

In narrowed it down to the subscription on the AppDomain.CurrentDomain.AssemblyResolve event, but couldn't find a way to solve it yet.

AppDomain.CurrentDomain.AssemblyResolve += (obj, arg) =>
{
  var name = $"{new AssemblyName(arg.Name).Name}.dll";
  var assemblyFile = referenceFiles.Where(x => x.EndsWith(name))
        .FirstOrDefault();
  if (assemblyFile != null)
      return Assembly.LoadFrom(assemblyFile);
  throw new Exception($"'{name}' Not found");
};

It doesn't matter if I am trying to view the DataSet in one of the loaded assemblies or the startup application.

I would like to keep working with this method to load assemblies at runtime, but since I am working with a lot of DataSets being able to use the DataSet Visualizer is crucial for me.

Any suggestions?

edit: usage of dataset visualizer during debugging:

Not an elegant solution, but as a workaround to unblock your debugging sessions, add the following to the beginning of the AssemblyResolve event handler:

#if DEBUG
    if (arg.Name.StartsWith("DataSetVisualizer"))
        return null;
#endif