Registration-free COM Interop between 32-bit client and 64-bit server

I need to use 64-bit dlls in 32-bit application.

First, I created a separate 64-bit executable and used Microsoft RPC to call 64-bit code from 32-bit application (originally binaries were native, but it doesn't matter here - we can always write .NET wrappers). Everything worked, but I didn't really like all this boilerplate code I need to write in order to launch RPC server, connect to it, run executable (and watch over it). I know that out-of-process COM can handle a lot of such work by hosting a dll in a dllhost.exe process invisibly for a client.

So I did as described here: create 64-bit server, 32-bit client, then register the server using regasm, change a couple of registry entries and everything works. But I can't use regasm on clients' machines, so I tried registration-free COM as described in the MSDN article. Everything worked without registration until server and client were both 32-bit. But when I rebuild server as a 64-bit binary, I obtain the following error:

Unhandled Exception: System.IO.FileLoadException: Retrieving the COM class factory for component with CLSID {D698BA94-AEFF-3D4F-9D11-BC6DE81D330B} failed due to the following error: 80131047 The given assembly name or codebase was invalid. (Exception from HRESULT: 0x80131047).

Can I have the best from the two methods: have registration-free COM Interop, but between 64-bit and 32-bit code?

UPDATE

Added source code. Here, if I remove processorArchitecture="amd64" from manifests and build server as x86 binary, everything works.

Client code

using System;
using System.Reflection;
using System.Runtime.InteropServices;

namespace ConsoleApp5
{
    class Program
    {
        static void Main(string[] args)
        {
            Type ComType = Type.GetTypeFromCLSID(new Guid("D698BA94-AEFF-3D4F-9D11-BC6DE81D330B"));

            var ComObject = Activator.CreateInstance(ComType);

            ComType.InvokeMember("TestMe", BindingFlags.InvokeMethod, null, ComObject, null);

            if (Marshal.IsComObject(ComObject))
                Marshal.ReleaseComObject(ComObject);
        }
    }
}

Client app manifest

<?xml version="1.0" encoding="utf-8"?>
<assembly manifestVersion="1.0" xmlns="urn:schemas-microsoft-com:asm.v1">
  <assemblyIdentity version="1.0.0.0" name="ConsoleApp5" type="win32"/>
  <dependency>
    <dependentAssembly>
      <assemblyIdentity
          name="ClassLibrary1"
          version="1.0.0.0"
          processorArchitecture="amd64"
        />
    </dependentAssembly>
  </dependency>
</assembly>

Server code

using System;
using System.Runtime.InteropServices;
using System.Windows.Forms;

namespace ClassLibrary1
{
    [ComVisible(true)]
    [Guid("D698BA94-AEFF-3D4F-9D11-BC6DE81D330B")]
    public class ComServer
    {
        /// <summary>
        /// Default constructor - necessary for using with COM
        /// </summary>
        public ComServer() { }

        /// <summary>
        /// Test method to be called by COM sonsumer
        /// </summary>
        public void TestMe()
        {
            MessageBox.Show("Hello from the 64-bit world!");
        }
    }
}

Server app manifest

<?xml version="1.0" encoding="utf-8"?>
<assembly manifestVersion="1.0" xmlns="urn:schemas-microsoft-com:asm.v1">
  <assemblyIdentity version="1.0.0.0" name="ClassLibrary1" processorArchitecture="amd64"/>
  <file name="ClassLibrary.dll"/>
  <clrClass
        clsid="{D698BA94-AEFF-3D4F-9D11-BC6DE81D330B}"
        progid="ClassLibrary1.ComServer"
        threadingModel="Both"
        name="ClassLibrary1.ComServer"
        runtimeVersion="v4.0.30319">
  </clrClass>
</assembly>