Dynamic loading NET46 dll from NETCORE11

I have a netcore1.1 library project using EFCore and more NETCore based libraries, which uses plugins.

some of the plug-ins was not ported into net-core and are targeting net40 and net46...
and they are not targeting the frameworks of my running, loading, assembly.

Now it is logical, I will run into problems loading assemblies from different frameworks into my NetCore1.1 App...

Adding net4x to the frameworks list requires refactoring the code for both .netCore and .net, either way, one target won't be able to work with both framework's plug-ins.

How can I (from a NET-CORE App) run a plug-in in it's own environment that fits it? while maintaining the interface ?
Note: If it helps, My interface with the plugins is pretty plain, basic types.

Edit: I try using AssemblyLoader.Resolve, and it when trying to reference
{System, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089},
(which surprisingly wasn't loaded from GAC), i loaded it from C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6, gives me an exception:

Could not load file or assembly 'System, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089, processorArchitecture=MSIL'.  
Reference assemblies should not be loaded for execution.  
They can only be loaded in the Reflection-only loader context.  
(Exception from HRESULT: 0x80131058)

Thanks.