C# - How to properly use a referenced library

I am trying to use a library (RabbitMQ.Client to be precise) without referencing it, rather by referencing a library that is referencing RabbitMQ. I hope it's not confusing. By the way if by any chance you think it's weird, my reason behind this is I wanted to have Message Queuing library that is dynamically switchable using SimpleInjector. My problem is that every time I use a function it throws out an FileNotFound Exception on my reference. Here is a sample code:

DependencyResolver

Reference: SimpleInjector

public class DependencyResolver
{
   public static GetImplementation(string location, Type service)
   {
      Container container = new Container();
      System.Reflection.Assembly thisAsm = System.Reflection.Assembly.LoadFile(location);
      var qry = from type in thisAsm.GetTypes()
      where service.IsAssignableFrom(type)
      && type.Name == dependency.Name
      select new
      {
           Service = type.GetInterfaces().FirstOrDefault(),
           Implementation = type
      };
      if (qry.Count() > 0)
      {
           container.Register(qry.FirstOrDefault().Service,qry.FirstOrDefault().Implementation, Lifestyle.Transient);
           var obj = this.Container.GetInstance(serviceType);
           return obj.GetType();
      }
      return null; 
   }
}

MessageQ.dll

public interface IMessageQ
{
    string ConnectionString {get;set;}
    void Connect();
    void Send(string msg);
}

Rabbit-Style-MQ.dll

Reference: RabbitMQ.Client.dll, MessageQ.dll

public class RabbitMQStyle:IMessageQ
{
    public string ConnectionString {get;set;}

    private QueueingBasicConsumer Consumer { get; set; }
    private IModel Channel { get; set; }
    private IConnection Connection { get; set; }
    private ConnectionFactory Factory { get; set; }

    public void Connect()
    {
        this.Factory = new ConnectionFactory();
        this.Factory.Uri = this.ConnectionString;
        this.Connection = this.Factory.CreateConnection();
        this.Channel = Connection.CreateModel();
        this.Channel.QueueDeclare(this.QueueName, true, false, false, null);
    }

    public void Send(string msg)
    {


    }
}

Application Code

Reference: MessageQ.dll,DependencyResolver.dll;

static void Main(string[] args)
{

  var mq = (IMessageQ)Activator.CreateInstance(DependencyResolver.GetImplementation("E:\Rabbit-Style-MQ.dll",typeof(IMessageQ)));

  //Type of 'mq' is RabbitMQStyle =)

  mq.Connect();//Here is the part where my application threw an exception
}

Everything works if I reference RabbitMQ.Client on my application, but I don't want to, I only want it to be reference on my Rabbit-Style-Mq.dll. Thanks in advance.

Exception: Could not load file or assembly 'RabbitMQ.Client, Version=3.1.5.0, Culture=neutral, PublicKeyToken=89e7d7c5feba84ce' or one of its dependencies. Invalid pointer (Exception from HRESULT: 0x80004003 (E_POINTER))