FileLoadException when accessing WCF service

After publishing to production server we receive this error from time to time after application is restarted. To fix it our administrator is restarting app and it works again. We have no idea what is going on. What can I check?

Edit: In this same application we have asmx webservices that need identity impersonate=true so that is set in web.config.

Edit 2: I need to correct what I said. We receive this error after app pool is recycled. But not always.

Error in : http://SERVER_NAME/APP_NAME/OurService.svc LastError: System.IO.FileLoadException: Could not load file or assembly 'System.IdentityModel, Version=3.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089' or one of its dependencies.
The handle is invalid. (Exception from HRESULT: 0x80070006 (E_HANDLE)) 
File name: 'System.IdentityModel, Version=3.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089'
---> System.Runtime.InteropServices.COMException (0x80070006): The handle is invalid. (Exception from HRESULT: 0x80070006 (E_HANDLE))    
at System.ServiceModel.Activation.HostedImpersonationContext..ctor()    
at System.ServiceModel.Activation.HostedHttpRequestAsyncResult..ctor(HttpApplication context, Boolean flowContext, AsyncCallback callback, Object state)    
at System.ServiceModel.Activation.HostedHttpRequestAsyncResult.ExecuteSynchronous(HttpApplication context, Boolean flowContext)    
at System.ServiceModel.Activation.HttpModule.ProcessRequest(Object sender, EventArgs e)    
at System.Web.HttpApplication.SyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()    
at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)
Message: Could not load file or assembly 'System.IdentityModel, Version=3.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089' or one of its dependencies.
The handle is invalid. (Exception from HRESULT: 0x80070006 (E_HANDLE)) Source: System.ServiceModel