Intermittent Could not load file or assembly / PolicyExceptions

Intermittently we'll get errors like these from our .NET 3.5 web applications:

Exception:
System.Configuration.ConfigurationErrorsException: Could not load file or assembly 'itextsharp, Version=4.1.2.0, Culture=neutral, PublicKeyToken=8354ae6d2174ddca' or one of its dependencies. Failed to grant permission to execute. (Exception from HRESULT: 0x80131418) (C:\Windows\Microsoft.NET\Framework64\v2.0.50727\Config\web.config line 59) ---> System.IO.FileLoadException: Could not load file or assembly 'itextsharp, Version=4.1.2.0, Culture=neutral, PublicKeyToken=8354ae6d2174ddca' or one of its dependencies. Failed to grant permission to execute. (Exception from HRESULT: 0x80131418) File name: 'itextsharp, Version=4.1.2.0, Culture=neutral, PublicKeyToken=8354ae6d2174ddca' ---> System.Security.Policy.PolicyException: Execution permission cannot be acquired.
   at System.Security.SecurityManager.ResolvePolicy(Evidence evidence, PermissionSet reqdPset, PermissionSet optPset, PermissionSet denyPset, PermissionSet& denied, Boolean checkExecutionPermission)
   at System.Security.SecurityManager.ResolvePolicy(Evidence evidence, PermissionSet reqdPset, PermissionSet optPset, PermissionSet denyPset, PermissionSet& denied, Int32& securitySpecialFlags, Boolean checkExecutionPermission)
   at System.Reflection.Assembly._nLoad(AssemblyName fileName, String codeBase, Evidence assemblySecurity, Assembly locationHint, StackCrawlMark& stackMark, Boolean throwOnFileNotFound, Boolean forIntrospection)
   at System.Reflection.Assembly.InternalLoad(AssemblyName assemblyRef, Evidence assemblySecurity, StackCrawlMark& stackMark, Boolean forIntrospection)
   at System.Reflection.Assembly.InternalLoad(String assemblyString, Evidence assemblySecurity, StackCrawlMark& stackMark, Boolean forIntrospection)
   at System.Reflection.Assembly.Load(String assemblyString)
   at System.Web.Configuration.CompilationSection.LoadAssemblyHelper(String assemblyName, Boolean starDirective)


   --- End of inner exception stack trace ---
   at System.Web.Configuration.CompilationSection.LoadAssemblyHelper(String assemblyName, Boolean starDirective)
   at System.Web.Configuration.CompilationSection.LoadAllAssembliesFromAppDomainBinDirectory()
   at System.Web.Configuration.CompilationSection.LoadAssembly(AssemblyInfo ai)
   at System.Web.Configuration.AssemblyInfo.get_AssemblyInternal()
   at System.Web.Compilation.BuildManager.GetReferencedAssemblies(CompilationSection compConfig)
   at System.Web.Compilation.WebDirectoryBatchCompiler..ctor(VirtualDirectory vdir)
   at System.Web.Compilation.BuildManager.BatchCompileWebDirectoryInternal(VirtualDirectory vdir, Boolean ignoreErrors)
   at System.Web.Compilation.BuildManager.CompileWebFile(VirtualPath virtualPath)
   at System.Web.Compilation.BuildManager.GetVPathBuildResultInternal(VirtualPath virtualPath, Boolean noBuild, Boolean allowCrossApp, Boolean allowBuildInPrecompile)
   at System.Web.Compilation.BuildManager.GetVPathBuildResultWithNoAssert(HttpContext context, VirtualPath virtualPath, Boolean noBuild, Boolean allowCrossApp, Boolean allowBuildInPrecompile)
   at System.Web.Compilation.BuildManager.GetVirtualPathObjectFactory(VirtualPath virtualPath, HttpContext context, Boolean allowCrossApp, Boolean noAssert)
   at System.Web.Compilation.BuildManager.GetCompiledType(String virtualPath)
   at System.Web.Script.Services.WebServiceData.GetWebServiceData(HttpContext context, String virtualPath, Boolean failIfNoData, Boolean pageMethods, Boolean inlineScript)
   at System.Web.Script.Services.RestHandler.CreateHandler(HttpContext context)
   at System.Web.Script.Services.ScriptHandlerFactory.GetHandler(HttpContext context, String requestType, String url, String pathTranslated)
   at System.Web.HttpApplication.MaterializeHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
   at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)

Inner exception:
System.IO.FileLoadException: Could not load file or assembly 'itextsharp, Version=4.1.2.0, Culture=neutral, PublicKeyToken=8354ae6d2174ddca' or one of its dependencies. Failed to grant permission to execute. (Exception from HRESULT: 0x80131418) File name: 'itextsharp, Version=4.1.2.0, Culture=neutral, PublicKeyToken=8354ae6d2174ddca' ---> System.Security.Policy.PolicyException: Execution permission cannot be acquired.
   at System.Security.SecurityManager.ResolvePolicy(Evidence evidence, PermissionSet reqdPset, PermissionSet optPset, PermissionSet denyPset, PermissionSet& denied, Boolean checkExecutionPermission)
   at System.Security.SecurityManager.ResolvePolicy(Evidence evidence, PermissionSet reqdPset, PermissionSet optPset, PermissionSet denyPset, PermissionSet& denied, Int32& securitySpecialFlags, Boolean checkExecutionPermission)
   at System.Reflection.Assembly._nLoad(AssemblyName fileName, String codeBase, Evidence assemblySecurity, Assembly locationHint, StackCrawlMark& stackMark, Boolean throwOnFileNotFound, Boolean forIntrospection)
   at System.Reflection.Assembly.InternalLoad(AssemblyName assemblyRef, Evidence assemblySecurity, StackCrawlMark& stackMark, Boolean forIntrospection)
   at System.Reflection.Assembly.InternalLoad(String assemblyString, Evidence assemblySecurity, StackCrawlMark& stackMark, Boolean forIntrospection)
   at System.Reflection.Assembly.Load(String assemblyString)
   at System.Web.Configuration.CompilationSection.LoadAssemblyHelper(String assemblyName, Boolean starDirective)

web.config line 59 being:

<add assembly="*"/>

When these occur, the sites will YSOD untill we recycle the application pool. The sites may run for days/weeks before this occurs, or it might happen twice within the hour. I have not been able to pinpoint this to any specific request/function in our system.

In this case it points to itextsharp, but it randomly points to any assembly referenced by our application, both internal and external.

Running caspol verifies that the DLL has full trust permissions:

C:\Windows\Microsoft.NET\Framework64\v2.0.50727>caspol -rsg D:\...\bin\itextsharp.dll
Microsoft (R) .NET Framework CasPol 2.0.50727.3053
Copyright (c) Microsoft Corporation.  All rights reserved.

Level = Enterprise

Code Groups:

1.  All code: FullTrust


Level = Machine

Code Groups:

1.  All code: Nothing
   1.1.  Zone - MyComputer: FullTrust


Level = User

Code Groups:

1.  All code: FullTrust

Success

Our application is running on three servers, two of them are on Server 2008 Web x64 while the third is running Server 2008 R2 Web x64, all have .NET 3.5 installed, no .NET 4.0 installations.

The problem only occurs on the first two that are running 2008 non R2.

Running depends.exe on all three servers gives equal results for the nonR2 servers:

• My DLL is shown as x86 (compiled as AnyCPU, running in x64 w3wp), all other modules show as x64.
• Missing IESHIMS.DLL and LINKINFO.DLL - both of these seem to be red herrings according to Google.

The third server shows the same, except it does not miss LINKINFO.DLL

All servers are running IIS7 (7.5 for the R2 one) under a custom domain account that has been granted the necessary permissions:

aspnet_regiis -ga [user]

Load user profile is set to false on all three servers. I've tried setting this to true on one of the faulting servers, according to: IIS7 Failed to grant minimum permission requests

By running processmonitor I can see that it's now using the C:\Users\TEMP\AppData\Local\Temp directory for various temp files - the other ones are not using any such directory. So far I'll let it run in this way to see if this changes anything. I'm in doubt however given that the third server is not exhibiting the problems, yet still has "Load user profile" set to the same value, false.

I've also tried running Fuslogvw on all three servers, logging binding failures to disk. All three servers report the same binding errors for VJSharpCodeProvider and CppCodeProvider, but these seem to be normal as well and can be solved by not defining the DEBUG and TRACE constants during build.

We're running about 500 websites on each server (identical, load balanced), of which 50 are under moderate load, the problem has arisen both under heavy load as well as under minimal load however.

Right now I'm waiting for the errors to happen again so I can hopefully see a pattern and determine whether "Load user profile" alleviates the issue. Any suggestions in the meantime would be very welcome! Also, I don't understand how the lack of "Load user profile" would cause an issue like this? And even further, how it would seemingly work on R2 but not on plain 2008?

Thanks!

Might be an issue with how the runtime locates the assemblies and the effect it has on a load balanced environment.

You can try specifying where your assemblies are retrieved from using the <probing> element.

<configuration>
   <runtime>
      <assemblyBinding xmlns="urn:schemas-microsoft-com:asm.v1">
         <probing privatePath="bin;bin2\subbin;bin3"/>
      </assemblyBinding>
   </runtime>
</configuration>

EDIT:

Try this:

<configuration>
   <runtime>
      <disableCachingBindingFailures enabled="1" />
   </runtime>
</configuration>