Native image for System.Data.Entity crashing IIS
I have an MVC 3 website using entity framework and it has just recently started crashing on a specific action. I wish I had more information, but all I can pull from the Windows event viewer is this:
Faulting application name: w3wp.exe, version: 7.5.7601.17514, time stamp: 0x4ce7a5f8
Faulting module name: System.Data.Entity.ni.dll, version: 4.0.30319.1, time stamp: 0x4ba1e2fd
Exception code: 0xc00000fd
Fault offset: 0x003aac6a
Faulting process id: 0x2f0
Faulting application start time: 0x01ce540a70477360
Faulting application path: C:\Windows\SysWOW64\inetsrv\w3wp.exe
Faulting module path: C:\Windows\assembly\NativeImages_v4.0.30319_32\System.Data.Entity\dc0c74bc42bbaeffcb7158c7ed0f1653\System.Data.Entity.ni.dll
Report Id: 3659a820-bffe-11e2-8207-404094d3cf82
If I stop IIS, delete the native image, and then restart IIS everything is fine. However, after a day or two the problem will come right back.
Does anyone have ideas on this? It seems as if something is happening in an image created automatically by ngen.exe, but I have no idea where to start debugging. Thanks!
Update
I was finally able to catch a complete memory dump of the fault. It's a stack overflow problem at the core. Here is the stack trace, with the repeating bits...
System.Data.Query.InternalTrees.BasicOpVisitorOfT`1[[System.__Canon, mscorlib]].VisitNode(System.Data.Query.InternalTrees.Node) 086a9990 0609755c 0f16e1a0 59c4a149
System.Data.Query.InternalTrees.NodeInfoVisitor.RecomputeNodeInfo(System.Data.Query.InternalTrees.Node) 01f8f258 0609755c 06412d6c 0f16e1ec
System.Data.Query.InternalTrees.Node.InitializeNodeInfo(System.Data.Query.InternalTrees.Command) 086a97f0 59c64fb9 0f16e1b8 5a1d255e
System.Data.Query.InternalTrees.Node.GetExtendedNodeInfo(System.Data.Query.InternalTrees.Command) 06412da4 0f16e204 59c64ef1 06412da4
System.Data.Query.InternalTrees.BasicOpVisitorOfT`1[[System.__Canon, mscorlib]].Visit(System.Data.Query.InternalTrees.OuterApplyOp, System.Data.Query.InternalTrees.Node) 06412da4 0f16e220 59c12799 5996fb84
System.Data.Query.InternalTrees.OuterApplyOp.Accept[[System.__Canon, mscorlib]](System.Data.Query.InternalTrees.BasicOpVisitorOfT`1<System.__Canon>, System.Data.Query.InternalTrees.Node) 5996fb84 06412da4 01f8f098 06412da4
System.Data.Query.InternalTrees.BasicOpVisitorOfT`1[[System.__Canon, mscorlib]].VisitNode(System.Data.Query.InternalTrees.Node) 086a97f0 06412da4 0f16e244 59c4a149
System.Data.Query.InternalTrees.NodeInfoVisitor.RecomputeNodeInfo(System.Data.Query.InternalTrees.Node) 01f8f258 06412da4 086a9650 0f16e2b4
System.Data.Query.InternalTrees.Node.InitializeNodeInfo(System.Data.Query.InternalTrees.Command) 06097914 59c4a751 0f16e26c 00000000
System.Data.Query.InternalTrees.Node.GetExtendedNodeInfo(System.Data.Query.InternalTrees.Command) 0609794c 0f16e2dc 59c12799 599713cc
System.Data.Query.InternalTrees.ProjectOp.Accept[[System.__Canon, mscorlib]](System.Data.Query.InternalTrees.BasicOpVisitorOfT`1<System.__Canon>, System.Data.Query.InternalTrees.Node) 599713cc 0609794c 01f8f098 0609794c
System.Data.Query.InternalTrees.BasicOpVisitorOfT`1[[System.__Canon, mscorlib]].VisitNode(System.Data.Query.InternalTrees.Node) 086a9650 0609794c 0f16e300 59c4a149
System.Data.Query.InternalTrees.NodeInfoVisitor.RecomputeNodeInfo(System.Data.Query.InternalTrees.Node) 086a94b0 0609794c 0f16e334 0f16e354
System.Data.Query.InternalTrees.Node.InitializeNodeInfo(System.Data.Query.InternalTrees.Command) 086a94b0 03be2b5c 01f8f258 06097c50
System.Data.Query.InternalTrees.NodeInfoVisitor.Visit(System.Data.Query.InternalTrees.PhysicalProjectOp, System.Data.Query.InternalTrees.Node) 06097c88 0f16e37c 59c12799 59971884
System.Data.Query.InternalTrees.PhysicalProjectOp.Accept[[System.__Canon, mscorlib]](System.Data.Query.InternalTrees.BasicOpVisitorOfT`1<System.__Canon>, System.Data.Query.InternalTrees.Node) 59971884 06097c88 01f8f098 06097c88
System.Data.Query.InternalTrees.BasicOpVisitorOfT`1[[System.__Canon, mscorlib]].VisitNode(System.Data.Query.InternalTrees.Node) 086a94b0 06097c88 0f16e3a0 59c4a149
System.Data.Query.InternalTrees.NodeInfoVisitor.RecomputeNodeInfo(System.Data.Query.InternalTrees.Node) 06097c88 06097c88 086a9460 0f16e3d4
System.Data.Query.InternalTrees.Node.InitializeNodeInfo(System.Data.Query.InternalTrees.Command) 086a9494 59c49e5c 0f16e3c8 59c0ffb8
System.Data.Query.PlanCompiler.TransformationRulesContext.GetHashCode(System.Data.Query.InternalTrees.Node) 00000000 00000000 06097c88 01deb0f0
System.Data.Query.PlanCompiler.TransformationRules.Process(System.Data.Query.PlanCompiler.PlanCompiler, System.Collections.ObjectModel.ReadOnlyCollection`1<System.Collections.ObjectModel.ReadOnlyCollection`1<System.Data.Query.InternalTrees.Rule>>, Boolean ByRef) 0f16e40c 00000000 00000000 06b605bc
System.Data.Query.PlanCompiler.TransformationRules.Process(System.Data.Query.PlanCompiler.PlanCompiler, System.Data.Query.PlanCompiler.TransformationRulesGroup) 05d6bd90 06a8ac64 06b605bc 0f16e4f4
System.Data.Query.PlanCompiler.PlanCompiler.Compile(System.Collections.Generic.List`1<System.Data.Query.PlanCompiler.ProviderCommandInfo> ByRef, System.Data.Query.InternalTrees.ColumnMap ByRef, Int32 ByRef, System.Data.Common.Utils.Set`1<System.Data.Metadata.Edm.EntitySet> ByRef) 01f8f04c 0f16e4ec 0f16e4f0 00000000
System.Data.EntityClient.EntityCommandDefinition..ctor(System.Data.Common.DbProviderFactory, System.Data.Common.CommandTrees.DbCommandTree) 01f8efe0 01de3f40 01c645c4 0213e214
System.Data.EntityClient.EntityProviderServices.CreateCommandDefinition(System.Data.Common.DbProviderFactory, System.Data.Common.CommandTrees.DbCommandTree) 0213e214 01de3f40 01c645c4 0213e214
System.Data.EntityClient.EntityProviderServices.CreateDbCommandDefinition(System.Data.Common.DbProviderManifest, System.Data.Common.CommandTrees.DbCommandTree) 0213e214 0208a8f0 01f47934 0213e214
System.Data.Common.DbProviderServices.CreateCommandDefinition(System.Data.Common.CommandTrees.DbCommandTree) 00000000 00000000 00000000 00000000
System.Data.Objects.Internal.ObjectQueryExecutionPlan.Prepare(System.Data.Objects.ObjectContext, System.Data.Common.CommandTrees.DbQueryCommandTree, System.Type, System.Data.Objects.MergeOption, System.Data.Objects.Span, System.Collections.ObjectModel.ReadOnlyCollection`1<System.Collections.Generic.KeyValuePair`2<System.Data.Objects.ObjectParameter,System.Data.Objects.ELinq.QueryParameterExpression>>) 00000000 00000000 00000003 01f5b740
System.Data.Objects.ELinq.ELinqQueryState.GetExecutionPlan(System.Nullable`1<System.Data.Objects.MergeOption>) 00000000 00000000 01f5c228 00000000
System.Data.Objects.ObjectQuery`1[[System.__Canon, mscorlib]].GetResults(System.Nullable`1<System.Data.Objects.MergeOption>) 00000000 00000000 0f16e670 0208a8cc
System.Data.Objects.ObjectQuery`1[[System.__Canon, mscorlib]].System.Collections.Generic.IEnumerable<T>.GetEnumerator() 00000000 00000000 0f16e704 00000000
System.Linq.Buffer`1[[System.__Canon, mscorlib]]..ctor(System.Collections.Generic.IEnumerable`1<System.__Canon>) 0f4cb654 00000000 00000000 00000000
System.Linq.Enumerable.ToArray[[System.__Canon, mscorlib]](System.Collections.Generic.IEnumerable`1<System.__Canon>) 02089768 020883d4 020881fc 02088104
OptimizerCore.Models.agField.GetFieldPreloadMidpointInformation(System.Collections.Generic.IEnumerable`1<Int32>) 05e2e50c 05e2e50c 5d664381 05db9a64
System.Web.Mvc.ActionMethodDispatcher.Execute(System.Web.Mvc.ControllerBase, System.Object[]) 05db9a64 05e22d60 05daee58 01d68350
System.Web.Mvc.ReflectedActionDescriptor.Execute(System.Web.Mvc.ControllerContext, System.Collections.Generic.IDictionary`2<System.String,System.Object>) 05daf334 05e23b4c 05e23b0c 05db9a3c
System.Web.Mvc.ControllerActionInvoker.InvokeActionMethod(System.Web.Mvc.ControllerContext, System.Web.Mvc.ActionDescriptor, System.Collections.Generic.IDictionary`2<System.String,System.Object>) 05daf334 05e22ca0 05e23bcc 00000000
System.Web.Mvc.ControllerActionInvoker+<>c__DisplayClass15.<InvokeActionMethodWithFilters>b__12() 0f16e90c 05daf394 05daf1b0 00000000
System.Web.Mvc.ControllerActionInvoker.InvokeActionMethodFilter(System.Web.Mvc.IActionFilter, System.Web.Mvc.ActionExecutingContext, System.Func`1<System.Web.Mvc.ActionExecutedContext>) 05daf3b8 05e23c00 5d66622e 61954b4e
System.Web.Mvc.ControllerActionInvoker+<>c__DisplayClass15+<>c__DisplayClass17.<InvokeActionMethodWithFilters>b__14() 05daf3ec 05e23b0c 5d666580 05e22ca0
System.Web.Mvc.ControllerActionInvoker+<>c__DisplayClass15+<>c__DisplayClass17.<InvokeActionMethodWithFilters>b__14() 05daf334 05daee58 05daf0b4 05daef68
System.Web.Mvc.ControllerActionInvoker.InvokeAction(System.Web.Mvc.ControllerContext, System.String) 05bf96e4 05bf961c 05c0974c 00000000
System.Web.Mvc.Controller.ExecuteCore() 01d6842c 00000000 00000000 00000000
System.Web.Mvc.ControllerBase.Execute(System.Web.Routing.RequestContext) 0f16eb08 5d672ca6 05bfdc94 0f16eb2c
System.Web.Mvc.ControllerBase.System.Web.Mvc.IController.Execute(System.Web.Routing.RequestContext) 05bfdc94 0f16eb2c 5d66ad06 00000000
System.Web.Mvc.MvcHandler+<>c__DisplayClass6+<>c__DisplayClassb.<BeginProcessRequest>b__5() 00000000 01dcf71c 5d68febd 5d68fcdf
System.Web.Mvc.Async.AsyncResultWrapper+<>c__DisplayClass1.<MakeVoidDelegate>b__0() 5d5b8c28 01dcf7ac 0f16eb4c 5d65aaf8
System.Web.Mvc.MvcHandler+<>c__DisplayClasse.<EndProcessRequest>b__d() 5d65ab27 01dcf7a0 0f16eb5c 5d67244d
System.Web.Mvc.SecurityUtil.<GetCallInAppTrustThunk>b__0(System.Action) 01dcf71c 05bfbb70 0f16eb64 5d67233a
System.Web.Mvc.MvcHandler.EndProcessRequest(System.IAsyncResult) 0f16eb90 5e0ef242 05be7470 00000000
System.Web.Mvc.MvcHandler.System.Web.IHttpAsyncHandler.EndProcessRequest(System.IAsyncResult) 05be7470 00000000 00000000 00000000
System.Web.HttpApplication+CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute() 0f16ec5c 61d1965e 61d19673 00000000
System.Web.HttpApplication.ExecuteStep(IExecutionStep, Boolean ByRef) 0f16ec24 05d11148 00000000 05bebd0c
System.Web.HttpApplication+PipelineStepManager.ResumeSteps(System.Exception) 00000000 00000000 0f16eca0 0f16ecb0
System.Web.HttpApplication.BeginProcessRequestNotification(System.Web.HttpContext, System.AsyncCallback) 05bc3134 05be7294 05bc2e34 00000080
System.Web.HttpRuntime.ProcessRequestNotificationPrivate(System.Web.Hosting.IIS7WorkerRequest, System.Web.HttpContext) 05be7470 9572145a 61cf5d80 0f16f218
System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, IntPtr, Int32) 00000000 0000000c 00000000 00000000
System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr, IntPtr, IntPtr, Int32) 00c11198 01872c94 0000000c 00000000
webengine4!W3_MGD_HANDLER::ProcessNotification+5b 01871ba8 00000100 00000000 0f16ee1c
webengine4!W3_MGD_HANDLER::DoWork+250 00000080 00000000 00efb22c 01871c00
webengine4!RequestDoWork+2da 00000080 00000000 00efb22c 01871ba8
webengine4!CMgdEngHttpModule::OnExecuteRequestHandler+1a 01871ba8 01871c00 01871bac 0f16eec8
iiscore!NOTIFICATION_CONTEXT::RequestDoWork+128 00efb22c 01871ba8 00000000 01871c00
iiscore!NOTIFICATION_CONTEXT::CallModulesInternal+305 00000000 00000000 00000000 01871ba8
iiscore!NOTIFICATION_CONTEXT::CallModules+28 00000000 00000000 00000000 01871ba8
iiscore!W3_CONTEXT::DoStateRequestExecuteHandler+36 00000000 00000000 00000000 00000000
iiscore!W3_CONTEXT::DoWork+d7 00000000 00000000 00000001 01871ba8
iiscore!W3_MAIN_CONTEXT::ContinueNotificationLoop+1f 00000000 00000000 768c1484 0f16f1e0
iiscore!W3_MAIN_CONTEXT::ProcessIndicateCompletion+1f 00000000 01872c94 0f16f1f4 71b4d641
iiscore!W3_CONTEXT::IndicateCompletion+75 00000000 0f16f2a8 0f16f218 0f16f204
webengine4!W3_MGD_HANDLER::IndicateCompletion+3d 00000000 0e8ecc18 0f16f254 5d8d8bbd
webengine4!MgdIndicateCompletion+24 01872c94 0f16f2a8 9572145a 61cf5d80
DomainNeutralILStubClass.IL_STUB_PInvoke(IntPtr, System.Web.RequestNotificationStatus ByRef) 9572145a 61cf5d80 0f16f3ec 5d7467e4
System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, IntPtr, Int32) 00000004 00000003 00000000 00000000
System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr, IntPtr, IntPtr, Int32) 0e8ecc18 0f16f3a0 61fbf077 0f16f510
clr!UM2MThunk_WrapperHelper+10 0f16f510 00000010 00671fa3 00672038
clr!UM2MThunk_Wrapper+8c 0f16f4a8 9a0857ca 0e8ecc18 00000002
clr!Thread::DoADCallBack+f0 00000002 61fbefeb 0f16f4a8 00000001
clr!UM2MDoADCallBack+c0 01924570 ffffffff 0f16f510 00000010
0x00672020 00000000 01872c94 00000003 00000004
webengine4!W3_MGD_HANDLER::ProcessNotification+5b 01872c94 71b508c8 0f16f58c 61e75eaf
webengine4!ProcessNotificationCallback+36 01872c94 9a0856c6 0000ffff 00010002
clr!UnManagedPerAppDomainTPCount::DispatchWorkItem+195 0f16f5ef 0f16f5ee 9a0856b6 00000000
clr!ThreadpoolMgr::NewWorkerThreadStart+20b 00000000 9a08552e 00000000 61e76a99
clr!ThreadpoolMgr::WorkerThreadStart+3d1 00000000 0000027f 00000000 00000000
clr!Thread::intermediateThreadProc+4b 0e8b0b60 0f16f854 778e9f42 0e8b0b60
kernel32!BaseThreadInitThunk+e 0e8b0b60 78e16e6f 00000000 00000000
ntdll!__RtlUserThreadStart+70 61da59c0 0e8b0b60 ffffffff 779775ad
ntdll!_RtlUserThreadStart+1b
This bit in particular seems to go recursive and eventually hard fault:
System.Data.Query.InternalTrees.Node.GetExtendedNodeInfo(System.Data.Query.InternalTrees.Command) 0609794c 0f16e2dc 59c12799 599713cc
System.Data.Query.InternalTrees.ProjectOp.Accept[[System.__Canon, mscorlib]](System.Data.Query.InternalTrees.BasicOpVisitorOfT`1<System.__Canon>, System.Data.Query.InternalTrees.Node) 599713cc 0609794c 01f8f098 0609794c
System.Data.Query.InternalTrees.BasicOpVisitorOfT`1[[System.__Canon, mscorlib]].VisitNode(System.Data.Query.InternalTrees.Node) 086a9650 0609794c 0f16e300 59c4a149
System.Data.Query.InternalTrees.NodeInfoVisitor.RecomputeNodeInfo(System.Data.Query.InternalTrees.Node) 086a94b0 0609794c 0f16e334 0f16e354
System.Data.Query.InternalTrees.Node.InitializeNodeInfo(System.Data.Query.InternalTrees.Command)
I believe this is the chunk of code that causes the problem: (where fieldIDs is an int array containing about 1,600 items):
var query = db.BaseDB.agZoneCoordinates
.Where(x => fieldIDs.Contains(x.agZone.FieldID))
.Select(x => new {
x.agZone.FieldID,
x.Latitude,
x.Longitude
})
.AsNoTracking()
.ToArray();
I've tried reducing the query size so that it works in chunks of 200 or so, but it's still crashing. Are there any Entity Framework developers out there that have seen this happen? What's making EF go recursive?
jocull2 Answers
We recently ran into a stack overflow very much like yours where we were doing something very similar to your scenario (a large list of Ids) and EF usage except we were also calling Distinct on the list of Ids we were passing. So, building on your example, our call looked something like:
var query = db.BaseDB.agZoneCoordinates
.Where(x => fieldIDs.Distinct().Contains(x.agZone.FieldID))
.Select(x => new {
x.agZone.FieldID,
x.Latitude,
x.Longitude
})
.AsNoTracking()
.ToArray();
Since the lambda is for an IQueryable, this has the effect of creating a huge expression tree that EF has to deal with and, apparently, it doesn't handle it well. We solved our problem and actually improved performance for large lists by doing the Distinct call outside of the lambda. FWIW, I tried it w/ a large list (100K) w/o the Distinct call and that also worked so our issue was definitely w/ having the Distinct call as part of the lambda rather than the sheer size of the list.
I would consider taking your statement and changing it to have an Include. I'm pretty sure that otherwise the agZone is being lazy loaded within the select:
var query = db.BaseDB.agZoneCoordinates
.Where(x => fieldIDs.Contains(x.agZone.FieldID))
.Include(x=>x.agZone)
.Select(x => new {
x.agZone.FieldID,
x.Latitude,
x.Longitude
})
.AsNoTracking()
.ToArray();
If you are using EF 4 and not 4.1 or newer, you'll have to write it without the lambda.
var query = db.BaseDB.agZoneCoordinates
.Where(x => fieldIDs.Contains(x.agZone.FieldID))
.Include("BaseDb.agZone") //May require tweaking to how your model is defined
.Select(x => new {
x.agZone.FieldID,
x.Latitude,
x.Longitude
})
.AsNoTracking()
.ToArray();
answered on Stack Overflow Mar 28, 2015 by 
to11mtm
to11mtmUser contributions licensed under CC BY-SA 3.0