I am fighting a faddish problem. After several days of running Asp.Net 4.5 site becomes not responsible. My configuration: Windows 7, IIS 7.5, 64 bit, 16Gb. Memory consumption of application is small and should not be an issue (w3wp.exe takes around 500Mb of memory during its lifetime).
I've checked the Application Pool and it was in running state. So my first guess was that there should be a deadlock inside garbage collection process. All threads in applicateon are suspended by GC and new requests are not allowed to be assigned new threads, so all requests just hang.
So I've collected the process dump and gave it to DebugDiag for analyzing. And memory analysis confirmed my guess. It issued me a warning:
The process is currently in the middle of a garbage collection
Then I switched to crush-hang analysis to look at the threads. Usually the reason fo GC to hang is having a thread with Preemptive mode disabled. By all threads in my application looked just fine:
Debugger Thread ID Managed Thread ID OS Thread ID Thread Object GC Mode Domain Lock Count Apt Exception
7 1 3552 14601f0 Preemptive 1456e80 0 Unknown
15 2 3476 c5b31e0 Preemptive 1456e80 0 MTA (Finalizer)
17 3 4696 c608ee0 Preemptive 1456e80 0 MTA (Threadpool Worker)
18 4 3756 c60cb10 Preemptive 1456e80 0 Unknown
19 6 4672 d6e51d0 Preemptive 1456e80 0 Unknown (Threadpool Worker)
20 7 4652 c6caf90 Preemptive c61f8f0 0 MTA
9 9 4684 d749120 Preemptive 1456e80 0 MTA (Threadpool Completion Port)
21 10 3220 d738fa0 Preemptive c61f8f0 0 MTA
22 11 3084 d74af40 Preemptive c61f8f0 0 MTA
23 12 4280 d740450 Preemptive c61f8f0 0 MTA
24 13 1212 d740bf0 Preemptive c61f8f0 0 MTA
25 14 1012 d741390 Preemptive c61f8f0 0 MTA
26 17 3548 d802660 Preemptive 1456e80 0 Unknown (GC Thread)
27 18 1800 d802e30 Preemptive 1456e80 0 Unknown (GC Thread)
28 19 4088 d803600 Preemptive 1456e80 0 Unknown (GC Thread)
29 20 2880 d803dd0 Preemptive 1456e80 0 Unknown (GC Thread)
3 8 3872 d805d10 Preemptive 1456e80 0 Unknown
54 eac7280 Preemptive 1456e80 0
100 eae16b0 Preemptive 1456e80 0
94 da67c80 Preemptive 1456e80 0
107 d6daf30 Preemptive 1456e80 0
76 d6db700 Preemptive 1456e80 0
56 eaecf40 Preemptive 1456e80 0
86 146b210 Preemptive 1456e80 0
65 d82c930 Preemptive 1456e80 0
102 da43750 Preemptive 1456e80 0
101 d90e220 Preemptive 1456e80 0
25 eb18770 Preemptive 1456e80 0
85 da64570 Preemptive 1456e80 0
88 d898820 Preemptive 1456e80 0
89 eb17fa0 Preemptive 1456e80 0
93 da5fdd0 Preemptive 1456e80 0
64 d82d8d0 Preemptive 1456e80 0
81 d8d4260 Preemptive 1456e80 0
31 92 3340 eaf17d0 Preemptive 1456e80 0 MTA (Threadpool Worker)
32 79 4012 eaf1fa0 Preemptive 1456e80 0 MTA (Threadpool Worker)
55 eaebfa0 Preemptive 1456e80 0
80 eaf2770 Preemptive 1456e80 0
108 d82c160 Preemptive 1456e80 0
50 d90da50 Preemptive 1456e80 0
49 d8d4a30 Preemptive 1456e80 0
48 eaf2f40 Preemptive 1456e80 0
33 84 3556 eb16770 Preemptive 1456e80 0 MTA (Threadpool Worker)
35 103 740 d808bf0 Preemptive 1456e80 0 MTA (Threadpool Worker)
34 68 4160 d82d100 Preemptive 1456e80 0 MTA (Threadpool Worker)
66 eaeb7d0 Preemptive 1456e80 0
98 eae1e80 Preemptive 1456e80 0
36 91 5048 eaec770 Preemptive 1456e80 0 Unknown
37 63 4768 da61540 Preemptive 1456e80 0 MTA (Threadpool Worker)
38 75 2300 da64d40 Preemptive 1456e80 0 MTA (Threadpool Worker)
39 99 3980 d90f1c0 Preemptive 1456e80 0 MTA (Threadpool Worker)
40 52 2512 eb15fa0 Preemptive 1456e80 0 MTA (Threadpool Worker)
41 90 3560 d8d3a90 Preemptive 1456e80 0 MTA (Threadpool Worker)
42 67 3624 d90e9f0 Preemptive 1456e80 0 MTA (Threadpool Worker)
44 53 1404 eae2e20 Preemptive 1456e80 0 MTA (Threadpool Worker)
43 74 4860 da60d70 Preemptive 1456e80 0 MTA (Threadpool Worker)
45 83 4424 da63da0 Preemptive 1456e80 0 MTA (Threadpool Worker)
46 106 1408 d8d5200 Preemptive 1456e80 0 MTA (Threadpool Worker)
47 61 3100 da605a0 Preemptive 1456e80 0 Unknown
48 58 2228 eac5b10 Preemptive 1456e80 0 MTA (Threadpool Worker)
49 43 1448 d807c50 Preemptive 1456e80 0 Unknown
Moreover, looking through the callstacks I was not able to find some call that would indicate that GC process was really triggered. I was looking for something like GarbageCollectGeneration
or (GC) (Threadpool Worker)
but none of that was found.
So I am king of uncertain: do I really have a GC running or not? And if I do then which thread triggered it? And why is it not completing?
Another interesting observation in the dump is 'Previous .NET Exceptions' section. It says:
Exception Type Count Message Stack Trace
System.Exception 1 <none>
System.OutOfMemoryException 1 <none>
System.StackOverflowException 1 <none>
System.ExecutionEngineException 1 <none>
System.Threading.ThreadAbortException 2 <none>
System.Web.HttpException 2 The remote host closed the connection. The error code is 0x800704CD.
I understand that having such things like OutOfMemoryException
, StackOverflowException
or ExecutionEngineException
is no good. But could the be the reason for the site running out of servece? And how do I track those exceptions if the stack is not available?
User contributions licensed under CC BY-SA 3.0