further steps for analysing .NET memory leak in WinDbg

An application when opening large documents (400 pages of scanned documents) recently started getting an error, the developer of the application is having trouble finding root cause so we are providing some assistance. Apparently this has started occurring without a change to the application.

From what I've done so far can someone please recommend next debugging steps?

0:000> !pe
Exception object: 07559bb0
Exception type:   System.IO.FileLoadException
Message:          Could not load file or assembly 'System.ServiceModel, Version=3.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089' or one of its dependencies. Not enough storage is available to process this command. (Exception from HRESULT: 0x80070008)
InnerException:   <none>
StackTrace (generated):
    SP       IP       Function
    00000000 00000001 UNKNOWN!DocumentAutomation.Common.ServiceAdaptor.ESBV4ServicesAdaptor..ctor(System.String, System.String, System.String[,], System.String, System.TimeSpan)+0x2
    005DE908 04CE30E5 DocumentAutomation_Indexing!DocumentAutomation.Indexing.ViewModels.IndexingViewModel.OnValidateCase()+0x265

StackTraceString: <none>
HResult: 80070008

Looking at memory usage can see heap is large but managed objects size relatively small, also comparing against dmp files as heap grows these objects are only growing by a small amount.

    0:000> !address -summary


    Mapping file section regions...
    Mapping module regions...
    Mapping PEB regions...
    Mapping TEB and stack regions...
    Mapping heap regions...
    Mapping page heap regions...
    Mapping other regions...
    Mapping stack trace database regions...
    Mapping activation context regions...

    --- Usage Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal
    Heap                                    344          5ad52000 (   1.419 GB)  78.89%   70.97%
    Free                                    320           cdd5000 ( 205.832 MB)           10.05%
    Image                                  1037           c462000 ( 196.383 MB)  10.66%    9.59%
    <unknown>                               757           b053000 ( 176.324 MB)   9.57%    8.61%
    Stack                                    55            fc0000 (  15.750 MB)   0.85%    0.77%
    Other                                     9             41000 ( 260.000 kB)   0.01%    0.01%
    TEB                                      18             12000 (  72.000 kB)   0.00%    0.00%
    PEB                                       1              1000 (   4.000 kB)   0.00%    0.00%

    --- Type Summary (for busy) ------ RgnCount ----------- Total Size -------- %ofBusy %ofTotal
    MEM_PRIVATE                             705          60ec8000 (   1.514 GB)  84.19%   75.72%
    MEM_IMAGE                              1467           d1cd000 ( 209.801 MB)  11.39%   10.24%
    MEM_MAPPED                               49           5186000 (  81.523 MB)   4.43%    3.98%

    --- State Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal
    MEM_COMMIT                             1686          6e10d000 (   1.720 GB)  95.60%   85.99%
    MEM_FREE                                320           cdd5000 ( 205.832 MB)           10.05%
    MEM_RESERVE                             535           510e000 (  81.055 MB)   4.40%    3.96%

    --- Protect Summary (for commit) - RgnCount ----------- Total Size -------- %ofBusy %ofTotal
    PAGE_READWRITE                          635          5d9f3000 (   1.463 GB)  81.32%   73.14%
    PAGE_EXECUTE_READ                       118           8ac2000 ( 138.758 MB)   7.53%    6.78%
    PAGE_READONLY                           496           623a000 (  98.227 MB)   5.33%    4.80%
    PAGE_WRITECOPY                          252           1681000 (  22.504 MB)   1.22%    1.10%
    PAGE_EXECUTE_READWRITE                   97            1f5000 (   1.957 MB)   0.11%    0.10%
    PAGE_EXECUTE_WRITECOPY                   50            14c000 (   1.297 MB)   0.07%    0.06%
    PAGE_READWRITE|PAGE_GUARD                36             5a000 ( 360.000 kB)   0.02%    0.02%
    PAGE_NOACCESS                             2              2000 (   8.000 kB)   0.00%    0.00%

    --- Largest Region by Usage ----------- Base Address -------- Region Size ----------
    Heap                                         e570000            fcf000 (  15.809 MB)
    Free                                        5e820000            d80000 (  13.500 MB)
    Image                                       57388000            f1a000 (  15.102 MB)
    <unknown>                                    2120000           1cae000 (  28.680 MB)
    Stack                                        2000000             fd000 (1012.000 kB)
    Other                                       7ed90000             23000 ( 140.000 kB)
    TEB                                         7ebf7000              1000 (   4.000 kB)
    PEB                                         7edbe000              1000 (   4.000 kB)


    0:000> !heap -stat
    _HEAP 006f0000
         Segments            00000004
             Reserved  bytes 007fc000
             Committed bytes 005c5000
         VirtAllocBlocks     0000005d
             VirtAlloc bytes e255b09c
    _HEAP 00b80000
         Segments            00000009
             Reserved  bytes 04e47000
             Committed bytes 041cb000
         VirtAllocBlocks     00000000
             VirtAlloc bytes 00000000
    _HEAP 09170000
         Segments            00000002
             Reserved  bytes 0010e000
             Committed bytes 00018000
         VirtAllocBlocks     00000000
             VirtAlloc bytes 00000000
    _HEAP 00990000
         Segments            00000001
             Reserved  bytes 0000f000
             Committed bytes 0000f000
         VirtAllocBlocks     00000000
             VirtAlloc bytes 00000000
    _HEAP 04780000
         Segments            00000001
             Reserved  bytes 0000f000
             Committed bytes 00008000
         VirtAllocBlocks     00000000
             VirtAlloc bytes 00000000
    _HEAP 04750000
         Segments            00000001
             Reserved  bytes 0000f000
             Committed bytes 00007000
         VirtAllocBlocks     00000000
             VirtAlloc bytes 00000000
    _HEAP 02110000
         Segments            00000001
             Reserved  bytes 0000f000
             Committed bytes 00003000
         VirtAllocBlocks     00000000
             VirtAlloc bytes 00000000
    _HEAP 01ff0000
         Segments            00000001
             Reserved  bytes 0000f000
             Committed bytes 00001000
         VirtAllocBlocks     00000000
             VirtAlloc bytes 00000000
    0:000> !heap -s


    ************************************************************************************************************************
                                                  NT HEAP STATS BELOW
    ************************************************************************************************************************
    LFH Key                   : 0x271d989d
    Termination on corruption : DISABLED
      Heap     Flags   Reserv  Commit  Virt   Free  List   UCR  Virt  Lock  Fast 
                        (k)     (k)    (k)     (k) length      blocks cont. heap 
    -----------------------------------------------------------------------------
    006f0000 00000002    8284   5924   8176   1328   202     4   93      1   LFH
        External fragmentation  22 % (202 free blocks)
    00990000 00001002     168     64     60      9     7     1    0      0   LFH
    00b80000 00001002   80264  67380  80156    294    30     9    0      0   LFH
    02110000 00001002      60     12     60      3     6     1    0      0      
    01ff0000 00041002      60      4     60      2     1     1    0      0      
    04780000 00001002     168     36     60      2     3     1    0      0   LFH
    04750000 00041002     168     32     60      1     5     1    0      0   LFH
    09170000 00001002    1188    100   1080     50     5     2    0      0   LFH
    -----------------------------------------------------------------------------

    0:000> !eeheap -gc
    Number of GC Heaps: 1
    generation 0 starts at 0x07ca8d10
    generation 1 starts at 0x07c51000
    generation 2 starts at 0x02121000
    ephemeral segment allocation context: none
             segment             begin         allocated  size
    02120000  02121000  0311ffe4  0xffefe4(16773092)
    06ab0000  06ab1000  0757e71c  0xacd71c(11327260)
    07c50000  07c51000  0819fca8  0x54eca8(5565608)
    Large object heap starts at 0x03121000
             segment             begin         allocated  size
    03120000  03121000  03dadaa8  0xc8caa8(13159080)
    Total Size:              Size: 0x2ca7e50 (46825040) bytes.
    ------------------------------
    GC Heap Size:    Size: 0x2ca7e50 (46825040) bytes.

    0:000> !dumpheap -stat
    Statistics:

          MT    Count    TotalSize Class Name
etc...

    60f3b340     9234       258552 System.Reflection.Emit.ScopeTree
    60f6b8fc     4647       260232 System.Reflection.RuntimePropertyInfo
    60f642d0     8139       260448 System.EventHandler
    60f70b28     9622       269416 System.Collections.ArrayList+ArrayListEnumeratorSimple
    57501c04     3616       274816 System.Windows.Data.BindingExpression
    06256688     4414       300152 TiS.Core.eFlowAPI.TisROIMiscellaneousParams
    06256160     4414       300152 TiS.Core.eFlowAPI.TisROIPostProcessingParams
    06256b64     4622       314296 TiS.Core.eFlowAPI.TisOCRGroupParamsLinksContainer
    60f7a740     3312       318672 System.Runtime.CompilerServices.ConditionalWeakTable`2+Entry[[System.Object, mscorlib],[System.Object, mscorlib]][]
    574fbd18     1294       320912 System.Windows.Controls.Border
    60f4356c     9234       332424 System.Reflection.Emit.DynamicResolver
    60f763d4    30341       364092 System.WeakReference
    60f75a78     8363       367972 System.Signature
    60f3b1d0    11905       380960 System.Reflection.Emit.SignatureHelper
    60f71238     2920       424800 System.Collections.Hashtable+bucket[]
    009846b0     4196       429232 System.Collections.Generic.Dictionary`2+Entry[[System.Type, mscorlib],[System.Attribute[], mscorlib]][]
    60f6c290     9901       475248 System.RuntimeMethodInfoStub
    062551fc     4414       476712 TiS.Core.eFlowAPI.TisROIParams
    0500b718    15820       506240 TiS.Core.eFlowAPI.NamedObjectListEvent
    60f65af8     9234       517104 System.Reflection.Emit.DynamicMethod
    60f73acc    13901       641256 System.Int32[]
    60f6bed4    10749       644940 System.Reflection.RuntimeMethodInfo
    58eccce8     9114       846936 System.Windows.EffectiveValueEntry[]
    60f433b8     9234       886464 System.Reflection.Emit.DynamicILGenerator
    60f35738   107092      3501976 System.Object[]
    60f75670    51775      5672668 System.Byte[]
    60f721b4    57971      6750108 System.String
    00702bd0      223      8557414      Free
    Total 783513 objects
    Fragmented blocks larger than 0.5 MB:
        Addr     Size      Followed by
    08039088    1.4MB         08194298 System.String

Taking several dmp files as memory rapidly increases (within a minute or so heap increases from 300MB to 1GB+ while processing document) I do see several threads allocating to the heap always present with a stack like below ( igdumdim32 is intel user mode gfx driver)

Current frame: ntdll!NtRemoveIoCompletion+0xc
ChildEBP RetAddr  Caller, Callee
05e6e988 748f2dfa KERNELBASE!GetQueuedCompletionStatus+0x2a, calling ntdll!NtRemoveIoCompletion
05e6e9c0 673f94e1 clr!ThreadpoolMgr::CompletionPortThreadStart+0x22f, calling kernel32!GetQueuedCompletionStatusStub
05e6ea54 6734814d clr!Thread::intermediateThreadProc+0x4d
05e6ed20 76fb81b5 ntdll!bsearch+0x6f
05e6ed44 76fb83bb ntdll!RtlpLocateActivationContextSection+0xe3, calling ntdll!bsearch
05e6ed64 76fb882c ntdll!RtlpFindUnicodeStringInSection+0x193, calling ntdll!__security_check_cookie
05e6ed7c 71db107b mscoreei!ShimLog::Log+0x1a4, calling mscoreei!__security_check_cookie
05e6ed9c 76fb8295 ntdll!RtlpFindNextActivationContextSection+0x7a, calling ntdll!RtlpLocateActivationContextSection
05e6edb8 76fb85f2 ntdll!RtlFindActivationContextSectionString+0x146, calling ntdll!RtlpFindUnicodeStringInSection
05e6edf8 76fdb1b7 ntdll!LdrpApplyLookupReference+0x1e, calling ntdll!RtlIsCriticalSectionLockedByThread
05e6ee04 76fb8972 ntdll!LdrpFindLoadedDllByName+0x13d, calling ntdll!RtlReleaseSRWLockExclusive
05e6ee08 76fb8891 ntdll!RtlWow64EnableFsRedirectionEx+0x51, calling ntdll!_SEH_epilog4
05e6ee38 76fb8891 ntdll!RtlWow64EnableFsRedirectionEx+0x51, calling ntdll!_SEH_epilog4
05e6ee3c 76fc53eb ntdll!LdrpFindOrMapDll+0xfd6, calling ntdll!RtlWow64EnableFsRedirectionEx
05e6ee48 76fc544d ntdll!LdrpFindOrMapDll+0xf06, calling ntdll!LdrpFindOrMapDll+0xf40
05e6ee58 76fc54ac ntdll!LdrpFindOrMapDll+0x107e, calling ntdll!__security_check_cookie
05e6eeec 76fb8c78 ntdll!RtlDosApplyFileIsolationRedirection_Ustr+0x2d8, calling ntdll!memset
05e6efb4 76fb83bb ntdll!RtlpLocateActivationContextSection+0xe3, calling ntdll!bsearch
05e6efd4 76fb882c ntdll!RtlpFindUnicodeStringInSection+0x193, calling ntdll!__security_check_cookie
05e6f00c 76fb8295 ntdll!RtlpFindNextActivationContextSection+0x7a, calling ntdll!RtlpLocateActivationContextSection
05e6f028 76fb85f2 ntdll!RtlFindActivationContextSectionString+0x146, calling ntdll!RtlpFindUnicodeStringInSection
05e6f05c 76fb9493 ntdll!LdrpApplyFileNameRedirection+0x96, calling ntdll!RtlDosApplyFileIsolationRedirection_Ustr
05e6f08c 76fb9424 ntdll!LdrpApplyFileNameRedirection+0xac, calling ntdll!__security_check_cookie
05e6f0ac 76fc5c5b ntdll!LdrpFindOrMapDependency+0x209, calling ntdll!LdrpFindOrMapDll
05e6f0c8 76fc5c79 ntdll!LdrpFindOrMapDependency+0x227, calling ntdll!LdrpFindOrMapDependency+0x2d9
05e6f0d8 76fc5cba ntdll!LdrpFindOrMapDependency+0x268, calling ntdll!__security_check_cookie
05e6f15c 76fb8972 ntdll!LdrpFindLoadedDllByName+0x13d, calling ntdll!RtlReleaseSRWLockExclusive
05e6f164 76fb8891 ntdll!RtlWow64EnableFsRedirectionEx+0x51, calling ntdll!_SEH_epilog4
05e6f194 76fb8891 ntdll!RtlWow64EnableFsRedirectionEx+0x51, calling ntdll!_SEH_epilog4
05e6f198 76fcff38 ntdll!LdrpFindLoadedDll+0x2ea, calling ntdll!RtlWow64EnableFsRedirectionEx
05e6f1ac 76fee967 ntdll!LdrpFindLoadedDll+0x342, calling ntdll!__security_check_cookie
05e6f1bc 76fb8891 ntdll!RtlWow64EnableFsRedirectionEx+0x51, calling ntdll!_SEH_epilog4
05e6f304 76fc4d3e ntdll!LdrpGetProcedureAddress+0x3d, calling ntdll!RtlImageNtHeaderEx
05e6f350 76f8e385 ntdll!LdrpResolveNonStaticDependency+0x22f, calling ntdll!LdrpDereferenceNode
05e6f384 76fc5a4d ntdll!LdrpReleaseModuleEnumLock+0x19, calling ntdll!RtlReleaseSRWLockShared
05e6f398 76fc5a0b ntdll!LdrpReleaseLoaderLock+0x1a, calling ntdll!LdrpReleaseModuleEnumLock
05e6f3a8 76f8e47c ntdll!LdrGetProcedureAddressForCaller+0x312, calling ntdll!LdrpReleaseLoaderLock
05e6f3c4 748bf769 KERNELBASE!_KernelBaseBaseDllInitialize+0x313, calling KERNELBASE!RegKrnInitialize
05e6f3d4 748bf77a KERNELBASE!_KernelBaseBaseDllInitialize+0x324, calling KERNELBASE!__security_check_cookie
05e6f45c 76fd04c3 ntdll!RtlpLowFragHeapAllocFromContext+0x34d, calling ntdll!memset
05e6f4d8 76fb10e9 ntdll!RtlAllocateHeap+0xc5, calling ntdll!RtlpLowFragHeapAllocFromContext
05e6f4f0 76fb10e9 ntdll!RtlAllocateHeap+0xc5, calling ntdll!RtlpLowFragHeapAllocFromContext
05e6f518 76fb10e9 ntdll!RtlAllocateHeap+0xc5, calling ntdll!RtlpLowFragHeapAllocFromContext
05e6f528 7479b2e4 msvcrt!_VEC_memzero+0x36, calling msvcrt!fastzero_I
05e6f550 6315ba71 igdumdim32!OpenAdapter+0x15aa71, calling ntdll!RtlLeaveCriticalSection
05e6f55c 631550ef igdumdim32!OpenAdapter+0x1540ef, calling igdumdim32!OpenAdapter+0x15aa5e
05e6f564 631550d8 igdumdim32!OpenAdapter+0x1540d8, calling igdumdim32!OpenAdapter+0x155225
05e6f590 631550d8 igdumdim32!OpenAdapter+0x1540d8, calling igdumdim32!OpenAdapter+0x155225
05e6f594 63151741 igdumdim32!OpenAdapter+0x150741, calling igdumdim32!OpenAdapter+0x15403e
05e6f5a0 681b30a1 msctfui!_CRT_INIT+0x1ea, calling msctfui!__security_check_cookie
05e6f5ac 681a32c6 msctfui!DllMain+0x10, calling kernel32!GetCurrentThreadId
05e6f5bc 681b31f7 msctfui!__DllMainCRTStartup+0x107, calling msctfui!DllMain
05e6f5c4 681b366e msctfui!_SEH_epilog4_GS+0xa, calling msctfui!__security_check_cookie
05e6f5c8 681b3379 msctfui!__DllMainCRTStartup+0x289, calling msctfui!_SEH_epilog4_GS
05e6f608 631518a5 igdumdim32!OpenAdapter+0x1508a5, calling igdumdim32!OpenAdapter+0x155225
05e6f624 76fb96c4 ntdll!RtlDeactivateActivationContextUnsafeFast+0x9c, calling ntdll!__security_check_cookie
05e6f640 76fb9758 ntdll!LdrpCallInitRoutine+0x43, calling ntdll!LdrxCallInitRoutine
05e6f650 76fb9776 ntdll!LdrpCallInitRoutine+0x61, calling ntdll!_SEH_epilog4
05e6f668 76fc5a4d ntdll!LdrpReleaseModuleEnumLock+0x19, calling ntdll!RtlReleaseSRWLockShared
05e6f67c 76fc5a0b ntdll!LdrpReleaseLoaderLock+0x1a, calling ntdll!LdrpReleaseModuleEnumLock
05e6f68c 76fd5b55 ntdll!LdrpInitializeThread+0x1af, calling ntdll!LdrpReleaseLoaderLock
05e6f698 76fd5b43 ntdll!LdrpInitializeThread+0x139, calling ntdll!_SEH_epilog4
05e6f6ec 76fd5ab4 ntdll!LdrpInitializeThread+0xeb, calling ntdll!RtlActivateActivationContextUnsafeFast
05e6f6f0 76fd5b1d ntdll!LdrpInitializeThread+0x1a4, calling ntdll!RtlDeactivateActivationContextUnsafeFast
05e6f72c 76fcaa56 ntdll!_LdrpInitialize+0x80, calling ntdll!_SEH_epilog4
05e6f774 76fca9ca ntdll!LdrInitializeThunk+0x1a, calling ntdll!NtContinue
05e6f9e0 6734813b clr!Thread::intermediateThreadProc+0x3b, calling clr!_alloca_probe_16
05e6f9f4 74cc7c04 kernel32!BaseThreadInitThunk+0x24
05e6fa08 76fcad1f ntdll!__RtlUserThreadStart+0x2f
05e6fa50 76fcacea ntdll!_RtlUserThreadStart+0x1b, calling ntdll!__RtlUserThreadStart

These dmp files also contain several CLR exceptions on the heap, but have no StackTrace present, these exceptions are present even before the program crashes:

0:000> .foreach (ex {!dumpheap -type Exception -short}){.echo "********************************";!pe ${ex} } 
********************************
Exception object: 02ac1024
Exception type:   System.Exception
Message:          <none>
InnerException:   <none>
StackTrace (generated):
<none>
StackTraceString: <none>
HResult: 80131500
********************************
Exception object: 02ac1078
Exception type:   System.OutOfMemoryException
Message:          <none>
InnerException:   <none>
StackTrace (generated):
<none>
StackTraceString: <none>
HResult: 8007000e
********************************
Exception object: 02ac10cc
Exception type:   System.StackOverflowException
Message:          <none>
InnerException:   <none>
StackTrace (generated):
<none>
StackTraceString: <none>
HResult: 800703e9
********************************
Exception object: 02ac1120
Exception type:   System.ExecutionEngineException
Message:          <none>
InnerException:   <none>
StackTrace (generated):
<none>
StackTraceString: <none>
HResult: 80131506
********************************
Exception object: 02ac1174
Exception type:   System.Threading.ThreadAbortException
Message:          <none>
InnerException:   <none>
StackTrace (generated):
<none>
StackTraceString: <none>
HResult: 80131530
********************************
Exception object: 02ac11c8
Exception type:   System.Threading.ThreadAbortException
Message:          <none>
InnerException:   <none>
StackTrace (generated):
<none>
StackTraceString: <none>
HResult: 80131530

Finally looking at the dmp file in a hex editor large portions of memory appear to be filled with 0xFF.

ADDITIONAL NOTES:

Found out the following additional info:

1) This app works fine on XP - but will fail with same issue on XP under RDP 2) This app works fine on Win7 32-bit, failed on RDP 3) App fails on Win8 (RDP or NOT)

On the affected Win8 machine ran some simple code to check WPF RenderCapbility. On affected machines (including Win8) Tier = 0 (Software Acceleration) On working machines Tier = 2

DxDiag on affected machine shows nothing for "Video Accel:" Wonder if this might be related.