WinDbg with dump files when you have a classic ASP app which uses a lot of .Net via interop

Ok, first please don't ask why this application is the way it is. This is a classic ASP application which in several areas uses .Net and COM and some of that .Net also reaches into COM! It's all to do with code reuse really, at some point .Net was introduced and used for a whole load of features, which were then required within the classic ASP app and voile! some kind of hellish beast appeared.

So my question? I can see what looks like a serious memory leak in the application (it's 32 bit) where private bytes will suddenly rocket up to 900mb+ and users will start to receive out of memory errors and some odd issues like part of the page being rendered. Initially I simply took some perfmon logs and added some user action tracing code and tied the data from the trace to the perfmon (working set and private bytes on w3wp). This told me a few things, which I did find issues with, "resolved" and release - though they had NO impact. A stress test on the app based on the actions in the IIS logs, also told me nothing - I can't replicate this at all locally.

So I got a memory dump when it hits around 900mb and am analysing with WinDbg. The problem is - I have hardly used this tool in the past, and it was a long time ago when I did. I ran a few things like:

.loadby sos mscorwks
!dumpheap -stat

I get System.String as the biggest user of memory and I dumped out all the strings to the console, but it's fairly meaningless, just a load of strings I might expect to see. I can't see the wood for the trees it seems.

Does anyone have any advice on how I can interpret a dmp file and determine what is REALLY using most of the memory? The fact that System.String "appears" to be using most of it (it's at the bottom), isn't quite true I believe. This is also not strictly a .Net app, so how can I get an idea of whether .Net or classic ASP/COM is using up most of the memory?

UPDATE 1

Thanks for the answers so far from Thomas and Alex. I'm struggling to find the root cause, but have gone a little deeper to the point where I see a common pattern/theme emerging. Please see the following:

I look for a summary of memory usage:

>!heap -s
 Heap     Flags   Reserv  Commit  Virt   Free  List   UCR  Virt  Lock  Fast 
                    (k)     (k)    (k)     (k) length      blocks cont. heap 
-----------------------------------------------------------------------------
002a0000 58001062  695360 677844 680772   2102   100   420    8  1f960   L  

I redacted the results to show one, but the clear majority of memory in the head is used by this address. Then I look at that address and can see most of the objects are of size 2404, so I look to see what these are:

>!heap -stat -h 002a0000
size     #blocks     total     ( %) (percent of total busy bytes)
2404 cfce - 1d3c3738  (71.05)
>!heap -flt s 2404

A VERY large list is returned as expected and by ensuring +ust gflags are on for w3wp before collecting the dump and investigating a number of these i get a fairly descriptive stack trace:

>!heap -p -a 4fc4d7b8
 HEAP_ENTRY Size Prev Flags    UserPtr UserSize - state
    4fc4d7b0 0484 0000  [07]   4fc4d7b8    02404 - (busy)
    Trace: 414f
    7c833ba5 ntdll!RtlInitializeCriticalSection+0x00000010
    77e67877 kernel32!InitializeCriticalSection+0x0000000e
    79e8539b mscorwks!CrstBase::InitWorker+0x000000a7
    79e853d4 mscorwks!Crst::Crst+0x00000016
    79e92b58 mscorwks!PendingTypeLoadEntry::PendingTypeLoadEntry+0x0000001c
    79e92992 mscorwks!ClassLoader::LoadTypeHandleForTypeKey_Inner+0x000000bd
    79e92b01 mscorwks!ClassLoader::LoadTypeHandleForTypeKey_Body+0x000001da
    79e9272a mscorwks!ClassLoader::LoadTypeHandleForTypeKey+0x000000ae
    79e9359f mscorwks!ClassLoader::EnsureLoaded+0x000000a6
    79e932d9 mscorwks!MethodTable::DoFullyLoad+0x000000c1
    79e9302f mscorwks!MethodTable::DoFullyLoad+0x00000180
    79e9333c mscorwks!MethodTable::DoFullyLoad+0x00000142
    79e92dc9 mscorwks!ClassLoader::Notify+0x00000104
    79e92764 mscorwks!ClassLoader::LoadTypeHandleForTypeKey+0x000000e8
    79e934b3 mscorwks!ClassLoader::LoadTypeDefThrowing+0x00000193
    79e94515 mscorwks!ClassLoader::LoadTypeHandleThrowing+0x000001f4
    79e8c1f0 mscorwks!ClassLoader::LoadTypeHandleThrowIfFailed+0x0000001b
    79efd389 mscorwks!ClassLoader::LoadTypeByNameThrowing+0x0000003b
    79efd315 mscorwks!Binder::LookupClass+0x00000032
    79efd3ab mscorwks!Binder::FetchClass+0x0000001f
    79f1ec2b mscorwks!RefSecContext::Init+0x000000ba
    79ef3726 mscorwks!RuntimeTypeHandle::CreateInstance+0x00000334
    7d0663a +0x07d0663a
    7d06567 +0x07d06567
    7d063b5 +0x07d063b5
    7d05ec3 +0x07d05ec3
    7d05e60 +0x07d05e60
    7d05e22 +0x07d05e22
    7d05dda +0x07d05dda
    7d05cbe +0x07d05cbe
    7d05c44 +0x07d05c44
    7d0582d +0x07d0582d

or

> !heap -p -a 4fca7ff0
address 4fca7ff0 found in
_HEAP @ 2a0000
  HEAP_ENTRY Size Prev Flags    UserPtr UserSize - state
    4fca7fe8 0484 0000  [07]   4fca7ff0    02404 - (busy)
    Trace: 414b
    7c854f44 ntdll!RtlAllocateHeapSlowly+0x00000041
    7c83d7f0 ntdll!RtlAllocateHeap+0x00000e9f
    79e747ac mscorwks!EEHeapAlloc+0x00000142
    79e7482a mscorwks!EEHeapAllocInProcessHeap+0x00000052
    79e74853 mscorwks!operator new[]+0x00000025
    79e927d9 mscorwks!PendingTypeLoadTable::AllocNewEntry+0x0000000c
    79e929ae mscorwks!ClassLoader::LoadTypeHandleForTypeKey_Inner+0x000000d9
    79e92b01 mscorwks!ClassLoader::LoadTypeHandleForTypeKey_Body+0x000001da
    79e9272a mscorwks!ClassLoader::LoadTypeHandleForTypeKey+0x000000ae
    79e9359f mscorwks!ClassLoader::EnsureLoaded+0x000000a6
    79e92faa mscorwks!ClassLoader::LoadExactParentAndInterfacesTransitively+0x000000c2
    79e92fe7 mscorwks!ClassLoader::LoadExactParents+0x00000027
    79e92f24 mscorwks!ClassLoader::DoIncrementalLoad+0x00000057
    79e929e1 mscorwks!ClassLoader::LoadTypeHandleForTypeKey_Inner+0x0000012c
    79e92b01 mscorwks!ClassLoader::LoadTypeHandleForTypeKey_Body+0x000001da
    79e9272a mscorwks!ClassLoader::LoadTypeHandleForTypeKey+0x000000ae
    79e934b3 mscorwks!ClassLoader::LoadTypeDefThrowing+0x00000193
    79e94515 mscorwks!ClassLoader::LoadTypeHandleThrowing+0x000001f4
    79e8c1f0 mscorwks!ClassLoader::LoadTypeHandleThrowIfFailed+0x0000001b
    79efd389 mscorwks!ClassLoader::LoadTypeByNameThrowing+0x0000003b
    79efd315 mscorwks!Binder::LookupClass+0x00000032
    79efd3ab mscorwks!Binder::FetchClass+0x0000001f
    79f1ec2b mscorwks!RefSecContext::Init+0x000000ba
    79ef3726 mscorwks!RuntimeTypeHandle::CreateInstance+0x00000334
    7d0663a +0x07d0663a
    7d06567 +0x07d06567
    7d063b5 +0x07d063b5
    7d05ec3 +0x07d05ec3
    7d05e60 +0x07d05e60
    7d05e22 +0x07d05e22
    7d05dda +0x07d05dda
    7d05cbe +0x07d05cbe

What I would love to be able to do at this point is see exactly what classes are being instantiated to give me a clue which user action (always from a classic ASP page) is calling what (e.g. cASP calls COM or .Net which in turn could call COM or .Net).

You need to find the roots that holds these strings in memory. I have a few examples in my article: http://alexatnet.com/articles/net-memory-management-and-garbage-collector but generally what you might need to do is to use !gcroot command - it should traverse object graph to one of the roots that holds this object.