Windbg - Tracking native cryptography leak in C++/VB6/C# application


I'm currently attempting to track down a leak in a C++ service that loads VB6 modules that load .NET libraries using windbg, and as of now, feel like I'm so close to the answer, but can't seem to lock it in.

Starting with !heap -s, I identify the heap in question:

0:000> !heap -s
LFH Key                   : 0x48d335eb
Termination on corruption : ENABLED
  Heap     Flags   Reserv  Commit  Virt   Free  List   UCR  Virt  Lock  Fast 
                    (k)     (k)    (k)     (k) length      blocks cont. heap 
Virtual block: 03d20000 - 03d20000 (size 00000000)
007a0000 00000002 1457472 1447856 1457472   7003  1888    94    1      e   LFH

With that identified, I identify what is clogging it up:

0:000> !heap -stat -h 007a0000
 heap @ 007a0000
group-by: TOTSIZE max-display: 20
    size     #blocks     total     ( %) (percent of total busy bytes)
    298 cb166 - 20ec2090  (76.28)

With gflags enabled, I identify and sample how the allocations are being called:

0:000> !heap -flt s 298
    604473b8 0055 0055  [00]   604473c0    00298 - (busy)

0:000> !heap -p -a  604473c0 
    address 604473c0 found in
    _HEAP @ 007a0000
      HEAP_ENTRY Size Prev Flags    UserPtr UserSize - state
        604473b8 0059 0000  [00]   604473c0    00298 - (busy)
        772a34e5 ntdll!RtlAllocateHeap+0x0000021d
        74c54568 rsaenh!ContAlloc+0x00000017
        74c7aa8f rsaenh!CopyKey+0x00000030
        74c7abd4 rsaenh!CPDuplicateKey+0x0000008b
        75584c4b advapi32!CryptDuplicateKey+0x0000007c

Once I knew the allocations were coming from a call to CryptDuplicateKey, I then hunted down where this call is being made from. We never directly call this in our code, so I used WinDBG again to identify how this is being called.

0:198> bp advapi32!CryptDuplicateKey+0x0000007c
0:198> g

Breakpoint 0 hit
eax=212bc380 ebx=22222222 ecx=26d0ddc8 edx=74c7ab49 esi=2f90bc10 edi=00000001
eip=75584c48 esp=26d0dd9c ebp=26d0dde8 iopl=0         nv up ei pl nz na po nc
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000202
75584c48 ff5624          call    dword ptr [esi+24h]  ds:002b:2f90bc34={rsaenh!CPDuplicateKey (74c7ab49)}

0:034> !clrstack
OS Thread Id: 0x12f24 (34)
ESP       EIP     
26d0de40 75584c48 [NDirectMethodFrameStandaloneCleanup: 26d0de40] System.Security.Cryptography.CapiNative+UnsafeNativeMethods.CryptDuplicateKey(Microsoft.Win32.SafeHandles.SafeCapiKeyHandle, IntPtr, Int32, Microsoft.Win32.SafeHandles.SafeCapiKeyHandle ByRef)
26d0de58 742aa535 Microsoft.Win32.SafeHandles.SafeCapiKeyHandle.Duplicate()
26d0de68 742ac382 System.Security.Cryptography.CapiSymmetricAlgorithm..ctor(Int32, Int32, Microsoft.Win32.SafeHandles.SafeCspHandle, Microsoft.Win32.SafeHandles.SafeCapiKeyHandle, Byte[], System.Security.Cryptography.CipherMode, System.Security.Cryptography.PaddingMode, System.Security.Cryptography.EncryptionMode)
26d0de98 742ab5b2 System.Security.Cryptography.AesCryptoServiceProvider.CreateDecryptor(Microsoft.Win32.SafeHandles.SafeCapiKeyHandle, Byte[])
26d0deb4 742ab521 System.Security.Cryptography.AesCryptoServiceProvider.CreateDecryptor(Byte[], Byte[])
26d0def0 1097028b CryptoLib.CryptoObject.DecryptMessage(System.String, System.String ByRef)

With CryptoLib being our code, I was now armed with where the leak is coming from. Obviously, CreateDecryptor is making the call to DuplicateKey which is leaking allocations of size 298, so my thought process was that we were probably not calling dispose properly. However, when I examine the implementation for this area in CryptoLib, I find:

using (var aes = new AesCryptoServiceProvider()) {
    aes.Key = vbKey
    aes.IV = vbIV
    byte[] decryptBuffer;
    using (var mstream = new MemoryStream()) {
        using (var decryptor = aes.CreateDecryptor(aes.Key, aes.IV)) {
            using (var cstream = new CryptoStream(mstream, decryptor, CryptoStreamMode.Write) {                            {
                cstream.Write(messagebuffer, 0, messagebuffer.Length);
        decryptBuffer = mstream.ToArray();
    decryptedMessage = Encoding.GetEncoding(1252).GetString(decryptBuffer);

Looking into CreateDecryptor, it appears that this should all be disposing properly, and yet, the application is still collecting 1.1 GB of garbage from the calls to CreateDecryptor.

Looking at the documentation for this, the implementation seems correct, and I cannot seemingly identify a reason for these native objects to be leaking.

The library otherwise behaves, I just can't figure out why these objects aren't being disposed.

asked on Stack Overflow Feb 9, 2016 by Fidsah • edited Feb 13, 2016 by Fidsah

0 Answers

Nobody has answered this question yet.

User contributions licensed under CC BY-SA 3.0