How can I find the reason of HANG with WinDBG? (C++)

2

I got the full dump using this command when the process doesn't respond and doesn't work.

procdump.exe -ma [PID]

Also, I tried to find the reason why this process showed strange behavior using WinDBG. But I'm just a WinDBG beginner, so I didn't get any hint from the FULL dump file. (Actually I can't understand a WinDBG message) Is there someone who can help me with this problem?

This is !locks command result.

0:000> !locks

CritSec +218daf0 at 0218daf0
WaiterWoken        No
LockCount          14
RecursionCount     1
OwningThread       1554
EntryCount         0
ContentionCount    e
*** Locked

Scanned 789 critical sections

This result says "NO LOCKS", right?

This is ~*k (all thread call stack) result.

0:000> ~*k

.  0  Id: 1ff0.1d74 Suspend: 0 Teb: 7efdd000 Unfrozen
ChildEBP RetAddr  
0015fa08 75bd149d ntdll!ZwWaitForSingleObject+0x15
0015fa74 75f21194 KERNELBASE!WaitForSingleObjectEx+0x98
0015fa8c 75f21148 kernel32!WaitForSingleObjectExImplementation+0x75
0015faa0 100500fe kernel32!WaitForSingleObject+0x12
WARNING: Stack unwind information not available. Following frames may be wrong.
0015fac4 10050edb ACE_5_8_3!ACE_OS::cond_wait+0x3e
00000000 00000000 ACE_5_8_3!ACE_OS::cond_timedwait+0x1b

   1  Id: 1ff0.200c Suspend: 0 Teb: 7efda000 Unfrozen
ChildEBP RetAddr  
0449fdf4 77e02f91 ntdll!NtWaitForMultipleObjects+0x15
0449ff88 75f2336a ntdll!TppWaiterpThread+0x33d
0449ff94 77de9f72 kernel32!BaseThreadInitThunk+0xe
0449ffd4 77de9f45 ntdll!__RtlUserThreadStart+0x70
0449ffec 00000000 ntdll!_RtlUserThreadStart+0x1b

   2  Id: 1ff0.1cbc Suspend: 0 Teb: 7efd7000 Unfrozen
ChildEBP RetAddr  
0477ff5c 750e635c ntdll!ZwRemoveIoCompletion+0x15
0477ff88 75f2336a mswsock!SockAsyncThread+0x83
0477ff94 77de9f72 kernel32!BaseThreadInitThunk+0xe
0477ffd4 77de9f45 ntdll!__RtlUserThreadStart+0x70
0477ffec 00000000 ntdll!_RtlUserThreadStart+0x1b

   3  Id: 1ff0.1c78 Suspend: 0 Teb: 7ef9c000 Unfrozen
ChildEBP RetAddr  
0497fd78 75bd149d ntdll!ZwWaitForSingleObject+0x15
0497fde4 75f21194 KERNELBASE!WaitForSingleObjectEx+0x98
0497fdfc 75f21148 kernel32!WaitForSingleObjectExImplementation+0x75
0497fe10 100500fe kernel32!WaitForSingleObject+0x12
WARNING: Stack unwind information not available. Following frames may be wrong.
0497fe34 10050edb ACE_5_8_3!ACE_OS::cond_wait+0x3e
00000000 00000000 ACE_5_8_3!ACE_OS::cond_timedwait+0x1b

   4  Id: 1ff0.11ac Suspend: 0 Teb: 7ef99000 Unfrozen
ChildEBP RetAddr  
04a7fcf8 75bd149d ntdll!ZwWaitForSingleObject+0x15
04a7fd64 75f21194 KERNELBASE!WaitForSingleObjectEx+0x98
04a7fd7c 75f21148 kernel32!WaitForSingleObjectExImplementation+0x75
04a7fd90 10050f67 kernel32!WaitForSingleObject+0x12
WARNING: Stack unwind information not available. Following frames may be wrong.
04a7fde0 1001cf14 ACE_5_8_3!ACE_OS::cond_timedwait+0xa7
04a7fdfc 0041c4ac ACE_5_8_3!ACE_Condition<ACE_Recursive_Thread_Mutex>::wait+0x44
04a7fee0 10074b5c FatDatabaseCenter!ACE_Thread_Timer_Queue_Adapter<ACE_Timer_Heap_T<ACE_Event_Handler *,ACE_Event_Handler_Handle_Timeout_Upcall<ACE_Recursive_Thread_Mutex>,ACE_Recursive_Thread_Mutex>,ACE_Event_Handler *>::svc+0x16c [d:\00.buildserver\jenkins\workspace\c++_server_Fatdatabasecenter_v30\extlib\ace_5_8_3\ace\timer_queue_adapters.cpp @ 275]
04a7ff28 10075097 ACE_5_8_3!ACE_Task_Base::svc_run+0x2c
04a7ff80 73832a47 ACE_5_8_3!ACE_Thread_Adapter::invoke+0x87
04a7ff88 75f2336a msvcr80!_threadstartex+0x66 [f:\dd\vctools\crt_bld\self_x86\crt\src\threadex.c @ 326]
04a7ff94 77de9f72 kernel32!BaseThreadInitThunk+0xe
04a7ffd4 77de9f45 ntdll!__RtlUserThreadStart+0x70
04a7ffec 00000000 ntdll!_RtlUserThreadStart+0x1b

   5  Id: 1ff0.2708 Suspend: 0 Teb: 7ef96000 Unfrozen
ChildEBP RetAddr  
04bbab2c 77de8e44 ntdll!ZwWaitForSingleObject+0x15
04bbab90 77de8d28 ntdll!RtlpWaitOnCriticalSection+0x13e
04bbabb8 004159ea ntdll!RtlEnterCriticalSection+0x150
04bbabc0 004151a4 FatDatabaseCenter!FatLog::FatLogSyncObj_CriticalSection::Enter+0xa [d:\...\syncobj\Fatlogsyncobj_criticalsection.cpp @ 29]
04bbabd8 00414fcc FatDatabaseCenter!FatLog::FatLogStore::write+0x14 [d:\00.buildserver\jenkins\workspace\c++_server_Fatdatabasecenter_v30\src\Fatlib\Fatlog\Fatlogstore.cpp @ 101]
04bbebec 004162bf FatDatabaseCenter!FatLog::WriteLog+0x6c [d:\...\src\Fatlib\Fatlog\Fatlog.cpp @ 156]
04bbebfc 10033287 FatDatabaseCenter!FatLog::FatLog_AceLogLinker::log+0x6f [d:\...\src\Fatlib\Fatlog\plugin\ace\Fatlog_aceloglinker.cpp @ 47]
WARNING: Stack unwind information not available. Following frames may be wrong.
00000000 00000000 ACE_5_8_3!ACE_Log_Msg::log+0x67

   6  Id: 1ff0.1b24 Suspend: 0 Teb: 7ef93000 Unfrozen
ChildEBP RetAddr  
04cfab2c 77de8e44 ntdll!ZwWaitForSingleObject+0x15
04cfab90 77de8d28 ntdll!RtlpWaitOnCriticalSection+0x13e
04cfabb8 004159ea ntdll!RtlEnterCriticalSection+0x150
04cfabc0 004151a4 FatDatabaseCenter!FatLog::FatLogSyncObj_CriticalSection::Enter+0xa [d:\...\syncobj\Fatlogsyncobj_criticalsection.cpp @ 29]
04cfabd8 00414fcc FatDatabaseCenter!FatLog::FatLogStore::write+0x14 [d:\...\src\Fatlib\Fatlog\Fatlogstore.cpp @ 101]
04cfebec 004162bf FatDatabaseCenter!FatLog::WriteLog+0x6c [d:\...\src\Fatlib\Fatlog\Fatlog.cpp @ 156]
04cfebfc 10033287 FatDatabaseCenter!FatLog::FatLog_AceLogLinker::log+0x6f [d:\...\src\Fatlib\Fatlog\plugin\ace\Fatlog_aceloglinker.cpp @ 47]
WARNING: Stack unwind information not available. Following frames may be wrong.
00000000 00000000 ACE_5_8_3!ACE_Log_Msg::log+0x67

   7  Id: 1ff0.217c Suspend: 0 Teb: 7ef90000 Unfrozen
ChildEBP RetAddr  
04e3ab2c 77de8e44 ntdll!ZwWaitForSingleObject+0x15
04e3ab90 77de8d28 ntdll!RtlpWaitOnCriticalSection+0x13e
04e3abb8 004159ea ntdll!RtlEnterCriticalSection+0x150
04e3abc0 004151a4 FatDatabaseCenter!FatLog::FatLogSyncObj_CriticalSection::Enter+0xa [d:\...\syncobj\Fatlogsyncobj_criticalsection.cpp @ 29]
04e3abd8 00414fcc FatDatabaseCenter!FatLog::FatLogStore::write+0x14 [d:\...\src\Fatlib\Fatlog\Fatlogstore.cpp @ 101]
04e3ebec 004162bf FatDatabaseCenter!FatLog::WriteLog+0x6c [d:\...\src\Fatlib\Fatlog\Fatlog.cpp @ 156]
04e3ebfc 10033287 FatDatabaseCenter!FatLog::FatLog_AceLogLinker::log+0x6f [d:\...\src\Fatlib\Fatlog\plugin\ace\Fatlog_aceloglinker.cpp @ 47]
WARNING: Stack unwind information not available. Following frames may be wrong.
00000000 00000000 ACE_5_8_3!ACE_Log_Msg::log+0x67

   8  Id: 1ff0.2508 Suspend: 0 Teb: 7ef8d000 Unfrozen
ChildEBP RetAddr  
04f7ab2c 77de8e44 ntdll!ZwWaitForSingleObject+0x15
04f7ab90 77de8d28 ntdll!RtlpWaitOnCriticalSection+0x13e
04f7abb8 004159ea ntdll!RtlEnterCriticalSection+0x150
04f7abc0 004151a4 FatDatabaseCenter!FatLog::FatLogSyncObj_CriticalSection::Enter+0xa [d:\...\syncobj\Fatlogsyncobj_criticalsection.cpp @ 29]
04f7abd8 00414fcc FatDatabaseCenter!FatLog::FatLogStore::write+0x14 [d:\...\src\Fatlib\Fatlog\Fatlogstore.cpp @ 101]
04f7ebec 004162bf FatDatabaseCenter!FatLog::WriteLog+0x6c [d:\...\src\Fatlib\Fatlog\Fatlog.cpp @ 156]
04f7ebfc 10033287 FatDatabaseCenter!FatLog::FatLog_AceLogLinker::log+0x6f [d:\...\src\Fatlib\Fatlog\plugin\ace\Fatlog_aceloglinker.cpp @ 47]
WARNING: Stack unwind information not available. Following frames may be wrong.
00000000 00000000 ACE_5_8_3!ACE_Log_Msg::log+0x67

   9  Id: 1ff0.2780 Suspend: 0 Teb: 7ef8a000 Unfrozen
ChildEBP RetAddr  
050bab2c 77de8e44 ntdll!ZwWaitForSingleObject+0x15
050bab90 77de8d28 ntdll!RtlpWaitOnCriticalSection+0x13e
050babb8 004159ea ntdll!RtlEnterCriticalSection+0x150
050babc0 004151a4 FatDatabaseCenter!FatLog::FatLogSyncObj_CriticalSection::Enter+0xa [d:\...\syncobj\Fatlogsyncobj_criticalsection.cpp @ 29]
050babd8 00414fcc FatDatabaseCenter!FatLog::FatLogStore::write+0x14 [d:\...\src\Fatlib\Fatlog\Fatlogstore.cpp @ 101]
050bebec 004162bf FatDatabaseCenter!FatLog::WriteLog+0x6c [d:\...\src\Fatlib\Fatlog\Fatlog.cpp @ 156]
050bebfc 10033287 FatDatabaseCenter!FatLog::FatLog_AceLogLinker::log+0x6f [d:\...\src\Fatlib\Fatlog\plugin\ace\Fatlog_aceloglinker.cpp @ 47]
WARNING: Stack unwind information not available. Following frames may be wrong.
00000000 00000000 ACE_5_8_3!ACE_Log_Msg::log+0x67

  10  Id: 1ff0.13d0 Suspend: 0 Teb: 7ef87000 Unfrozen
ChildEBP RetAddr  
051fab2c 77de8e44 ntdll!ZwWaitForSingleObject+0x15
051fab90 77de8d28 ntdll!RtlpWaitOnCriticalSection+0x13e
051fabb8 004159ea ntdll!RtlEnterCriticalSection+0x150
051fabc0 004151a4 FatDatabaseCenter!FatLog::FatLogSyncObj_CriticalSection::Enter+0xa [d:\...\syncobj\Fatlogsyncobj_criticalsection.cpp @ 29]
051fabd8 00414fcc FatDatabaseCenter!FatLog::FatLogStore::write+0x14 [d:\...\src\Fatlib\Fatlog\Fatlogstore.cpp @ 101]
051febec 004162bf FatDatabaseCenter!FatLog::WriteLog+0x6c [d:\...\src\Fatlib\Fatlog\Fatlog.cpp @ 156]
051febfc 10033287 FatDatabaseCenter!FatLog::FatLog_AceLogLinker::log+0x6f [d:\...\src\Fatlib\Fatlog\plugin\ace\Fatlog_aceloglinker.cpp @ 47]
WARNING: Stack unwind information not available. Following frames may be wrong.
00000000 00000000 ACE_5_8_3!ACE_Log_Msg::log+0x67

  11  Id: 1ff0.294 Suspend: 0 Teb: 7ef84000 Unfrozen
ChildEBP RetAddr  
0533ab2c 77de8e44 ntdll!ZwWaitForSingleObject+0x15
0533ab90 77de8d28 ntdll!RtlpWaitOnCriticalSection+0x13e
0533abb8 004159ea ntdll!RtlEnterCriticalSection+0x150
0533abc0 004151a4 FatDatabaseCenter!FatLog::FatLogSyncObj_CriticalSection::Enter+0xa [d:\...\syncobj\Fatlogsyncobj_criticalsection.cpp @ 29]
0533abd8 00414fcc FatDatabaseCenter!FatLog::FatLogStore::write+0x14 [d:\...\src\Fatlib\Fatlog\Fatlogstore.cpp @ 101]
0533ebec 004162bf FatDatabaseCenter!FatLog::WriteLog+0x6c [d:\...\src\Fatlib\Fatlog\Fatlog.cpp @ 156]
0533ebfc 10033287 FatDatabaseCenter!FatLog::FatLog_AceLogLinker::log+0x6f [d:\...\src\Fatlib\Fatlog\plugin\ace\Fatlog_aceloglinker.cpp @ 47]
WARNING: Stack unwind information not available. Following frames may be wrong.
00000000 00000000 ACE_5_8_3!ACE_Log_Msg::log+0x67

  12  Id: 1ff0.270c Suspend: 0 Teb: 7ef81000 Unfrozen
ChildEBP RetAddr  
0547ab2c 77de8e44 ntdll!ZwWaitForSingleObject+0x15
0547ab90 77de8d28 ntdll!RtlpWaitOnCriticalSection+0x13e
0547abb8 004159ea ntdll!RtlEnterCriticalSection+0x150
0547abc0 004151a4 FatDatabaseCenter!FatLog::FatLogSyncObj_CriticalSection::Enter+0xa [d:\...\syncobj\Fatlogsyncobj_criticalsection.cpp @ 29]
0547abd8 00414fcc FatDatabaseCenter!FatLog::FatLogStore::write+0x14 [d:\...\src\Fatlib\Fatlog\Fatlogstore.cpp @ 101]
0547ebec 004162bf FatDatabaseCenter!FatLog::WriteLog+0x6c [d:\...\src\Fatlib\Fatlog\Fatlog.cpp @ 156]
0547ebfc 10033287 FatDatabaseCenter!FatLog::FatLog_AceLogLinker::log+0x6f [d:\...\src\Fatlib\Fatlog\plugin\ace\Fatlog_aceloglinker.cpp @ 47]
WARNING: Stack unwind information not available. Following frames may be wrong.
00000000 00000000 ACE_5_8_3!ACE_Log_Msg::log+0x67

  13  Id: 1ff0.1fe4 Suspend: 0 Teb: 7ef7e000 Unfrozen
ChildEBP RetAddr  
055bab2c 77de8e44 ntdll!ZwWaitForSingleObject+0x15
055bab90 77de8d28 ntdll!RtlpWaitOnCriticalSection+0x13e
055babb8 004159ea ntdll!RtlEnterCriticalSection+0x150
055babc0 004151a4 FatDatabaseCenter!FatLog::FatLogSyncObj_CriticalSection::Enter+0xa [d:\...\syncobj\Fatlogsyncobj_criticalsection.cpp @ 29]
055babd8 00414fcc FatDatabaseCenter!FatLog::FatLogStore::write+0x14 [d:\...\src\Fatlib\Fatlog\Fatlogstore.cpp @ 101]
055bebec 004162bf FatDatabaseCenter!FatLog::WriteLog+0x6c [d:\...\src\Fatlib\Fatlog\Fatlog.cpp @ 156]
055bebfc 10033287 FatDatabaseCenter!FatLog::FatLog_AceLogLinker::log+0x6f [d:\...\src\Fatlib\Fatlog\plugin\ace\Fatlog_aceloglinker.cpp @ 47]
WARNING: Stack unwind information not available. Following frames may be wrong.
00000000 00000000 ACE_5_8_3!ACE_Log_Msg::log+0x67

  14  Id: 1ff0.e6c Suspend: 0 Teb: 7ef7b000 Unfrozen
ChildEBP RetAddr  
056fab2c 77de8e44 ntdll!ZwWaitForSingleObject+0x15
056fab90 77de8d28 ntdll!RtlpWaitOnCriticalSection+0x13e
056fabb8 004159ea ntdll!RtlEnterCriticalSection+0x150
056fabc0 004151a4 FatDatabaseCenter!FatLog::FatLogSyncObj_CriticalSection::Enter+0xa [d:\...\syncobj\Fatlogsyncobj_criticalsection.cpp @ 29]
056fabd8 00414fcc FatDatabaseCenter!FatLog::FatLogStore::write+0x14 [d:\...\src\Fatlib\Fatlog\Fatlogstore.cpp @ 101]
056febec 004162bf FatDatabaseCenter!FatLog::WriteLog+0x6c [d:\...\src\Fatlib\Fatlog\Fatlog.cpp @ 156]
056febfc 10033287 FatDatabaseCenter!FatLog::FatLog_AceLogLinker::log+0x6f [d:\...\src\Fatlib\Fatlog\plugin\ace\Fatlog_aceloglinker.cpp @ 47]
WARNING: Stack unwind information not available. Following frames may be wrong.
00000000 00000000 ACE_5_8_3!ACE_Log_Msg::log+0x67

  15  Id: 1ff0.2744 Suspend: 0 Teb: 7ef78000 Unfrozen
ChildEBP RetAddr  
0583ab2c 77de8e44 ntdll!ZwWaitForSingleObject+0x15
0583ab90 77de8d28 ntdll!RtlpWaitOnCriticalSection+0x13e
0583abb8 004159ea ntdll!RtlEnterCriticalSection+0x150
0583abc0 004151a4 FatDatabaseCenter!FatLog::FatLogSyncObj_CriticalSection::Enter+0xa [d:\...\syncobj\Fatlogsyncobj_criticalsection.cpp @ 29]
0583abd8 00414fcc FatDatabaseCenter!FatLog::FatLogStore::write+0x14 [d:\...\src\Fatlib\Fatlog\Fatlogstore.cpp @ 101]
0583ebec 004162bf FatDatabaseCenter!FatLog::WriteLog+0x6c [d:\...\src\Fatlib\Fatlog\Fatlog.cpp @ 156]
0583ebfc 10033287 FatDatabaseCenter!FatLog::FatLog_AceLogLinker::log+0x6f [d:\...\src\Fatlib\Fatlog\plugin\ace\Fatlog_aceloglinker.cpp @ 47]
WARNING: Stack unwind information not available. Following frames may be wrong.
00000000 00000000 ACE_5_8_3!ACE_Log_Msg::log+0x67

  16  Id: 1ff0.1f78 Suspend: 0 Teb: 7ef75000 Unfrozen
ChildEBP RetAddr  
0597ab2c 77de8e44 ntdll!ZwWaitForSingleObject+0x15
0597ab90 77de8d28 ntdll!RtlpWaitOnCriticalSection+0x13e
0597abb8 004159ea ntdll!RtlEnterCriticalSection+0x150
0597abc0 004151a4 FatDatabaseCenter!FatLog::FatLogSyncObj_CriticalSection::Enter+0xa [d:\...\syncobj\Fatlogsyncobj_criticalsection.cpp @ 29]
0597abd8 00414fcc FatDatabaseCenter!FatLog::FatLogStore::write+0x14 [d:\...\src\Fatlib\Fatlog\Fatlogstore.cpp @ 101]
0597ebec 004162bf FatDatabaseCenter!FatLog::WriteLog+0x6c [d:\...\src\Fatlib\Fatlog\Fatlog.cpp @ 156]
0597ebfc 10033287 FatDatabaseCenter!FatLog::FatLog_AceLogLinker::log+0x6f [d:\...\src\Fatlib\Fatlog\plugin\ace\Fatlog_aceloglinker.cpp @ 47]
WARNING: Stack unwind information not available. Following frames may be wrong.
00000000 00000000 ACE_5_8_3!ACE_Log_Msg::log+0x67

  17  Id: 1ff0.48c Suspend: 0 Teb: 7ef72000 Unfrozen
ChildEBP RetAddr  
05abab2c 77de8e44 ntdll!ZwWaitForSingleObject+0x15
05abab90 77de8d28 ntdll!RtlpWaitOnCriticalSection+0x13e
05ababb8 004159ea ntdll!RtlEnterCriticalSection+0x150
05ababc0 004151a4 FatDatabaseCenter!FatLog::FatLogSyncObj_CriticalSection::Enter+0xa [d:\...\syncobj\Fatlogsyncobj_criticalsection.cpp @ 29]
05ababd8 00414fcc FatDatabaseCenter!FatLog::FatLogStore::write+0x14 [d:\...\src\Fatlib\Fatlog\Fatlogstore.cpp @ 101]
05abebec 004162bf FatDatabaseCenter!FatLog::WriteLog+0x6c [d:\...\src\Fatlib\Fatlog\Fatlog.cpp @ 156]
05abebfc 10033287 FatDatabaseCenter!FatLog::FatLog_AceLogLinker::log+0x6f [d:\...\src\Fatlib\Fatlog\plugin\ace\Fatlog_aceloglinker.cpp @ 47]
WARNING: Stack unwind information not available. Following frames may be wrong.
00000000 00000000 ACE_5_8_3!ACE_Log_Msg::log+0x67

  18  Id: 1ff0.225c Suspend: 0 Teb: 7ef6f000 Unfrozen
ChildEBP RetAddr  
05bfab2c 77de8e44 ntdll!ZwWaitForSingleObject+0x15
05bfab90 77de8d28 ntdll!RtlpWaitOnCriticalSection+0x13e
05bfabb8 004159ea ntdll!RtlEnterCriticalSection+0x150
05bfabc0 004151a4 FatDatabaseCenter!FatLog::FatLogSyncObj_CriticalSection::Enter+0xa [d:\...\syncobj\Fatlogsyncobj_criticalsection.cpp @ 29]
05bfabd8 00414fcc FatDatabaseCenter!FatLog::FatLogStore::write+0x14 [d:\...\src\Fatlib\Fatlog\Fatlogstore.cpp @ 101]
05bfebec 004162bf FatDatabaseCenter!FatLog::WriteLog+0x6c [d:\...\src\Fatlib\Fatlog\Fatlog.cpp @ 156]
05bfebfc 10033287 FatDatabaseCenter!FatLog::FatLog_AceLogLinker::log+0x6f [d:\...\src\Fatlib\Fatlog\plugin\ace\Fatlog_aceloglinker.cpp @ 47]
WARNING: Stack unwind information not available. Following frames may be wrong.
00000000 00000000 ACE_5_8_3!ACE_Log_Msg::log+0x67

  19  Id: 1ff0.1554 Suspend: 0 Teb: 7ef6c000 Unfrozen
ChildEBP RetAddr  
05d3b96c 750e6f1f ntdll!ZwWaitForSingleObject+0x15
05d3b9ac 750ee425 mswsock!SockWaitForSingleObject+0x1ba
05d3ba28 76f06f79 mswsock!WSPSend+0x218
05d3ba70 100067a9 ws2_32!send+0x86
WARNING: Stack unwind information not available. Following frames may be wrong.
05d3ba94 1006eb4f ACE_5_8_3!ACE::send_n_i+0x39
05d3baa8 0041690f ACE_5_8_3!ACE_SOCK_Stream::send_n+0x1f
05d3bac4 004151ff FatDatabaseCenter!FatLog::FatLog_NetworkMonitor_Handler_ACE::WriteLog+0x3f [d:\...\src\Fatlib\Fatlog\plugin\ace\Fatlog_networkmonitor_handler_ace.cpp @ 56]
05d3bae4 00414fcc FatDatabaseCenter!FatLog::FatLogStore::write+0x6f [d:\...\src\Fatlib\Fatlog\Fatlogstore.cpp @ 101]
05d3faf8 0042f20f FatDatabaseCen## Heading ##ter!FatLog::WriteLog+0x6c [d:\...\src\Fatlib\Fatlog\Fatlog.cpp @ 156]
05d3fb6c 0042b4ff FatDatabaseCenter!FatSqlSelect::Select+0xef [d:\...\src\Fatdatabasecenter30\service\oracleservice\ver2\sql\select\Fatsqlselect.cpp @ 43]
05d3fc3c 0042d1c9 FatDatabaseCenter!FatOracleService_V2::Select+0x19f [d:\...\src\Fatdatabasecenter30\service\oracleservice\ver2\Fatoracleservice_v2.cpp @ 76]
05d3fc44 0042fd72 FatDatabaseCenter!FatOracleService_V2::ClientMsg+0x39 [d:\...\src\Fatdatabasecenter30\service\oracleservice\ver2\Fatoracleservice_v2.cpp @ 58]
05d3fc58 00430771 FatDatabaseCenter!IFDCClientStream::process+0x32 [d:\...\src\Fatdatabasecenter30\readhandler\iFDCclientstream.cpp @ 62]
05d3fd1c 0043059a FatDatabaseCenter!FDCClientStream_RIJINDAEL::recv_from_Client+0x341 [d:\...\src\Fatdatabasecenter30\readhandler\FDCclientstream_rijindael.cpp @ 136]
05d3fde0 0042fe3c FatDatabaseCenter!FDCClientStream_RIJINDAEL::recv_from_Client+0x16a [d:\...\src\Fatdatabasecenter30\readhandler\FDCclientstream_rijindael.cpp @ 67]
05d3fe04 1007a797 FatDatabaseCenter!FatReadHandler::handle_input+0xac [d:\...\src\Fatdatabasecenter30\readhandler\Fatreadhandler.cpp @ 83]
05d3fe1c 1007a86e ACE_5_8_3!ACE_TP_Reactor::dispatch_socket_event+0x27
00000000 00000000 ACE_5_8_3!ACE_TP_Reactor::handle_socket_events+0xbe

  20  Id: 1ff0.21e4 Suspend: 0 Teb: 7ef69000 Unfrozen
ChildEBP RetAddr  
0627fee0 7362a41c ntdll!NtWaitForMultipleObjects+0x15
0627ff88 75f2336a winmm!timeThread+0x3c
0627ff94 77de9f72 kernel32!BaseThreadInitThunk+0xe
0627ffd4 77de9f45 ntdll!__RtlUserThreadStart+0x70
0627ffec 00000000 ntdll!_RtlUserThreadStart+0x1b

  21  Id: 1ff0.2704 Suspend: 0 Teb: 7ef66000 Unfrozen
ChildEBP RetAddr  
065ffe28 77e03392 ntdll!NtWaitForWorkViaWorkerFactory+0x12
065fff88 75f2336a ntdll!TppWorkerThread+0x216
065fff94 77de9f72 kernel32!BaseThreadInitThunk+0xe
065fffd4 77de9f45 ntdll!__RtlUserThreadStart+0x70
065fffec 00000000 ntdll!_RtlUserThreadStart+0x1b

And, Here is the !analyze -v command result.

0:000> !analyze -v
*******************************************************************************
*                                                                             *
*                        Exception Analysis                                   *
*                                                                             *
*******************************************************************************


FAULTING_IP: 
+38
00000000 ??              ???

EXCEPTION_RECORD:  ffffffff -- (.exr 0xffffffffffffffff)
ExceptioFatdress: 00000000
   ExceptionCode: 80000003 (Break instruction exception)
  ExceptionFlags: 00000000
NumberParameters: 0

CONTEXT:  00000000 -- (.cxr 0x0;r)
eax=00000000 ebx=00000000 ecx=00000000 edx=00000000 esi=00000398 edi=00000000
eip=77dcf8d1 esp=0015fa08 ebp=0015fa74 iopl=0         nv up ei pl zr na pe nc
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000246
ntdll!ZwWaitForSingleObject+0x15:
77dcf8d1 83c404          add     esp,4

FAULTING_THREAD:  00001d74
DEFAULT_BUCKET_ID:  STATUS_BREAKPOINT
PROCESS_NAME:  FatDatabaseCenter.exe
ERROR_CODE: (NTSTATUS) 0x80000003 - {
EXCEPTION_CODE: (HRESULT) 0x80000003 (2147483651) -                                 .
NTGLOBALFLAG:  0
APPLICATION_VERIFIER_FLAGS:  0
APP:  Fatdatabasecenter.exe
ANALYSIS_VERSION: 6.3.9600.17298 (debuggers(dbg).141024-1500) x86fre
PRIMARY_PROBLEM_CLASS:  STATUS_BREAKPOINT
BUGCHECK_STR:  APPLICATION_FAULT_STATUS_BREAKPOINT
LAST_CONTROL_TRANSFER:  from 75bd149d to 77dcf8d1
STACK_TEXT:  
0015fa08 75bd149d 00000398 00000000 00000000 ntdll!ZwWaitForSingleObject+0x15
0015fa74 75f21194 00000398 ffffffff 00000000 KERNELBASE!WaitForSingleObjectEx+0x98
0015fa8c 75f21148 00000398 ffffffff 00000000 kernel32!WaitForSingleObjectExImplementation+0x75
0015faa0 100500fe 00000398 ffffffff 042d5718 kernel32!WaitForSingleObject+0x12
WARNING: Stack unwind information not available. Following frames may be wrong.
0015fac4 10050edb 042d5718 042d56fc 00000000 ACE_5_8_3!ACE_OS::cond_wait+0x3e
00000000 00000000 00000000 00000000 00000000 ACE_5_8_3!ACE_OS::cond_timedwait+0x1b

STACK_COMMAND:  ~0s; .ecxr ; kb
FOLLOWUP_IP: 
ACE_5_8_3!ACE_OS::cond_wait+3e
100500fe 57              push    edi
SYMBOL_STACK_INDEX:  4
SYMBOL_NAME:  ace_5_8_3!ACE_OS::cond_wait+3e
FOLLOWUP_NAME:  MachineOwner
MODULE_NAME: ACE_5_8_3
IMAGE_NAME:  ACE_5.8.3.dll
DEBUG_FLR_IMAGE_TIMESTAMP:  4f5eea5c
FAILURE_BUCKET_ID:  STATUS_BREAKPOINT_80000003_ACE_5.8.3.dll!ACE_OS::cond_wait
BUCKET_ID:  APPLICATION_FAULT_STATUS_BREAKPOINT_ace_5_8_3!ACE_OS::cond_wait+3e
ANALYSIS_SOURCE:  UM
FAILURE_ID_HASH_STRING:  um:status_breakpoint_80000003_ace_5.8.3.dll!ace_os::cond_wait
FAILURE_ID_HASH:  {3b6cdfb5-4df8-55cb-7886-936eef582ae1}
Followup: MachineOwner
---------

Say everything if you want more information. Please give me some hint why this process fall in HANG !!!

Edit

Here is the !analyze -v -hang command result.

0:000> !analyze -v -hang
*******************************************************************************
*                                                                             *
*                        Exception Analysis                                   *
*                                                                             *
*******************************************************************************


FAULTING_IP: 
+0
00000000 ??              ???

EXCEPTION_RECORD:  ffffffff -- (.exr 0xffffffffffffffff)
ExceptioFADdress: 00000000
   ExceptionCode: 80000003 (Break instruction exception)
  ExceptionFlags: 00000000
NumberParameters: 0

CONTEXT:  00000000 -- (.cxr 0x0;r)
eax=00000000 ebx=00000000 ecx=00000000 edx=00000000 esi=00000398 edi=00000000
eip=77dcf8d1 esp=0015fa08 ebp=0015fa74 iopl=0         nv up ei pl zr na pe nc
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000246
ntdll!ZwWaitForSingleObject+0x15:
77dcf8d1 83c404          add     esp,4

FAULTING_THREAD:  00000013

BUGCHECK_STR:  HANG

PROCESS_NAME:  FADDatabaseCenter.exe

ERROR_CODE: (NTSTATUS) 0xcfffffff - <Unable to get error code text>

EXCEPTION_CODE: (NTSTATUS) 0xcfffffff - <Unable to get error code text>

NTGLOBALFLAG:  0

APPLICATION_VERIFIER_FLAGS:  0

APP:  FADdatabasecenter.exe

ANALYSIS_VERSION: 6.3.9600.17298 (debuggers(dbg).141024-1500) x86fre

DERIVED_WAIT_CHAIN:  

Dl Eid Cid     WaitType
-- --- ------- --------------------------
   0   1ff0.1d74 Speculated (Triage)    -->
   5   1ff0.2708 Event                  -->
   19  1ff0.1554 Event                  

WAIT_CHAIN_COMMAND:  ~0s;k;;~5s;k;;~19s;k;;

BLOCKING_THREAD:  00001554

DEFAULT_BUCKET_ID:  APPLICATION_HANG_BlockedOn_NetworkCall

PRIMARY_PROBLEM_CLASS:  APPLICATION_HANG_BlockedOn_NetworkCall

LAST_CONTROL_TRANSFER:  from 750e6f1f to 77dcf8d1

STACK_TEXT:  
05d3b96c 750e6f1f 000004c8 00000001 05d3b994 ntdll!ZwWaitForSingleObject+0x15
05d3b9ac 750ee425 000004c8 000004d4 00000002 mswsock!SockWaitForSingleObject+0x1ba
05d3ba28 76f06f79 000004d4 05d3ba60 00000001 mswsock!WSPSend+0x218
05d3ba70 100067a9 000004d4 0043d634 00000002 ws2_32!send+0x86
WARNING: Stack unwind information not available. Following frames may be wrong.
05d3ba94 1006eb4f 000004d4 0043d634 00000002 ACE_5_8_3!ACE::send_n_i+0x39
05d3baa8 0041690f 0043d634 00000002 00000000 ACE_5_8_3!ACE_SOCK_Stream::send_n+0x1f
05d3bac4 004151ff 00000020 05d3baf4 04282878 FADDatabaseCenter!FADLog::FADLog_NetworkMonitor_Handler_ACE::WriteLog+0x3f
05d3bae4 00414fcc 00000020 05d3baf4 20202020 FADDatabaseCenter!FADLog::FADLogStore::write+0x6f
05d3faf8 0042f20f 00000020 0043f154 06613050 FADDatabaseCenter!FADLog::WriteLog+0x6c
05d3fb6c 0042b4ff 0439fc40 0438ba68 05d3fbc4 FADDatabaseCenter!FADSqlSelect::Select+0xef
05d3fc3c 0042d1c9 0438ba68 0042fd72 0000000b FADDatabaseCenter!FADOracleService_V2::Select+0x19f
05d3fc44 0042fd72 0000000b 0438ba68 0438ba68 FADDatabaseCenter!FADOracleService_V2::ClientMsg+0x39
05d3fc58 00430771 0438ba68 0000000b 000011f6 FADDatabaseCenter!IFDCClientStream::process+0x32
05d3fd1c 0043059a 0000000b 000011f6 0438ba68 FADDatabaseCenter!FDCClientStream_RIJINDAEL::recv_from_Client+0x341
05d3fde0 0042fe3c 0000f004 00001205 04382e7c FADDatabaseCenter!FDCClientStream_RIJINDAEL::recv_from_Client+0x16a
05d3fe04 1007a797 00000600 05d3fe88 0015fca8 FADDatabaseCenter!FADReadHandler::handle_input+0xac
05d3fe1c 1007a86e 05d3fe34 00000001 0015fca8 ACE_5_8_3!ACE_TP_Reactor::dispatch_socket_event+0x27
00000000 00000000 00000000 00000000 00000000 ACE_5_8_3!ACE_TP_Reactor::handle_socket_events+0xbe

FOLLOWUP_IP: 
ACE_5_8_3!ACE::send_n_i+39
100067a9 83f8ff          cmp     eax,0FFFFFFFFh
SYMBOL_STACK_INDEX:  4
SYMBOL_NAME:  ace_5_8_3!ACE::send_n_i+39
FOLLOWUP_NAME:  MachineOwner
MODULE_NAME: ACE_5_8_3
IMAGE_NAME:  ACE_5.8.3.dll
DEBUG_FLR_IMAGE_TIMESTAMP:  4f5eea5c
STACK_COMMAND:  ~19s ; kb
BUCKET_ID:  HANG_ace_5_8_3!ACE::send_n_i+39
FAILURE_BUCKET_ID:  APPLICATION_HANG_BlockedOn_NetworkCall_cfffffff_ACE_5.8.3.dll!ACE::send_n_i
ANALYSIS_SOURCE:  UM
FAILURE_ID_HASH_STRING:  um:application_hang_blockedon_networkcall_cfffffff_ace_5.8.3.dll!ace::send_n_i
FAILURE_ID_HASH:  {aea64438-3ae6-3ab0-78e9-92b7ba34a6f3}
Followup: MachineOwner

And Here is !runaway result.

    0:000> !runaway
 User Mode Time
  Thread       Time
   4:11ac      0 days 0:00:16.972
  18:225c      0 days 0:00:00.514
  17:48c       0 days 0:00:00.436
  14:e6c       0 days 0:00:00.436
   9:2780      0 days 0:00:00.421
   8:2508      0 days 0:00:00.421
   7:217c      0 days 0:00:00.421
  19:1554      0 days 0:00:00.358
  15:2744      0 days 0:00:00.358
   5:2708      0 days 0:00:00.358
  13:1fe4      0 days 0:00:00.327
  10:13d0      0 days 0:00:00.312
   6:1b24      0 days 0:00:00.296
  11:294       0 days 0:00:00.280
  16:1f78      0 days 0:00:00.265
  12:270c      0 days 0:00:00.249
   0:1d74      0 days 0:00:00.109
   2:1cbc      0 days 0:00:00.062
  20:21e4      0 days 0:00:00.015
  21:2704      0 days 0:00:00.000
   3:1c78      0 days 0:00:00.000
   1:200c      0 days 0:00:00.000
c++
windbg
freeze
asked on Stack Overflow Jan 27, 2015 by je-woong.ku • edited Oct 26, 2018 by GarCB613

1 Answer

1

Since you have 14 threads stuck (waiting - ZwWaitForSingleObject) in this, what seems to be logging logic trying to access shared state:

04e3ab2c 77de8e44 ntdll!ZwWaitForSingleObject+0x15
04e3ab90 77de8d28 ntdll!RtlpWaitOnCriticalSection+0x13e
04e3abb8 004159ea ntdll!RtlEnterCriticalSection+0x150
04e3abc0 004151a4 FatDatabaseCenter!FatLog::FatLogSyncObj_CriticalSection::Enter+0xa [d:\00.buildserver\jenkins\workspace\c++_server_Fatdatabasecenter_v30\src\Fatlib\Fatlog\syncobj\Fatlogsyncobj_criticalsection.cpp @ 29]
04e3abd8 00414fcc FatDatabaseCenter!FatLog::FatLogStore::write+0x14 [d:\00.buildserver\jenkins\workspace\c++_server_Fatdatabasecenter_v30\src\Fatlib\Fatlog\Fatlogstore.cpp @ 101]
04e3ebec 004162bf FatDatabaseCenter!FatLog::WriteLog+0x6c [d:\00.buildserver\jenkins\workspace\c++_server_Fatdatabasecenter_v30\src\Fatlib\Fatlog\Fatlog.cpp @ 156]
04e3ebfc 10033287 FatDatabaseCenter!FatLog::FatLog_AceLogLinker::log+0x6f [d:\00.buildserver\jenkins\workspace\c++_server_Fatdatabasecenter_v30\src\Fatlib\Fatlog\plugin\ace\Fatlog_aceloglinker.cpp @ 47]
WARNING: Stack unwind information not available. Following frames may be wrong.
00000000 00000000 ACE_5_8_3!ACE_Log_Msg::log+0x67

And only one stuck (waiting - ZwWaitForSingleObject) in this:

05d3b96c 750e6f1f ntdll!ZwWaitForSingleObject+0x15
05d3b9ac 750ee425 mswsock!SockWaitForSingleObject+0x1ba
05d3ba28 76f06f79 mswsock!WSPSend+0x218
05d3ba70 100067a9 ws2_32!send+0x86
WARNING: Stack unwind information not available. Following frames may be wrong.
05d3ba94 1006eb4f ACE_5_8_3!ACE::send_n_i+0x39
05d3baa8 0041690f ACE_5_8_3!ACE_SOCK_Stream::send_n+0x1f
05d3bac4 004151ff FatDatabaseCenter!FatLog::FatLog_NetworkMonitor_Handler_ACE::WriteLog+0x3f [d:\00.buildserver\jenkins\workspace\c++_server_Fatdatabasecenter_v30\src\Fatlib\Fatlog\plugin\ace\Fatlog_networkmonitor_handler_ace.cpp @ 56]
05d3bae4 00414fcc FatDatabaseCenter!FatLog::FatLogStore::write+0x6f [d:\00.buildserver\jenkins\workspace\c++_server_Fatdatabasecenter_v30\src\Fatlib\Fatlog\Fatlogstore.cpp @ 101]
05d3faf8 0042f20f FatDatabaseCen## Heading ##ter!FatLog::WriteLog+0x6c [d:\00.buildserver\jenkins\workspace\c++_server_Fatdatabasecenter_v30\src\Fatlib\Fatlog\Fatlog.cpp @ 156]
05d3fb6c 0042b4ff FatDatabaseCenter!FatSqlSelect::Select+0xef [d:\00.buildserver\jenkins\workspace\c++_server_Fatdatabasecenter_v30\src\Fatdatabasecenter30\service\oracleservice\ver2\sql\select\Fatsqlselect.cpp @ 43]
05d3fc3c 0042d1c9 FatDatabaseCenter!FatOracleService_V2::Select+0x19f [d:\00.buildserver\jenkins\workspace\c++_server_Fatdatabasecenter_v30\src\Fatdatabasecenter30\service\oracleservice\ver2\Fatoracleservice_v2.cpp @ 76]
05d3fc44 0042fd72 FatDatabaseCenter!FatOracleService_V2::ClientMsg+0x39 [d:\00.buildserver\jenkins\workspace\c++_server_Fatdatabasecenter_v30\src\Fatdatabasecenter30\service\oracleservice\ver2\Fatoracleservice_v2.cpp @ 58]
05d3fc58 00430771 FatDatabaseCenter!IFDCClientStream::process+0x32 [d:\00.buildserver\jenkins\workspace\c++_server_Fatdatabasecenter_v30\src\Fatdatabasecenter30\readhandler\iFDCclientstream.cpp @ 62]
05d3fd1c 0043059a FatDatabaseCenter!FDCClientStream_RIJINDAEL::recv_from_Client+0x341 [d:\00.buildserver\jenkins\workspace\c++_server_Fatdatabasecenter_v30\src\Fatdatabasecenter30\readhandler\FDCclientstream_rijindael.cpp @ 136]
05d3fde0 0042fe3c FatDatabaseCenter!FDCClientStream_RIJINDAEL::recv_from_Client+0x16a [d:\00.buildserver\jenkins\workspace\c++_server_Fatdatabasecenter_v30\src\Fatdatabasecenter30\readhandler\FDCclientstream_rijindael.cpp @ 67]
05d3fe04 1007a797 FatDatabaseCenter!FatReadHandler::handle_input+0xac [d:\00.buildserver\jenkins\workspace\c++_server_Fatdatabasecenter_v30\src\Fatdatabasecenter30\readhandler\Fatreadhandler.cpp @ 83]
05d3fe1c 1007a86e ACE_5_8_3!ACE_TP_Reactor::dispatch_socket_event+0x27
00000000 00000000 ACE_5_8_3!ACE_TP_Reactor::handle_socket_events+0xbe

My guess is that the second callstack deadlocked the other 14, or vice versa. You need to look if you anywhere in them take opposing locks, so that a thread that gets lock A doesn't wait for lock B held by another thread which is waiting for lock A.

It can also be that one of the network IO threads is deadlocking the others, specifically the only one that is not waiting for an event (all other callstacks are stuck in one of the wait-for-single or wait-for-multiple-objects functions):

0477ff5c 750e635c ntdll!ZwRemoveIoCompletion+0x15
0477ff88 75f2336a mswsock!SockAsyncThread+0x83
0477ff94 77de9f72 kernel32!BaseThreadInitThunk+0xe
0477ffd4 77de9f45 ntdll!__RtlUserThreadStart+0x70
0477ffec 00000000 ntdll!_RtlUserThreadStart+0x1b

But it's not running your code as it seems, so I don't know really.

answered on Stack Overflow Jan 27, 2015 by Johann Gerell • edited Jan 27, 2015 by Johann Gerell

User contributions licensed under CC BY-SA 3.0