I got the full dump using this command when the process doesn't respond and doesn't work.
procdump.exe -ma [PID]
Also, I tried to find the reason why this process showed strange behavior using WinDBG. But I'm just a WinDBG beginner, so I didn't get any hint from the FULL dump file. (Actually I can't understand a WinDBG message) Is there someone who can help me with this problem?
This is !locks command result.
0:000> !locks
CritSec +218daf0 at 0218daf0
WaiterWoken No
LockCount 14
RecursionCount 1
OwningThread 1554
EntryCount 0
ContentionCount e
*** Locked
Scanned 789 critical sections
This result says "NO LOCKS", right?
This is ~*k (all thread call stack) result.
0:000> ~*k
. 0 Id: 1ff0.1d74 Suspend: 0 Teb: 7efdd000 Unfrozen
ChildEBP RetAddr
0015fa08 75bd149d ntdll!ZwWaitForSingleObject+0x15
0015fa74 75f21194 KERNELBASE!WaitForSingleObjectEx+0x98
0015fa8c 75f21148 kernel32!WaitForSingleObjectExImplementation+0x75
0015faa0 100500fe kernel32!WaitForSingleObject+0x12
WARNING: Stack unwind information not available. Following frames may be wrong.
0015fac4 10050edb ACE_5_8_3!ACE_OS::cond_wait+0x3e
00000000 00000000 ACE_5_8_3!ACE_OS::cond_timedwait+0x1b
1 Id: 1ff0.200c Suspend: 0 Teb: 7efda000 Unfrozen
ChildEBP RetAddr
0449fdf4 77e02f91 ntdll!NtWaitForMultipleObjects+0x15
0449ff88 75f2336a ntdll!TppWaiterpThread+0x33d
0449ff94 77de9f72 kernel32!BaseThreadInitThunk+0xe
0449ffd4 77de9f45 ntdll!__RtlUserThreadStart+0x70
0449ffec 00000000 ntdll!_RtlUserThreadStart+0x1b
2 Id: 1ff0.1cbc Suspend: 0 Teb: 7efd7000 Unfrozen
ChildEBP RetAddr
0477ff5c 750e635c ntdll!ZwRemoveIoCompletion+0x15
0477ff88 75f2336a mswsock!SockAsyncThread+0x83
0477ff94 77de9f72 kernel32!BaseThreadInitThunk+0xe
0477ffd4 77de9f45 ntdll!__RtlUserThreadStart+0x70
0477ffec 00000000 ntdll!_RtlUserThreadStart+0x1b
3 Id: 1ff0.1c78 Suspend: 0 Teb: 7ef9c000 Unfrozen
ChildEBP RetAddr
0497fd78 75bd149d ntdll!ZwWaitForSingleObject+0x15
0497fde4 75f21194 KERNELBASE!WaitForSingleObjectEx+0x98
0497fdfc 75f21148 kernel32!WaitForSingleObjectExImplementation+0x75
0497fe10 100500fe kernel32!WaitForSingleObject+0x12
WARNING: Stack unwind information not available. Following frames may be wrong.
0497fe34 10050edb ACE_5_8_3!ACE_OS::cond_wait+0x3e
00000000 00000000 ACE_5_8_3!ACE_OS::cond_timedwait+0x1b
4 Id: 1ff0.11ac Suspend: 0 Teb: 7ef99000 Unfrozen
ChildEBP RetAddr
04a7fcf8 75bd149d ntdll!ZwWaitForSingleObject+0x15
04a7fd64 75f21194 KERNELBASE!WaitForSingleObjectEx+0x98
04a7fd7c 75f21148 kernel32!WaitForSingleObjectExImplementation+0x75
04a7fd90 10050f67 kernel32!WaitForSingleObject+0x12
WARNING: Stack unwind information not available. Following frames may be wrong.
04a7fde0 1001cf14 ACE_5_8_3!ACE_OS::cond_timedwait+0xa7
04a7fdfc 0041c4ac ACE_5_8_3!ACE_Condition<ACE_Recursive_Thread_Mutex>::wait+0x44
04a7fee0 10074b5c FatDatabaseCenter!ACE_Thread_Timer_Queue_Adapter<ACE_Timer_Heap_T<ACE_Event_Handler *,ACE_Event_Handler_Handle_Timeout_Upcall<ACE_Recursive_Thread_Mutex>,ACE_Recursive_Thread_Mutex>,ACE_Event_Handler *>::svc+0x16c [d:\00.buildserver\jenkins\workspace\c++_server_Fatdatabasecenter_v30\extlib\ace_5_8_3\ace\timer_queue_adapters.cpp @ 275]
04a7ff28 10075097 ACE_5_8_3!ACE_Task_Base::svc_run+0x2c
04a7ff80 73832a47 ACE_5_8_3!ACE_Thread_Adapter::invoke+0x87
04a7ff88 75f2336a msvcr80!_threadstartex+0x66 [f:\dd\vctools\crt_bld\self_x86\crt\src\threadex.c @ 326]
04a7ff94 77de9f72 kernel32!BaseThreadInitThunk+0xe
04a7ffd4 77de9f45 ntdll!__RtlUserThreadStart+0x70
04a7ffec 00000000 ntdll!_RtlUserThreadStart+0x1b
5 Id: 1ff0.2708 Suspend: 0 Teb: 7ef96000 Unfrozen
ChildEBP RetAddr
04bbab2c 77de8e44 ntdll!ZwWaitForSingleObject+0x15
04bbab90 77de8d28 ntdll!RtlpWaitOnCriticalSection+0x13e
04bbabb8 004159ea ntdll!RtlEnterCriticalSection+0x150
04bbabc0 004151a4 FatDatabaseCenter!FatLog::FatLogSyncObj_CriticalSection::Enter+0xa [d:\...\syncobj\Fatlogsyncobj_criticalsection.cpp @ 29]
04bbabd8 00414fcc FatDatabaseCenter!FatLog::FatLogStore::write+0x14 [d:\00.buildserver\jenkins\workspace\c++_server_Fatdatabasecenter_v30\src\Fatlib\Fatlog\Fatlogstore.cpp @ 101]
04bbebec 004162bf FatDatabaseCenter!FatLog::WriteLog+0x6c [d:\...\src\Fatlib\Fatlog\Fatlog.cpp @ 156]
04bbebfc 10033287 FatDatabaseCenter!FatLog::FatLog_AceLogLinker::log+0x6f [d:\...\src\Fatlib\Fatlog\plugin\ace\Fatlog_aceloglinker.cpp @ 47]
WARNING: Stack unwind information not available. Following frames may be wrong.
00000000 00000000 ACE_5_8_3!ACE_Log_Msg::log+0x67
6 Id: 1ff0.1b24 Suspend: 0 Teb: 7ef93000 Unfrozen
ChildEBP RetAddr
04cfab2c 77de8e44 ntdll!ZwWaitForSingleObject+0x15
04cfab90 77de8d28 ntdll!RtlpWaitOnCriticalSection+0x13e
04cfabb8 004159ea ntdll!RtlEnterCriticalSection+0x150
04cfabc0 004151a4 FatDatabaseCenter!FatLog::FatLogSyncObj_CriticalSection::Enter+0xa [d:\...\syncobj\Fatlogsyncobj_criticalsection.cpp @ 29]
04cfabd8 00414fcc FatDatabaseCenter!FatLog::FatLogStore::write+0x14 [d:\...\src\Fatlib\Fatlog\Fatlogstore.cpp @ 101]
04cfebec 004162bf FatDatabaseCenter!FatLog::WriteLog+0x6c [d:\...\src\Fatlib\Fatlog\Fatlog.cpp @ 156]
04cfebfc 10033287 FatDatabaseCenter!FatLog::FatLog_AceLogLinker::log+0x6f [d:\...\src\Fatlib\Fatlog\plugin\ace\Fatlog_aceloglinker.cpp @ 47]
WARNING: Stack unwind information not available. Following frames may be wrong.
00000000 00000000 ACE_5_8_3!ACE_Log_Msg::log+0x67
7 Id: 1ff0.217c Suspend: 0 Teb: 7ef90000 Unfrozen
ChildEBP RetAddr
04e3ab2c 77de8e44 ntdll!ZwWaitForSingleObject+0x15
04e3ab90 77de8d28 ntdll!RtlpWaitOnCriticalSection+0x13e
04e3abb8 004159ea ntdll!RtlEnterCriticalSection+0x150
04e3abc0 004151a4 FatDatabaseCenter!FatLog::FatLogSyncObj_CriticalSection::Enter+0xa [d:\...\syncobj\Fatlogsyncobj_criticalsection.cpp @ 29]
04e3abd8 00414fcc FatDatabaseCenter!FatLog::FatLogStore::write+0x14 [d:\...\src\Fatlib\Fatlog\Fatlogstore.cpp @ 101]
04e3ebec 004162bf FatDatabaseCenter!FatLog::WriteLog+0x6c [d:\...\src\Fatlib\Fatlog\Fatlog.cpp @ 156]
04e3ebfc 10033287 FatDatabaseCenter!FatLog::FatLog_AceLogLinker::log+0x6f [d:\...\src\Fatlib\Fatlog\plugin\ace\Fatlog_aceloglinker.cpp @ 47]
WARNING: Stack unwind information not available. Following frames may be wrong.
00000000 00000000 ACE_5_8_3!ACE_Log_Msg::log+0x67
8 Id: 1ff0.2508 Suspend: 0 Teb: 7ef8d000 Unfrozen
ChildEBP RetAddr
04f7ab2c 77de8e44 ntdll!ZwWaitForSingleObject+0x15
04f7ab90 77de8d28 ntdll!RtlpWaitOnCriticalSection+0x13e
04f7abb8 004159ea ntdll!RtlEnterCriticalSection+0x150
04f7abc0 004151a4 FatDatabaseCenter!FatLog::FatLogSyncObj_CriticalSection::Enter+0xa [d:\...\syncobj\Fatlogsyncobj_criticalsection.cpp @ 29]
04f7abd8 00414fcc FatDatabaseCenter!FatLog::FatLogStore::write+0x14 [d:\...\src\Fatlib\Fatlog\Fatlogstore.cpp @ 101]
04f7ebec 004162bf FatDatabaseCenter!FatLog::WriteLog+0x6c [d:\...\src\Fatlib\Fatlog\Fatlog.cpp @ 156]
04f7ebfc 10033287 FatDatabaseCenter!FatLog::FatLog_AceLogLinker::log+0x6f [d:\...\src\Fatlib\Fatlog\plugin\ace\Fatlog_aceloglinker.cpp @ 47]
WARNING: Stack unwind information not available. Following frames may be wrong.
00000000 00000000 ACE_5_8_3!ACE_Log_Msg::log+0x67
9 Id: 1ff0.2780 Suspend: 0 Teb: 7ef8a000 Unfrozen
ChildEBP RetAddr
050bab2c 77de8e44 ntdll!ZwWaitForSingleObject+0x15
050bab90 77de8d28 ntdll!RtlpWaitOnCriticalSection+0x13e
050babb8 004159ea ntdll!RtlEnterCriticalSection+0x150
050babc0 004151a4 FatDatabaseCenter!FatLog::FatLogSyncObj_CriticalSection::Enter+0xa [d:\...\syncobj\Fatlogsyncobj_criticalsection.cpp @ 29]
050babd8 00414fcc FatDatabaseCenter!FatLog::FatLogStore::write+0x14 [d:\...\src\Fatlib\Fatlog\Fatlogstore.cpp @ 101]
050bebec 004162bf FatDatabaseCenter!FatLog::WriteLog+0x6c [d:\...\src\Fatlib\Fatlog\Fatlog.cpp @ 156]
050bebfc 10033287 FatDatabaseCenter!FatLog::FatLog_AceLogLinker::log+0x6f [d:\...\src\Fatlib\Fatlog\plugin\ace\Fatlog_aceloglinker.cpp @ 47]
WARNING: Stack unwind information not available. Following frames may be wrong.
00000000 00000000 ACE_5_8_3!ACE_Log_Msg::log+0x67
10 Id: 1ff0.13d0 Suspend: 0 Teb: 7ef87000 Unfrozen
ChildEBP RetAddr
051fab2c 77de8e44 ntdll!ZwWaitForSingleObject+0x15
051fab90 77de8d28 ntdll!RtlpWaitOnCriticalSection+0x13e
051fabb8 004159ea ntdll!RtlEnterCriticalSection+0x150
051fabc0 004151a4 FatDatabaseCenter!FatLog::FatLogSyncObj_CriticalSection::Enter+0xa [d:\...\syncobj\Fatlogsyncobj_criticalsection.cpp @ 29]
051fabd8 00414fcc FatDatabaseCenter!FatLog::FatLogStore::write+0x14 [d:\...\src\Fatlib\Fatlog\Fatlogstore.cpp @ 101]
051febec 004162bf FatDatabaseCenter!FatLog::WriteLog+0x6c [d:\...\src\Fatlib\Fatlog\Fatlog.cpp @ 156]
051febfc 10033287 FatDatabaseCenter!FatLog::FatLog_AceLogLinker::log+0x6f [d:\...\src\Fatlib\Fatlog\plugin\ace\Fatlog_aceloglinker.cpp @ 47]
WARNING: Stack unwind information not available. Following frames may be wrong.
00000000 00000000 ACE_5_8_3!ACE_Log_Msg::log+0x67
11 Id: 1ff0.294 Suspend: 0 Teb: 7ef84000 Unfrozen
ChildEBP RetAddr
0533ab2c 77de8e44 ntdll!ZwWaitForSingleObject+0x15
0533ab90 77de8d28 ntdll!RtlpWaitOnCriticalSection+0x13e
0533abb8 004159ea ntdll!RtlEnterCriticalSection+0x150
0533abc0 004151a4 FatDatabaseCenter!FatLog::FatLogSyncObj_CriticalSection::Enter+0xa [d:\...\syncobj\Fatlogsyncobj_criticalsection.cpp @ 29]
0533abd8 00414fcc FatDatabaseCenter!FatLog::FatLogStore::write+0x14 [d:\...\src\Fatlib\Fatlog\Fatlogstore.cpp @ 101]
0533ebec 004162bf FatDatabaseCenter!FatLog::WriteLog+0x6c [d:\...\src\Fatlib\Fatlog\Fatlog.cpp @ 156]
0533ebfc 10033287 FatDatabaseCenter!FatLog::FatLog_AceLogLinker::log+0x6f [d:\...\src\Fatlib\Fatlog\plugin\ace\Fatlog_aceloglinker.cpp @ 47]
WARNING: Stack unwind information not available. Following frames may be wrong.
00000000 00000000 ACE_5_8_3!ACE_Log_Msg::log+0x67
12 Id: 1ff0.270c Suspend: 0 Teb: 7ef81000 Unfrozen
ChildEBP RetAddr
0547ab2c 77de8e44 ntdll!ZwWaitForSingleObject+0x15
0547ab90 77de8d28 ntdll!RtlpWaitOnCriticalSection+0x13e
0547abb8 004159ea ntdll!RtlEnterCriticalSection+0x150
0547abc0 004151a4 FatDatabaseCenter!FatLog::FatLogSyncObj_CriticalSection::Enter+0xa [d:\...\syncobj\Fatlogsyncobj_criticalsection.cpp @ 29]
0547abd8 00414fcc FatDatabaseCenter!FatLog::FatLogStore::write+0x14 [d:\...\src\Fatlib\Fatlog\Fatlogstore.cpp @ 101]
0547ebec 004162bf FatDatabaseCenter!FatLog::WriteLog+0x6c [d:\...\src\Fatlib\Fatlog\Fatlog.cpp @ 156]
0547ebfc 10033287 FatDatabaseCenter!FatLog::FatLog_AceLogLinker::log+0x6f [d:\...\src\Fatlib\Fatlog\plugin\ace\Fatlog_aceloglinker.cpp @ 47]
WARNING: Stack unwind information not available. Following frames may be wrong.
00000000 00000000 ACE_5_8_3!ACE_Log_Msg::log+0x67
13 Id: 1ff0.1fe4 Suspend: 0 Teb: 7ef7e000 Unfrozen
ChildEBP RetAddr
055bab2c 77de8e44 ntdll!ZwWaitForSingleObject+0x15
055bab90 77de8d28 ntdll!RtlpWaitOnCriticalSection+0x13e
055babb8 004159ea ntdll!RtlEnterCriticalSection+0x150
055babc0 004151a4 FatDatabaseCenter!FatLog::FatLogSyncObj_CriticalSection::Enter+0xa [d:\...\syncobj\Fatlogsyncobj_criticalsection.cpp @ 29]
055babd8 00414fcc FatDatabaseCenter!FatLog::FatLogStore::write+0x14 [d:\...\src\Fatlib\Fatlog\Fatlogstore.cpp @ 101]
055bebec 004162bf FatDatabaseCenter!FatLog::WriteLog+0x6c [d:\...\src\Fatlib\Fatlog\Fatlog.cpp @ 156]
055bebfc 10033287 FatDatabaseCenter!FatLog::FatLog_AceLogLinker::log+0x6f [d:\...\src\Fatlib\Fatlog\plugin\ace\Fatlog_aceloglinker.cpp @ 47]
WARNING: Stack unwind information not available. Following frames may be wrong.
00000000 00000000 ACE_5_8_3!ACE_Log_Msg::log+0x67
14 Id: 1ff0.e6c Suspend: 0 Teb: 7ef7b000 Unfrozen
ChildEBP RetAddr
056fab2c 77de8e44 ntdll!ZwWaitForSingleObject+0x15
056fab90 77de8d28 ntdll!RtlpWaitOnCriticalSection+0x13e
056fabb8 004159ea ntdll!RtlEnterCriticalSection+0x150
056fabc0 004151a4 FatDatabaseCenter!FatLog::FatLogSyncObj_CriticalSection::Enter+0xa [d:\...\syncobj\Fatlogsyncobj_criticalsection.cpp @ 29]
056fabd8 00414fcc FatDatabaseCenter!FatLog::FatLogStore::write+0x14 [d:\...\src\Fatlib\Fatlog\Fatlogstore.cpp @ 101]
056febec 004162bf FatDatabaseCenter!FatLog::WriteLog+0x6c [d:\...\src\Fatlib\Fatlog\Fatlog.cpp @ 156]
056febfc 10033287 FatDatabaseCenter!FatLog::FatLog_AceLogLinker::log+0x6f [d:\...\src\Fatlib\Fatlog\plugin\ace\Fatlog_aceloglinker.cpp @ 47]
WARNING: Stack unwind information not available. Following frames may be wrong.
00000000 00000000 ACE_5_8_3!ACE_Log_Msg::log+0x67
15 Id: 1ff0.2744 Suspend: 0 Teb: 7ef78000 Unfrozen
ChildEBP RetAddr
0583ab2c 77de8e44 ntdll!ZwWaitForSingleObject+0x15
0583ab90 77de8d28 ntdll!RtlpWaitOnCriticalSection+0x13e
0583abb8 004159ea ntdll!RtlEnterCriticalSection+0x150
0583abc0 004151a4 FatDatabaseCenter!FatLog::FatLogSyncObj_CriticalSection::Enter+0xa [d:\...\syncobj\Fatlogsyncobj_criticalsection.cpp @ 29]
0583abd8 00414fcc FatDatabaseCenter!FatLog::FatLogStore::write+0x14 [d:\...\src\Fatlib\Fatlog\Fatlogstore.cpp @ 101]
0583ebec 004162bf FatDatabaseCenter!FatLog::WriteLog+0x6c [d:\...\src\Fatlib\Fatlog\Fatlog.cpp @ 156]
0583ebfc 10033287 FatDatabaseCenter!FatLog::FatLog_AceLogLinker::log+0x6f [d:\...\src\Fatlib\Fatlog\plugin\ace\Fatlog_aceloglinker.cpp @ 47]
WARNING: Stack unwind information not available. Following frames may be wrong.
00000000 00000000 ACE_5_8_3!ACE_Log_Msg::log+0x67
16 Id: 1ff0.1f78 Suspend: 0 Teb: 7ef75000 Unfrozen
ChildEBP RetAddr
0597ab2c 77de8e44 ntdll!ZwWaitForSingleObject+0x15
0597ab90 77de8d28 ntdll!RtlpWaitOnCriticalSection+0x13e
0597abb8 004159ea ntdll!RtlEnterCriticalSection+0x150
0597abc0 004151a4 FatDatabaseCenter!FatLog::FatLogSyncObj_CriticalSection::Enter+0xa [d:\...\syncobj\Fatlogsyncobj_criticalsection.cpp @ 29]
0597abd8 00414fcc FatDatabaseCenter!FatLog::FatLogStore::write+0x14 [d:\...\src\Fatlib\Fatlog\Fatlogstore.cpp @ 101]
0597ebec 004162bf FatDatabaseCenter!FatLog::WriteLog+0x6c [d:\...\src\Fatlib\Fatlog\Fatlog.cpp @ 156]
0597ebfc 10033287 FatDatabaseCenter!FatLog::FatLog_AceLogLinker::log+0x6f [d:\...\src\Fatlib\Fatlog\plugin\ace\Fatlog_aceloglinker.cpp @ 47]
WARNING: Stack unwind information not available. Following frames may be wrong.
00000000 00000000 ACE_5_8_3!ACE_Log_Msg::log+0x67
17 Id: 1ff0.48c Suspend: 0 Teb: 7ef72000 Unfrozen
ChildEBP RetAddr
05abab2c 77de8e44 ntdll!ZwWaitForSingleObject+0x15
05abab90 77de8d28 ntdll!RtlpWaitOnCriticalSection+0x13e
05ababb8 004159ea ntdll!RtlEnterCriticalSection+0x150
05ababc0 004151a4 FatDatabaseCenter!FatLog::FatLogSyncObj_CriticalSection::Enter+0xa [d:\...\syncobj\Fatlogsyncobj_criticalsection.cpp @ 29]
05ababd8 00414fcc FatDatabaseCenter!FatLog::FatLogStore::write+0x14 [d:\...\src\Fatlib\Fatlog\Fatlogstore.cpp @ 101]
05abebec 004162bf FatDatabaseCenter!FatLog::WriteLog+0x6c [d:\...\src\Fatlib\Fatlog\Fatlog.cpp @ 156]
05abebfc 10033287 FatDatabaseCenter!FatLog::FatLog_AceLogLinker::log+0x6f [d:\...\src\Fatlib\Fatlog\plugin\ace\Fatlog_aceloglinker.cpp @ 47]
WARNING: Stack unwind information not available. Following frames may be wrong.
00000000 00000000 ACE_5_8_3!ACE_Log_Msg::log+0x67
18 Id: 1ff0.225c Suspend: 0 Teb: 7ef6f000 Unfrozen
ChildEBP RetAddr
05bfab2c 77de8e44 ntdll!ZwWaitForSingleObject+0x15
05bfab90 77de8d28 ntdll!RtlpWaitOnCriticalSection+0x13e
05bfabb8 004159ea ntdll!RtlEnterCriticalSection+0x150
05bfabc0 004151a4 FatDatabaseCenter!FatLog::FatLogSyncObj_CriticalSection::Enter+0xa [d:\...\syncobj\Fatlogsyncobj_criticalsection.cpp @ 29]
05bfabd8 00414fcc FatDatabaseCenter!FatLog::FatLogStore::write+0x14 [d:\...\src\Fatlib\Fatlog\Fatlogstore.cpp @ 101]
05bfebec 004162bf FatDatabaseCenter!FatLog::WriteLog+0x6c [d:\...\src\Fatlib\Fatlog\Fatlog.cpp @ 156]
05bfebfc 10033287 FatDatabaseCenter!FatLog::FatLog_AceLogLinker::log+0x6f [d:\...\src\Fatlib\Fatlog\plugin\ace\Fatlog_aceloglinker.cpp @ 47]
WARNING: Stack unwind information not available. Following frames may be wrong.
00000000 00000000 ACE_5_8_3!ACE_Log_Msg::log+0x67
19 Id: 1ff0.1554 Suspend: 0 Teb: 7ef6c000 Unfrozen
ChildEBP RetAddr
05d3b96c 750e6f1f ntdll!ZwWaitForSingleObject+0x15
05d3b9ac 750ee425 mswsock!SockWaitForSingleObject+0x1ba
05d3ba28 76f06f79 mswsock!WSPSend+0x218
05d3ba70 100067a9 ws2_32!send+0x86
WARNING: Stack unwind information not available. Following frames may be wrong.
05d3ba94 1006eb4f ACE_5_8_3!ACE::send_n_i+0x39
05d3baa8 0041690f ACE_5_8_3!ACE_SOCK_Stream::send_n+0x1f
05d3bac4 004151ff FatDatabaseCenter!FatLog::FatLog_NetworkMonitor_Handler_ACE::WriteLog+0x3f [d:\...\src\Fatlib\Fatlog\plugin\ace\Fatlog_networkmonitor_handler_ace.cpp @ 56]
05d3bae4 00414fcc FatDatabaseCenter!FatLog::FatLogStore::write+0x6f [d:\...\src\Fatlib\Fatlog\Fatlogstore.cpp @ 101]
05d3faf8 0042f20f FatDatabaseCen## Heading ##ter!FatLog::WriteLog+0x6c [d:\...\src\Fatlib\Fatlog\Fatlog.cpp @ 156]
05d3fb6c 0042b4ff FatDatabaseCenter!FatSqlSelect::Select+0xef [d:\...\src\Fatdatabasecenter30\service\oracleservice\ver2\sql\select\Fatsqlselect.cpp @ 43]
05d3fc3c 0042d1c9 FatDatabaseCenter!FatOracleService_V2::Select+0x19f [d:\...\src\Fatdatabasecenter30\service\oracleservice\ver2\Fatoracleservice_v2.cpp @ 76]
05d3fc44 0042fd72 FatDatabaseCenter!FatOracleService_V2::ClientMsg+0x39 [d:\...\src\Fatdatabasecenter30\service\oracleservice\ver2\Fatoracleservice_v2.cpp @ 58]
05d3fc58 00430771 FatDatabaseCenter!IFDCClientStream::process+0x32 [d:\...\src\Fatdatabasecenter30\readhandler\iFDCclientstream.cpp @ 62]
05d3fd1c 0043059a FatDatabaseCenter!FDCClientStream_RIJINDAEL::recv_from_Client+0x341 [d:\...\src\Fatdatabasecenter30\readhandler\FDCclientstream_rijindael.cpp @ 136]
05d3fde0 0042fe3c FatDatabaseCenter!FDCClientStream_RIJINDAEL::recv_from_Client+0x16a [d:\...\src\Fatdatabasecenter30\readhandler\FDCclientstream_rijindael.cpp @ 67]
05d3fe04 1007a797 FatDatabaseCenter!FatReadHandler::handle_input+0xac [d:\...\src\Fatdatabasecenter30\readhandler\Fatreadhandler.cpp @ 83]
05d3fe1c 1007a86e ACE_5_8_3!ACE_TP_Reactor::dispatch_socket_event+0x27
00000000 00000000 ACE_5_8_3!ACE_TP_Reactor::handle_socket_events+0xbe
20 Id: 1ff0.21e4 Suspend: 0 Teb: 7ef69000 Unfrozen
ChildEBP RetAddr
0627fee0 7362a41c ntdll!NtWaitForMultipleObjects+0x15
0627ff88 75f2336a winmm!timeThread+0x3c
0627ff94 77de9f72 kernel32!BaseThreadInitThunk+0xe
0627ffd4 77de9f45 ntdll!__RtlUserThreadStart+0x70
0627ffec 00000000 ntdll!_RtlUserThreadStart+0x1b
21 Id: 1ff0.2704 Suspend: 0 Teb: 7ef66000 Unfrozen
ChildEBP RetAddr
065ffe28 77e03392 ntdll!NtWaitForWorkViaWorkerFactory+0x12
065fff88 75f2336a ntdll!TppWorkerThread+0x216
065fff94 77de9f72 kernel32!BaseThreadInitThunk+0xe
065fffd4 77de9f45 ntdll!__RtlUserThreadStart+0x70
065fffec 00000000 ntdll!_RtlUserThreadStart+0x1b
And, Here is the !analyze -v command result.
0:000> !analyze -v
*******************************************************************************
* *
* Exception Analysis *
* *
*******************************************************************************
FAULTING_IP:
+38
00000000 ?? ???
EXCEPTION_RECORD: ffffffff -- (.exr 0xffffffffffffffff)
ExceptioFatdress: 00000000
ExceptionCode: 80000003 (Break instruction exception)
ExceptionFlags: 00000000
NumberParameters: 0
CONTEXT: 00000000 -- (.cxr 0x0;r)
eax=00000000 ebx=00000000 ecx=00000000 edx=00000000 esi=00000398 edi=00000000
eip=77dcf8d1 esp=0015fa08 ebp=0015fa74 iopl=0 nv up ei pl zr na pe nc
cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000246
ntdll!ZwWaitForSingleObject+0x15:
77dcf8d1 83c404 add esp,4
FAULTING_THREAD: 00001d74
DEFAULT_BUCKET_ID: STATUS_BREAKPOINT
PROCESS_NAME: FatDatabaseCenter.exe
ERROR_CODE: (NTSTATUS) 0x80000003 - {
EXCEPTION_CODE: (HRESULT) 0x80000003 (2147483651) - .
NTGLOBALFLAG: 0
APPLICATION_VERIFIER_FLAGS: 0
APP: Fatdatabasecenter.exe
ANALYSIS_VERSION: 6.3.9600.17298 (debuggers(dbg).141024-1500) x86fre
PRIMARY_PROBLEM_CLASS: STATUS_BREAKPOINT
BUGCHECK_STR: APPLICATION_FAULT_STATUS_BREAKPOINT
LAST_CONTROL_TRANSFER: from 75bd149d to 77dcf8d1
STACK_TEXT:
0015fa08 75bd149d 00000398 00000000 00000000 ntdll!ZwWaitForSingleObject+0x15
0015fa74 75f21194 00000398 ffffffff 00000000 KERNELBASE!WaitForSingleObjectEx+0x98
0015fa8c 75f21148 00000398 ffffffff 00000000 kernel32!WaitForSingleObjectExImplementation+0x75
0015faa0 100500fe 00000398 ffffffff 042d5718 kernel32!WaitForSingleObject+0x12
WARNING: Stack unwind information not available. Following frames may be wrong.
0015fac4 10050edb 042d5718 042d56fc 00000000 ACE_5_8_3!ACE_OS::cond_wait+0x3e
00000000 00000000 00000000 00000000 00000000 ACE_5_8_3!ACE_OS::cond_timedwait+0x1b
STACK_COMMAND: ~0s; .ecxr ; kb
FOLLOWUP_IP:
ACE_5_8_3!ACE_OS::cond_wait+3e
100500fe 57 push edi
SYMBOL_STACK_INDEX: 4
SYMBOL_NAME: ace_5_8_3!ACE_OS::cond_wait+3e
FOLLOWUP_NAME: MachineOwner
MODULE_NAME: ACE_5_8_3
IMAGE_NAME: ACE_5.8.3.dll
DEBUG_FLR_IMAGE_TIMESTAMP: 4f5eea5c
FAILURE_BUCKET_ID: STATUS_BREAKPOINT_80000003_ACE_5.8.3.dll!ACE_OS::cond_wait
BUCKET_ID: APPLICATION_FAULT_STATUS_BREAKPOINT_ace_5_8_3!ACE_OS::cond_wait+3e
ANALYSIS_SOURCE: UM
FAILURE_ID_HASH_STRING: um:status_breakpoint_80000003_ace_5.8.3.dll!ace_os::cond_wait
FAILURE_ID_HASH: {3b6cdfb5-4df8-55cb-7886-936eef582ae1}
Followup: MachineOwner
---------
Say everything if you want more information. Please give me some hint why this process fall in HANG !!!
Edit
Here is the !analyze -v -hang command result.
0:000> !analyze -v -hang
*******************************************************************************
* *
* Exception Analysis *
* *
*******************************************************************************
FAULTING_IP:
+0
00000000 ?? ???
EXCEPTION_RECORD: ffffffff -- (.exr 0xffffffffffffffff)
ExceptioFADdress: 00000000
ExceptionCode: 80000003 (Break instruction exception)
ExceptionFlags: 00000000
NumberParameters: 0
CONTEXT: 00000000 -- (.cxr 0x0;r)
eax=00000000 ebx=00000000 ecx=00000000 edx=00000000 esi=00000398 edi=00000000
eip=77dcf8d1 esp=0015fa08 ebp=0015fa74 iopl=0 nv up ei pl zr na pe nc
cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000246
ntdll!ZwWaitForSingleObject+0x15:
77dcf8d1 83c404 add esp,4
FAULTING_THREAD: 00000013
BUGCHECK_STR: HANG
PROCESS_NAME: FADDatabaseCenter.exe
ERROR_CODE: (NTSTATUS) 0xcfffffff - <Unable to get error code text>
EXCEPTION_CODE: (NTSTATUS) 0xcfffffff - <Unable to get error code text>
NTGLOBALFLAG: 0
APPLICATION_VERIFIER_FLAGS: 0
APP: FADdatabasecenter.exe
ANALYSIS_VERSION: 6.3.9600.17298 (debuggers(dbg).141024-1500) x86fre
DERIVED_WAIT_CHAIN:
Dl Eid Cid WaitType
-- --- ------- --------------------------
0 1ff0.1d74 Speculated (Triage) -->
5 1ff0.2708 Event -->
19 1ff0.1554 Event
WAIT_CHAIN_COMMAND: ~0s;k;;~5s;k;;~19s;k;;
BLOCKING_THREAD: 00001554
DEFAULT_BUCKET_ID: APPLICATION_HANG_BlockedOn_NetworkCall
PRIMARY_PROBLEM_CLASS: APPLICATION_HANG_BlockedOn_NetworkCall
LAST_CONTROL_TRANSFER: from 750e6f1f to 77dcf8d1
STACK_TEXT:
05d3b96c 750e6f1f 000004c8 00000001 05d3b994 ntdll!ZwWaitForSingleObject+0x15
05d3b9ac 750ee425 000004c8 000004d4 00000002 mswsock!SockWaitForSingleObject+0x1ba
05d3ba28 76f06f79 000004d4 05d3ba60 00000001 mswsock!WSPSend+0x218
05d3ba70 100067a9 000004d4 0043d634 00000002 ws2_32!send+0x86
WARNING: Stack unwind information not available. Following frames may be wrong.
05d3ba94 1006eb4f 000004d4 0043d634 00000002 ACE_5_8_3!ACE::send_n_i+0x39
05d3baa8 0041690f 0043d634 00000002 00000000 ACE_5_8_3!ACE_SOCK_Stream::send_n+0x1f
05d3bac4 004151ff 00000020 05d3baf4 04282878 FADDatabaseCenter!FADLog::FADLog_NetworkMonitor_Handler_ACE::WriteLog+0x3f
05d3bae4 00414fcc 00000020 05d3baf4 20202020 FADDatabaseCenter!FADLog::FADLogStore::write+0x6f
05d3faf8 0042f20f 00000020 0043f154 06613050 FADDatabaseCenter!FADLog::WriteLog+0x6c
05d3fb6c 0042b4ff 0439fc40 0438ba68 05d3fbc4 FADDatabaseCenter!FADSqlSelect::Select+0xef
05d3fc3c 0042d1c9 0438ba68 0042fd72 0000000b FADDatabaseCenter!FADOracleService_V2::Select+0x19f
05d3fc44 0042fd72 0000000b 0438ba68 0438ba68 FADDatabaseCenter!FADOracleService_V2::ClientMsg+0x39
05d3fc58 00430771 0438ba68 0000000b 000011f6 FADDatabaseCenter!IFDCClientStream::process+0x32
05d3fd1c 0043059a 0000000b 000011f6 0438ba68 FADDatabaseCenter!FDCClientStream_RIJINDAEL::recv_from_Client+0x341
05d3fde0 0042fe3c 0000f004 00001205 04382e7c FADDatabaseCenter!FDCClientStream_RIJINDAEL::recv_from_Client+0x16a
05d3fe04 1007a797 00000600 05d3fe88 0015fca8 FADDatabaseCenter!FADReadHandler::handle_input+0xac
05d3fe1c 1007a86e 05d3fe34 00000001 0015fca8 ACE_5_8_3!ACE_TP_Reactor::dispatch_socket_event+0x27
00000000 00000000 00000000 00000000 00000000 ACE_5_8_3!ACE_TP_Reactor::handle_socket_events+0xbe
FOLLOWUP_IP:
ACE_5_8_3!ACE::send_n_i+39
100067a9 83f8ff cmp eax,0FFFFFFFFh
SYMBOL_STACK_INDEX: 4
SYMBOL_NAME: ace_5_8_3!ACE::send_n_i+39
FOLLOWUP_NAME: MachineOwner
MODULE_NAME: ACE_5_8_3
IMAGE_NAME: ACE_5.8.3.dll
DEBUG_FLR_IMAGE_TIMESTAMP: 4f5eea5c
STACK_COMMAND: ~19s ; kb
BUCKET_ID: HANG_ace_5_8_3!ACE::send_n_i+39
FAILURE_BUCKET_ID: APPLICATION_HANG_BlockedOn_NetworkCall_cfffffff_ACE_5.8.3.dll!ACE::send_n_i
ANALYSIS_SOURCE: UM
FAILURE_ID_HASH_STRING: um:application_hang_blockedon_networkcall_cfffffff_ace_5.8.3.dll!ace::send_n_i
FAILURE_ID_HASH: {aea64438-3ae6-3ab0-78e9-92b7ba34a6f3}
Followup: MachineOwner
And Here is !runaway result.
0:000> !runaway
User Mode Time
Thread Time
4:11ac 0 days 0:00:16.972
18:225c 0 days 0:00:00.514
17:48c 0 days 0:00:00.436
14:e6c 0 days 0:00:00.436
9:2780 0 days 0:00:00.421
8:2508 0 days 0:00:00.421
7:217c 0 days 0:00:00.421
19:1554 0 days 0:00:00.358
15:2744 0 days 0:00:00.358
5:2708 0 days 0:00:00.358
13:1fe4 0 days 0:00:00.327
10:13d0 0 days 0:00:00.312
6:1b24 0 days 0:00:00.296
11:294 0 days 0:00:00.280
16:1f78 0 days 0:00:00.265
12:270c 0 days 0:00:00.249
0:1d74 0 days 0:00:00.109
2:1cbc 0 days 0:00:00.062
20:21e4 0 days 0:00:00.015
21:2704 0 days 0:00:00.000
3:1c78 0 days 0:00:00.000
1:200c 0 days 0:00:00.000
Since you have 14 threads stuck (waiting - ZwWaitForSingleObject) in this, what seems to be logging logic trying to access shared state:
04e3ab2c 77de8e44 ntdll!ZwWaitForSingleObject+0x15
04e3ab90 77de8d28 ntdll!RtlpWaitOnCriticalSection+0x13e
04e3abb8 004159ea ntdll!RtlEnterCriticalSection+0x150
04e3abc0 004151a4 FatDatabaseCenter!FatLog::FatLogSyncObj_CriticalSection::Enter+0xa [d:\00.buildserver\jenkins\workspace\c++_server_Fatdatabasecenter_v30\src\Fatlib\Fatlog\syncobj\Fatlogsyncobj_criticalsection.cpp @ 29]
04e3abd8 00414fcc FatDatabaseCenter!FatLog::FatLogStore::write+0x14 [d:\00.buildserver\jenkins\workspace\c++_server_Fatdatabasecenter_v30\src\Fatlib\Fatlog\Fatlogstore.cpp @ 101]
04e3ebec 004162bf FatDatabaseCenter!FatLog::WriteLog+0x6c [d:\00.buildserver\jenkins\workspace\c++_server_Fatdatabasecenter_v30\src\Fatlib\Fatlog\Fatlog.cpp @ 156]
04e3ebfc 10033287 FatDatabaseCenter!FatLog::FatLog_AceLogLinker::log+0x6f [d:\00.buildserver\jenkins\workspace\c++_server_Fatdatabasecenter_v30\src\Fatlib\Fatlog\plugin\ace\Fatlog_aceloglinker.cpp @ 47]
WARNING: Stack unwind information not available. Following frames may be wrong.
00000000 00000000 ACE_5_8_3!ACE_Log_Msg::log+0x67
And only one stuck (waiting - ZwWaitForSingleObject) in this:
05d3b96c 750e6f1f ntdll!ZwWaitForSingleObject+0x15
05d3b9ac 750ee425 mswsock!SockWaitForSingleObject+0x1ba
05d3ba28 76f06f79 mswsock!WSPSend+0x218
05d3ba70 100067a9 ws2_32!send+0x86
WARNING: Stack unwind information not available. Following frames may be wrong.
05d3ba94 1006eb4f ACE_5_8_3!ACE::send_n_i+0x39
05d3baa8 0041690f ACE_5_8_3!ACE_SOCK_Stream::send_n+0x1f
05d3bac4 004151ff FatDatabaseCenter!FatLog::FatLog_NetworkMonitor_Handler_ACE::WriteLog+0x3f [d:\00.buildserver\jenkins\workspace\c++_server_Fatdatabasecenter_v30\src\Fatlib\Fatlog\plugin\ace\Fatlog_networkmonitor_handler_ace.cpp @ 56]
05d3bae4 00414fcc FatDatabaseCenter!FatLog::FatLogStore::write+0x6f [d:\00.buildserver\jenkins\workspace\c++_server_Fatdatabasecenter_v30\src\Fatlib\Fatlog\Fatlogstore.cpp @ 101]
05d3faf8 0042f20f FatDatabaseCen## Heading ##ter!FatLog::WriteLog+0x6c [d:\00.buildserver\jenkins\workspace\c++_server_Fatdatabasecenter_v30\src\Fatlib\Fatlog\Fatlog.cpp @ 156]
05d3fb6c 0042b4ff FatDatabaseCenter!FatSqlSelect::Select+0xef [d:\00.buildserver\jenkins\workspace\c++_server_Fatdatabasecenter_v30\src\Fatdatabasecenter30\service\oracleservice\ver2\sql\select\Fatsqlselect.cpp @ 43]
05d3fc3c 0042d1c9 FatDatabaseCenter!FatOracleService_V2::Select+0x19f [d:\00.buildserver\jenkins\workspace\c++_server_Fatdatabasecenter_v30\src\Fatdatabasecenter30\service\oracleservice\ver2\Fatoracleservice_v2.cpp @ 76]
05d3fc44 0042fd72 FatDatabaseCenter!FatOracleService_V2::ClientMsg+0x39 [d:\00.buildserver\jenkins\workspace\c++_server_Fatdatabasecenter_v30\src\Fatdatabasecenter30\service\oracleservice\ver2\Fatoracleservice_v2.cpp @ 58]
05d3fc58 00430771 FatDatabaseCenter!IFDCClientStream::process+0x32 [d:\00.buildserver\jenkins\workspace\c++_server_Fatdatabasecenter_v30\src\Fatdatabasecenter30\readhandler\iFDCclientstream.cpp @ 62]
05d3fd1c 0043059a FatDatabaseCenter!FDCClientStream_RIJINDAEL::recv_from_Client+0x341 [d:\00.buildserver\jenkins\workspace\c++_server_Fatdatabasecenter_v30\src\Fatdatabasecenter30\readhandler\FDCclientstream_rijindael.cpp @ 136]
05d3fde0 0042fe3c FatDatabaseCenter!FDCClientStream_RIJINDAEL::recv_from_Client+0x16a [d:\00.buildserver\jenkins\workspace\c++_server_Fatdatabasecenter_v30\src\Fatdatabasecenter30\readhandler\FDCclientstream_rijindael.cpp @ 67]
05d3fe04 1007a797 FatDatabaseCenter!FatReadHandler::handle_input+0xac [d:\00.buildserver\jenkins\workspace\c++_server_Fatdatabasecenter_v30\src\Fatdatabasecenter30\readhandler\Fatreadhandler.cpp @ 83]
05d3fe1c 1007a86e ACE_5_8_3!ACE_TP_Reactor::dispatch_socket_event+0x27
00000000 00000000 ACE_5_8_3!ACE_TP_Reactor::handle_socket_events+0xbe
My guess is that the second callstack deadlocked the other 14, or vice versa. You need to look if you anywhere in them take opposing locks, so that a thread that gets lock A doesn't wait for lock B held by another thread which is waiting for lock A.
It can also be that one of the network IO threads is deadlocking the others, specifically the only one that is not waiting for an event (all other callstacks are stuck in one of the wait-for-single or wait-for-multiple-objects functions):
0477ff5c 750e635c ntdll!ZwRemoveIoCompletion+0x15
0477ff88 75f2336a mswsock!SockAsyncThread+0x83
0477ff94 77de9f72 kernel32!BaseThreadInitThunk+0xe
0477ffd4 77de9f45 ntdll!__RtlUserThreadStart+0x70
0477ffec 00000000 ntdll!_RtlUserThreadStart+0x1b
But it's not running your code as it seems, so I don't know really.
User contributions licensed under CC BY-SA 3.0