Windows Server Crash Dump Analysis

Question

Windows Server Crash Dump Analysis

I'm not certain that this is the right venue for this question, but a programmer friend of mine said I should try this here.

My company's main application is hosted on a terminal server running Windows Server 2008. Since last Thursday we have seen this server crash and reboot 3 times, and we just went live with this server on the previous Tuesday. I have used the the WinDbg program to analyze the crash dump file, but I'm a little outside by depth at this point and I'm hoping that someone out there can help me get this issue resolved.

The application that appears to me to be at fault is winoac.exe which is the executable for SmartWare 4.5 (www.smartware4.com). This is the platform that our application runs on. If this application is at fault, is there anything that I can do about it, other than complaining to SmartWare?

Thanks a million to anybody that can help.

Here are the results of the analysis.

Microsoft (R) Windows Debugger Version 6.10.0003.233 X86
Copyright (c) Microsoft Corporation. All rights reserved.


Loading Dump File [C:\Users\esinnard\Desktop\Windows Dumps\1-29-09\MEMORY.DMP]
Kernel Summary Dump File: Only kernel address space is available

Symbol search path is: SRV*C:\ProgramData\Symbols*http://msdl.microsoft.com/download/symbols
Executable search path is: 
Windows Server 2008/Windows Vista SP1 Kernel Version 6001 (Service Pack 1) MP (8 procs) Free x86 compatible
Product: Server, suite: TerminalServer
Built by: 6001.18145.x86fre.vistasp1_gdr.080917-1612
Machine Name:
Kernel base = 0x81c41000 PsLoadedModuleList = 0x81d4e930
Debug session time: Thu Jan 29 12:49:43.870 2009 (GMT-6)
System Uptime: 0 days 11:18:08.929
Loading Kernel Symbols
...............................................................
................................................................
..............
Loading User Symbols
PEB is paged out (Peb.Ldr = 7ffd400c).  Type ".hh dbgerr001" for details
Loading unloaded module list
.....
*******************************************************************************
*                                                                             *
*                        Bugcheck Analysis                                    *
*                                                                             *
*******************************************************************************

Use !analyze -v to get detailed debugging information.

BugCheck 8E, {c0000005, 81c88043, 9cef0840, 0}

Page bd1f2 not present in the dump file. Type ".hh dbgerr004" for details
Page bc9c3 not present in the dump file. Type ".hh dbgerr004" for details
PEB is paged out (Peb.Ldr = 7ffd400c).  Type ".hh dbgerr001" for details
PEB is paged out (Peb.Ldr = 7ffd400c).  Type ".hh dbgerr001" for details
Probably caused by : RDPDD.dll ( RDPDD!OE2_TableEncodeOrderFields+11e )

Followup: MachineOwner
---------

7: kd> !analyze -v
*******************************************************************************
*                                                                             *
*                        Bugcheck Analysis                                    *
*                                                                             *
*******************************************************************************

KERNEL_MODE_EXCEPTION_NOT_HANDLED (8e)
This is a very common bugcheck.  Usually the exception address pinpoints
the driver/function that caused the problem.  Always note this address
as well as the link date of the driver/image that contains this address.
Some common problems are exception code 0x80000003.  This means a hard
coded breakpoint or assertion was hit, but this system was booted
/NODEBUG.  This is not supposed to happen as developers should never have
hardcoded breakpoints in retail code, but ...
If this happens, make sure a debugger gets connected, and the
system is booted /DEBUG.  This will let us see why this breakpoint is
happening.
Arguments:
Arg1: c0000005, The exception code that was not handled
Arg2: 81c88043, The address that the exception occurred at
Arg3: 9cef0840, Trap Frame
Arg4: 00000000

Debugging Details:
------------------

PEB is paged out (Peb.Ldr = 7ffd400c).  Type ".hh dbgerr001" for details
PEB is paged out (Peb.Ldr = 7ffd400c).  Type ".hh dbgerr001" for details

EXCEPTION_CODE: (NTSTATUS) 0xc0000005 - The instruction at 0x%08lx referenced memory at 0x%08lx. The memory could not be %s.

FAULTING_IP: 
nt!RtlInitUnicodeString+1b
81c88043 f266af          repne scas word ptr es:[edi]

TRAP_FRAME:  9cef0840 -- (.trap 0xffffffff9cef0840)
ErrCode = 00000000
eax=00000000 ebx=fe414fd8 ecx=ffffffec edx=9cef0914 esi=fe40fcf0 edi=fe415000
eip=81c88043 esp=9cef08b4 ebp=9cef0924 iopl=0         nv up ei pl zr na pe nc
cs=0008  ss=0010  ds=0023  es=0023  fs=0030  gs=0000             efl=00010246
nt!RtlInitUnicodeString+0x1b:
81c88043 f266af          repne scas word ptr es:[edi]
Resetting default scope

DEFAULT_BUCKET_ID:  VISTA_DRIVER_FAULT

BUGCHECK_STR:  0x8E

PROCESS_NAME:  WINOAC.EXE

CURRENT_IRQL:  0

LAST_CONTROL_TRANSFER:  from 81c72fbe to 81cfc759

STACK_TEXT:  
9cef0400 81c72fbe 0000008e c0000005 81c88043 nt!KeBugCheckEx+0x1e
9cef07d0 81c9953a 9cef07ec 00000000 9cef0840 nt!KiDispatchException+0x1a9
9cef0838 81c994ee 9cef0924 81c88043 badb0d00 nt!CommonDispatchException+0x4a
9cef085c 9976011a 99771680 997708e8 00000000 nt!Kei386EoiHelper+0x186
9cef0924 9959efab 5d0102bb 00000006 00000002 RDPDD!OE2_TableEncodeOrderFields+0x11e
9cef0a0c 995aeaf8 5d0102bb 00000006 00000002 win32k!xxxRealDrawMenuItem+0x80b
9cef0abc 9958455b 5d0102bb 0110007e 9cef0b04 win32k!xxxDrawState+0x1c9
9cef0b2c 995853e1 5d0102bb fe40fc78 00c8d0d4 win32k!xxxDrawMenuItem+0x3f8
9cef0b98 9959f511 5d0102bb 00000000 fe414570 win32k!xxxMenuDraw+0x1f2
9cef0bf0 994ed1d6 00000017 5d0102bb 00000004 win32k!xxxMenuBarDraw+0x1bf
9cef0c38 9950c0f5 fe414570 5d0102bb 00000001 win32k!xxxDrawWindowFrame+0xf7
9cef0cb4 9950d73d fe414570 00000085 090402df win32k!xxxRealDefWindowProc+0x88b
9cef0ccc 994e673d fe414570 00000085 090402df win32k!xxxWrapRealDefWindowProc+0x2b
9cef0ce8 9950d6f4 fe414570 00000085 090402df win32k!NtUserfnNCDESTROY+0x27
9cef0d20 81c9897a 000200ba 00000085 090402df win32k!NtUserMessageCall+0xc6
9cef0d20 77089a94 000200ba 00000085 090402df nt!KiFastCallEntry+0x12a
WARNING: Frame IP not in any known module. Following frames may be wrong.
0012d7cc 00000000 00000000 00000000 00000000 0x77089a94


STACK_COMMAND:  kb

FOLLOWUP_IP: 
RDPDD!OE2_TableEncodeOrderFields+11e
9976011a 8b4518          mov     eax,dword ptr [ebp+18h]

SYMBOL_STACK_INDEX:  4

SYMBOL_NAME:  RDPDD!OE2_TableEncodeOrderFields+11e

FOLLOWUP_NAME:  MachineOwner

MODULE_NAME: RDPDD

IMAGE_NAME:  RDPDD.dll

DEBUG_FLR_IMAGE_TIMESTAMP:  4791923e

FAILURE_BUCKET_ID:  0x8E_RDPDD!OE2_TableEncodeOrderFields+11e

BUCKET_ID:  0x8E_RDPDD!OE2_TableEncodeOrderFields+11e

Followup: MachineOwner
---------

------------------------------------------------------------------------------------------


Microsoft (R) Windows Debugger Version 6.10.0003.233 X86
Copyright (c) Microsoft Corporation. All rights reserved.


Loading Dump File [C:\Users\esinnard\Desktop\Windows Dumps\1-29-09\MEMORY.DMP]
Kernel Summary Dump File: Only kernel address space is available

Symbol search path is: SRV*C:\ProgramData\Symbols*http://msdl.microsoft.com/download/symbols
Executable search path is: 
Windows Server 2008/Windows Vista SP1 Kernel Version 6001 (Service Pack 1) MP (8 procs) Free x86 compatible
Product: Server, suite: TerminalServer
Built by: 6001.18145.x86fre.vistasp1_gdr.080917-1612
Machine Name:
Kernel base = 0x81c41000 PsLoadedModuleList = 0x81d4e930
Debug session time: Thu Jan 29 12:49:43.870 2009 (GMT-6)
System Uptime: 0 days 11:18:08.929
Loading Kernel Symbols
...............................................................
................................................................
..............
Loading User Symbols
PEB is paged out (Peb.Ldr = 7ffd400c).  Type ".hh dbgerr001" for details
Loading unloaded module list
.....
*******************************************************************************
*                                                                             *
*                        Bugcheck Analysis                                    *
*                                                                             *
*******************************************************************************

Use !analyze -v to get detailed debugging information.

BugCheck 8E, {c0000005, 81c88043, 9cef0840, 0}

Page bd1f2 not present in the dump file. Type ".hh dbgerr004" for details
Page bc9c3 not present in the dump file. Type ".hh dbgerr004" for details
PEB is paged out (Peb.Ldr = 7ffd400c).  Type ".hh dbgerr001" for details
PEB is paged out (Peb.Ldr = 7ffd400c).  Type ".hh dbgerr001" for details
Probably caused by : RDPDD.dll ( RDPDD!OE2_TableEncodeOrderFields+11e )

Followup: MachineOwner
---------

7: kd> !analyze -v
*******************************************************************************
*                                                                             *
*                        Bugcheck Analysis                                    *
*                                                                             *
*******************************************************************************

KERNEL_MODE_EXCEPTION_NOT_HANDLED (8e)
This is a very common bugcheck.  Usually the exception address pinpoints
the driver/function that caused the problem.  Always note this address
as well as the link date of the driver/image that contains this address.
Some common problems are exception code 0x80000003.  This means a hard
coded breakpoint or assertion was hit, but this system was booted
/NODEBUG.  This is not supposed to happen as developers should never have
hardcoded breakpoints in retail code, but ...
If this happens, make sure a debugger gets connected, and the
system is booted /DEBUG.  This will let us see why this breakpoint is
happening.
Arguments:
Arg1: c0000005, The exception code that was not handled
Arg2: 81c88043, The address that the exception occurred at
Arg3: 9cef0840, Trap Frame
Arg4: 00000000

Debugging Details:
------------------

PEB is paged out (Peb.Ldr = 7ffd400c).  Type ".hh dbgerr001" for details
PEB is paged out (Peb.Ldr = 7ffd400c).  Type ".hh dbgerr001" for details

EXCEPTION_CODE: (NTSTATUS) 0xc0000005 - The instruction at 0x%08lx referenced memory at 0x%08lx. The memory could not be %s.

FAULTING_IP: 
nt!RtlInitUnicodeString+1b
81c88043 f266af          repne scas word ptr es:[edi]

TRAP_FRAME:  9cef0840 -- (.trap 0xffffffff9cef0840)
ErrCode = 00000000
eax=00000000 ebx=fe414fd8 ecx=ffffffec edx=9cef0914 esi=fe40fcf0 edi=fe415000
eip=81c88043 esp=9cef08b4 ebp=9cef0924 iopl=0         nv up ei pl zr na pe nc
cs=0008  ss=0010  ds=0023  es=0023  fs=0030  gs=0000             efl=00010246
nt!RtlInitUnicodeString+0x1b:
81c88043 f266af          repne scas word ptr es:[edi]
Resetting default scope

DEFAULT_BUCKET_ID:  VISTA_DRIVER_FAULT

BUGCHECK_STR:  0x8E

PROCESS_NAME:  WINOAC.EXE

CURRENT_IRQL:  0

LAST_CONTROL_TRANSFER:  from 81c72fbe to 81cfc759

STACK_TEXT:  
9cef0400 81c72fbe 0000008e c0000005 81c88043 nt!KeBugCheckEx+0x1e
9cef07d0 81c9953a 9cef07ec 00000000 9cef0840 nt!KiDispatchException+0x1a9
9cef0838 81c994ee 9cef0924 81c88043 badb0d00 nt!CommonDispatchException+0x4a
9cef085c 9976011a 99771680 997708e8 00000000 nt!Kei386EoiHelper+0x186
9cef0924 9959efab 5d0102bb 00000006 00000002 RDPDD!OE2_TableEncodeOrderFields+0x11e
9cef0a0c 995aeaf8 5d0102bb 00000006 00000002 win32k!xxxRealDrawMenuItem+0x80b
9cef0abc 9958455b 5d0102bb 0110007e 9cef0b04 win32k!xxxDrawState+0x1c9
9cef0b2c 995853e1 5d0102bb fe40fc78 00c8d0d4 win32k!xxxDrawMenuItem+0x3f8
9cef0b98 9959f511 5d0102bb 00000000 fe414570 win32k!xxxMenuDraw+0x1f2
9cef0bf0 994ed1d6 00000017 5d0102bb 00000004 win32k!xxxMenuBarDraw+0x1bf
9cef0c38 9950c0f5 fe414570 5d0102bb 00000001 win32k!xxxDrawWindowFrame+0xf7
9cef0cb4 9950d73d fe414570 00000085 090402df win32k!xxxRealDefWindowProc+0x88b
9cef0ccc 994e673d fe414570 00000085 090402df win32k!xxxWrapRealDefWindowProc+0x2b
9cef0ce8 9950d6f4 fe414570 00000085 090402df win32k!NtUserfnNCDESTROY+0x27
9cef0d20 81c9897a 000200ba 00000085 090402df win32k!NtUserMessageCall+0xc6
9cef0d20 77089a94 000200ba 00000085 090402df nt!KiFastCallEntry+0x12a
WARNING: Frame IP not in any known module. Following frames may be wrong.
0012d7cc 00000000 00000000 00000000 00000000 0x77089a94


STACK_COMMAND:  kb

FOLLOWUP_IP: 
RDPDD!OE2_TableEncodeOrderFields+11e
9976011a 8b4518          mov     eax,dword ptr [ebp+18h]

SYMBOL_STACK_INDEX:  4

SYMBOL_NAME:  RDPDD!OE2_TableEncodeOrderFields+11e

FOLLOWUP_NAME:  MachineOwner

MODULE_NAME: RDPDD

IMAGE_NAME:  RDPDD.dll

DEBUG_FLR_IMAGE_TIMESTAMP:  4791923e

FAILURE_BUCKET_ID:  0x8E_RDPDD!OE2_TableEncodeOrderFields+11e

BUCKET_ID:  0x8E_RDPDD!OE2_TableEncodeOrderFields+11e

Followup: MachineOwner
---------

------------------------------------------------------------------------------------------


Microsoft (R) Windows Debugger Version 6.10.0003.233 X86
Copyright (c) Microsoft Corporation. All rights reserved.


Loading Dump File [C:\Users\esinnard\Desktop\Windows Dumps\2-3-09-2\MEMORY.DMP]
Kernel Summary Dump File: Only kernel address space is available

Symbol search path is: SRV*C:\ProgramData\Symbols*http://msdl.microsoft.com/download/symbols
Executable search path is: 
Windows Server 2008/Windows Vista SP1 Kernel Version 6001 (Service Pack 1) MP (8 procs) Free x86 compatible
Product: Server, suite: TerminalServer
Built by: 6001.18145.x86fre.vistasp1_gdr.080917-1612
Machine Name:
Kernel base = 0x81c13000 PsLoadedModuleList = 0x81d20930
Debug session time: Tue Feb  3 14:20:03.117 2009 (GMT-6)
System Uptime: 0 days 2:00:33.869
Loading Kernel Symbols
...............................................................
................................................................
.............
Loading User Symbols
PEB is paged out (Peb.Ldr = 7ffdc00c).  Type ".hh dbgerr001" for details
Loading unloaded module list
.....
*******************************************************************************
*                                                                             *
*                        Bugcheck Analysis                                    *
*                                                                             *
*******************************************************************************

Use !analyze -v to get detailed debugging information.

BugCheck 8E, {c0000005, 81c5a043, d60a5840, 0}

Page bce51 not present in the dump file. Type ".hh dbgerr004" for details
Page bce22 not present in the dump file. Type ".hh dbgerr004" for details
Page bb16b not present in the dump file. Type ".hh dbgerr004" for details
Page bce5a not present in the dump file. Type ".hh dbgerr004" for details
Page bce5a not present in the dump file. Type ".hh dbgerr004" for details
Page bce5a not present in the dump file. Type ".hh dbgerr004" for details
PEB is paged out (Peb.Ldr = 7ffdc00c).  Type ".hh dbgerr001" for details
PEB is paged out (Peb.Ldr = 7ffdc00c).  Type ".hh dbgerr001" for details
Probably caused by : win32k.sys ( win32k!OffBitBlt+97 )

Followup: MachineOwner
---------

0: kd> !analyze -v
*******************************************************************************
*                                                                             *
*                        Bugcheck Analysis                                    *
*                                                                             *
*******************************************************************************

KERNEL_MODE_EXCEPTION_NOT_HANDLED (8e)
This is a very common bugcheck.  Usually the exception address pinpoints
the driver/function that caused the problem.  Always note this address
as well as the link date of the driver/image that contains this address.
Some common problems are exception code 0x80000003.  This means a hard
coded breakpoint or assertion was hit, but this system was booted
/NODEBUG.  This is not supposed to happen as developers should never have
hardcoded breakpoints in retail code, but ...
If this happens, make sure a debugger gets connected, and the
system is booted /DEBUG.  This will let us see why this breakpoint is
happening.
Arguments:
Arg1: c0000005, The exception code that was not handled
Arg2: 81c5a043, The address that the exception occurred at
Arg3: d60a5840, Trap Frame
Arg4: 00000000

Debugging Details:
------------------

Page bb16b not present in the dump file. Type ".hh dbgerr004" for details
Page bce5a not present in the dump file. Type ".hh dbgerr004" for details
Page bce5a not present in the dump file. Type ".hh dbgerr004" for details
Page bce5a not present in the dump file. Type ".hh dbgerr004" for details
PEB is paged out (Peb.Ldr = 7ffdc00c).  Type ".hh dbgerr001" for details
PEB is paged out (Peb.Ldr = 7ffdc00c).  Type ".hh dbgerr001" for details

EXCEPTION_CODE: (NTSTATUS) 0xc0000005 - The instruction at 0x%08lx referenced memory at 0x%08lx. The memory could not be %s.

FAULTING_IP: 
nt!RtlInitUnicodeString+1b
81c5a043 f266af          repne scas word ptr es:[edi]

TRAP_FRAME:  d60a5840 -- (.trap 0xffffffffd60a5840)
ErrCode = 00000000
eax=00000000 ebx=fe41afd8 ecx=ffffffec edx=d60a5914 esi=fe40f5e0 edi=fe41b000
eip=81c5a043 esp=d60a58b4 ebp=d60a5924 iopl=0         nv up ei pl zr na pe nc
cs=0008  ss=0010  ds=0023  es=0023  fs=0030  gs=0000             efl=00010246
nt!RtlInitUnicodeString+0x1b:
81c5a043 f266af          repne scas word ptr es:[edi]
Resetting default scope

DEFAULT_BUCKET_ID:  VISTA_DRIVER_FAULT

BUGCHECK_STR:  0x8E

PROCESS_NAME:  WINOAC.EXE

CURRENT_IRQL:  0

LAST_CONTROL_TRANSFER:  from 81c44fbe to 81cce759

STACK_TEXT:  
d60a5400 81c44fbe 0000008e c0000005 81c5a043 nt!KeBugCheckEx+0x1e
d60a57d0 81c6b53a d60a57ec 00000000 d60a5840 nt!KiDispatchException+0x1a9
d60a5838 81c6b4ee d60a5924 81c5a043 badb0d00 nt!CommonDispatchException+0x4a
d60a585c 999e2242 ff888010 00000000 00000000 nt!Kei386EoiHelper+0x186
d60a5924 999befab 1401009b 00000006 00000002 win32k!OffBitBlt+0x97
d60a5a0c 999ceaf8 1401009b 00000006 00000002 win32k!xxxRealDrawMenuItem+0x80b
d60a5abc 999a455b 1401009b 0110007e d60a5b04 win32k!xxxDrawState+0x1c9
d60a5b2c 999a53e1 1401009b fe40d168 00c8d0d4 win32k!xxxDrawMenuItem+0x3f8
d60a5b98 999bf511 1401009b 00000000 fe418398 win32k!xxxMenuDraw+0x1f2
d60a5bf0 9990d1d6 00000017 1401009b 00000004 win32k!xxxMenuBarDraw+0x1bf
d60a5c38 9992c0f5 fe418398 1401009b 00000001 win32k!xxxDrawWindowFrame+0xf7
d60a5cb4 9992d73d fe418398 00000085 0904035f win32k!xxxRealDefWindowProc+0x88b
d60a5ccc 9990673d fe418398 00000085 0904035f win32k!xxxWrapRealDefWindowProc+0x2b
d60a5ce8 9992d6f4 fe418398 00000085 0904035f win32k!NtUserfnNCDESTROY+0x27
d60a5d20 81c6a97a 0003001c 00000085 0904035f win32k!NtUserMessageCall+0xc6
d60a5d20 77049a94 0003001c 00000085 0904035f nt!KiFastCallEntry+0x12a
WARNING: Frame IP not in any known module. Following frames may be wrong.
0012d7cc 00000000 00000000 00000000 00000000 0x77049a94


STACK_COMMAND:  kb

FOLLOWUP_IP: 
win32k!OffBitBlt+97
999e2242 8b4d20          mov     ecx,dword ptr [ebp+20h]

SYMBOL_STACK_INDEX:  4

SYMBOL_NAME:  win32k!OffBitBlt+97

FOLLOWUP_NAME:  MachineOwner

MODULE_NAME: win32k

IMAGE_NAME:  win32k.sys

DEBUG_FLR_IMAGE_TIMESTAMP:  48d1b9ef

FAILURE_BUCKET_ID:  0x8E_win32k!OffBitBlt+97

BUCKET_ID:  0x8E_win32k!OffBitBlt+97

Followup: MachineOwner
---------

windows

crash-dumps

asked on Stack Overflow Feb 3, 2009 by

user62188 • edited Mar 21, 2017 by

Thomas Weller

2 Answers

You should apply any patches for the OS that might be out there (especially if they mention they are related to Terminal Server or RDP). You should also probably contact Microsoft support.

The crash dump looks like the crash is happening in the RDP driver.

Even if the winoac.exe application is passing bad data to win32k.sys (the display subsystem) that results in the crash, device drivers are never supposed to crash the system - they should detect and handle the problem appropriately, even if it means the application crashes. The driver should never crash, so MS should take an interest in this so they can fix it.

answered on Stack Overflow Feb 4, 2009 by

Michael Burr

Unless Smartware has developed their own drivers it should never be possible for a user mode application to bluescreen a windows NT server.

So, ignoring all that information, you are either looking at a buggy device driver - step 1 - find and install any updates for drivers on the system, OR the hardware is beginning to fail. even bug free drivers might need to throw a bug check when the actual hardware they depend on is failing.

win32k.sys is the kernel driver side of the win32 subsystem, not specifically a display driver at all. However the call stack does implicate that something related to drawing crashed, so, perhaps starting with updating the systems video drivers - or replacing the video card if its not onboard might help.

answered on Stack Overflow Feb 5, 2009 by

Chris Becke

User contributions licensed under CC BY-SA 3.0