I have a dotnet core, 3.1.8, program with five FileWatcher. They watch folders on a network filesystem. With some load (200 - 250 files in total), the program crash unexpectedly. These files arrive at the same time, moved by another process from another server.
In event viewer, I see something but that's not really relevant (for me?).
Faultinq module name: coreclr.dll, version: 470020.41105, time stamp: Ox5f3397ec
Exception code: Oxc0000005
Fault offset: Ox0000000000191dcb
Faultinq process id: Oxf7a0
Faultinq application start time: Ox01d686baa97d91f4
I have a dump generated when the crash occur. I'm not very comfortable with that. So I try some stuff with WinDBG.
*******************************************************************************
* *
* Exception Analysis *
* *
*******************************************************************************
SYMSRV: BYINDEX: 0x25D
https://msdl.microsoft.com/download/symbols
SOS_AMD64_AMD64_4.700.20.41105.dll
5F3397EC56f000
SYMSRV: UNC: C:\ProgramData\Dbg\sym\SOS_AMD64_AMD64_4.700.20.41105.dll\5F3397EC56f000\SOS_AMD64_AMD64_4.700.20.41105.dll - path not found
SYMSRV: UNC: C:\ProgramData\Dbg\sym\SOS_AMD64_AMD64_4.700.20.41105.dll\5F3397EC56f000\SOS_AMD64_AMD64_4.700.20.41105.dl_ - path not found
SYMSRV: UNC: C:\ProgramData\Dbg\sym\SOS_AMD64_AMD64_4.700.20.41105.dll\5F3397EC56f000\file.ptr - path not found
SYMSRV: HTTPGET: /download/symbols/SOS_AMD64_AMD64_4.700.20.41105.dll/5F3397EC56f000/SOS_AMD64_AMD64_4.700.20.41105.dll
SYMSRV: HttpQueryInfo: 80190194 - HTTP_STATUS_NOT_FOUND
SYMSRV: HTTPGET: /download/symbols/SOS_AMD64_AMD64_4.700.20.41105.dll/5F3397EC56f000/SOS_AMD64_AMD64_4.700.20.41105.dl_
SYMSRV: HttpQueryInfo: 80190194 - HTTP_STATUS_NOT_FOUND
SYMSRV: HTTPGET: /download/symbols/SOS_AMD64_AMD64_4.700.20.41105.dll/5F3397EC56f000/file.ptr
SYMSRV: HttpQueryInfo: 80190194 - HTTP_STATUS_NOT_FOUND
SYMSRV: RESULT: 0x80190194
SYMSRV: BYINDEX: 0x25E
C:\debug\symbols
SOS_AMD64_AMD64_4.700.20.41105.dll
5F3397EC56f000
SYMSRV: UNC: C:\debug\symbols\SOS_AMD64_AMD64_4.700.20.41105.dll\5F3397EC56f000\SOS_AMD64_AMD64_4.700.20.41105.dll - path not found
SYMSRV: UNC: C:\debug\symbols\SOS_AMD64_AMD64_4.700.20.41105.dll\5F3397EC56f000\SOS_AMD64_AMD64_4.700.20.41105.dl_ - path not found
SYMSRV: UNC: C:\debug\symbols\SOS_AMD64_AMD64_4.700.20.41105.dll\5F3397EC56f000\file.ptr - path not found
SYMSRV: RESULT: 0x80070003
SYMSRV: BYINDEX: 0x25F
https://msdl.microsoft.com/download/symbols
coreclr.dll
5F3397EC56f000
SYMSRV: PATH: C:\ProgramData\Dbg\sym\coreclr.dll\5F3397EC56f000\coreclr.dll
SYMSRV: RESULT: 0x00000000
DBGHELP: C:\ProgramData\Dbg\sym\coreclr.dll\5F3397EC56f000\coreclr.dll - OK
SYMSRV: BYINDEX: 0x260
https://msdl.microsoft.com/download/symbols
coreclr.dll
5F3397EC56f000
SYMSRV: PATH: C:\ProgramData\Dbg\sym\coreclr.dll\5F3397EC56f000\coreclr.dll
SYMSRV: RESULT: 0x00000000
DBGHELP: C:\ProgramData\Dbg\sym\coreclr.dll\5F3397EC56f000\coreclr.dll - OK
Failed to request MethodData, not in JIT code range
MethodDesc: 00007ffc6ca07240
Method Name: System.IO.FileSystemWatcher.ParseEventBufferAndNotifyForEach(Byte[])
Class: 00007ffc6c9f8a38
MethodTable: 00007ffc6ca073e0
mdToken: 000000000600005A
Module: 00007ffc6ca06520
IsJitted: yes
Current CodeAddr: 00007ffc6d7a4980
Version History:
ILCodeVersion: 0000000000000000
ReJIT ID: 0
IL Addr: 0000000000000000
CodeAddr: 00007ffc6d7a4980 (OptimizedTier1)
NativeCodeVersion: 000001E5EB703B40
CodeAddr: 00007ffce5c3bc10 (ReadyToRun)
NativeCodeVersion: 0000000000000000
SYMSRV: BYINDEX: 0x261
https://msdl.microsoft.com/download/symbols
System.IO.FileSystem.Watcher.pdb
FA68AFAB85EA4A57B6E929B2AB30C79D1
SYMSRV: PATH: C:\ProgramData\Dbg\sym\System.IO.FileSystem.Watcher.pdb\FA68AFAB85EA4A57B6E929B2AB30C79D1\System.IO.FileSystem.Watcher.pdb
SYMSRV: RESULT: 0x00000000
Source file: /_/src/System.IO.FileSystem.Watcher/src/System/IO/FileSystemWatcher.Win32.cs @ 249
DBGHELP: coreclr is not source indexed
KEY_VALUES_STRING: 1
Key : AV.Dereference
Value: NullPtr
Key : AV.Fault
Value: Read
Key : Analysis.CPU.mSec
Value: 2749
Key : Analysis.DebugAnalysisProvider.CPP
Value: Create: 8007007e on xxx
Key : Analysis.DebugData
Value: CreateObject
Key : Analysis.DebugModel
Value: CreateObject
Key : Analysis.Elapsed.mSec
Value: 3297
Key : Analysis.Memory.CommitPeak.Mb
Value: 502
Key : Analysis.System
Value: CreateObject
Key : CLR.Engine
Value: CORECLR
Key : CLR.Version
Value: 4.700.20.41105
Key : Timeline.OS.Boot.DeltaSec
Value: 10334662
Key : Timeline.Process.Start.DeltaSec
Value: 73117
Key : WER.OS.Branch
Value: rs5_release
Key : WER.OS.Timestamp
Value: 2018-09-14T14:34:00Z
Key : WER.OS.Version
Value: 10.0.17763.1
Key : WER.Process.Version
Value: 1.0.0.0
ADDITIONAL_XML: 1
OS_BUILD_LAYERS: 1
NTGLOBALFLAG: 0
PROCESS_BAM_CURRENT_THROTTLED: 0
PROCESS_BAM_PREVIOUS_THROTTLED: 0
APPLICATION_VERIFIER_FLAGS: 0
CONTEXT: 00007ffd20f787d1 -- (.cxr 0x7ffd20f787d1)
rax=480000021e850f0f rbx=8341d23300000086 rcx=894800000098838b
rdx=850f00007d830845 rsi=0327840ff8b54148 rdi=483cfe24068a0000
rip=0ffda817418d0000 rsp=000003e6830f02fd rbp=3e80d68b48ff3345
r8=0a8a00000341840f r9=0f583cc52241c18a r10=4cc18a0000052884
r11=0f403cf02401428d r12=f2f9800000022484 r13=418d000a293c840f
r14=000282860f013c3e r15=0565840ff3f98000
iopl=2 nv dn di ng nz na pe cy
cs=2b5b ss=1065 ds=000a es=8b44 fs=0870 gs=8b4c efl=00006583
0ffda817`418d0000 ?? ???
Resetting default scope
EXCEPTION_RECORD: 00000022d55bd850 -- (.exr 0x22d55bd850)
ExceptionAddress: 0000000000000000
ExceptionCode: 0000000b
ExceptionFlags: 00000000
NumberParameters: 0
PROCESS_NAME: xxx.exe
READ_ADDRESS: 0000000000000000
ERROR_CODE: (NTSTATUS) 0xc0000005 - The instruction at 0x%p referenced memory at 0x%p. The memory could not be %s.
EXCEPTION_CODE_STR: c0000005
FAULTING_THREAD: ffffffff
IP_ON_HEAP: 0ffda817418d0000
The fault address in not in any loaded module, please check your build's rebase
log at <releasedir>\bin\build_logs\timebuild\ntrebase.log for module which may
contain the address if it were loaded.
UNALIGNED_STACK_POINTER: 000003e6830f02fd
STACK_TEXT:
00000022`d55bc8a0 00007ffc`cc46789f coreclr!EEPolicy::HandleFatalError+0x8b
00000022`d55bceb0 00007ffc`cc381de6 coreclr!ProcessCLRException+0x148c86
00000022`d55bd0b0 00007ffd`2101478f ntdll!RtlpExecuteHandlerForException+0xf
00000022`d55bd0e0 00007ffd`20f74bef ntdll!RtlDispatchException+0x40f
00000022`d55bd810 00007ffd`20f789e6 ntdll!RtlRaiseException+0x316
00000022`d55be010 00007ffd`1d369709 KERNELBASE!RaiseException+0x69
00000022`d55be0f0 00007ffc`cc336619 coreclr!NakedThrowHelper2+0x9
00000022`d55be120 00007ffc`cc33664b coreclr!NakedThrowHelper_RspAligned+0x1e
00000022`d55be648 00007ffc`cc336655 coreclr!NakedThrowHelper_FixRsp+0x5
00000022`d55be650 00007ffc`6d7a49d4 unknown!unknown+0x0
00000022`d55be658 00007ffd`2100f874 ntdll!NtRemoveIoCompletion+0x14
00000022`d55be660 00007ffd`20fc9870 ntdll!RtlSetLastWin32Error+0x40
STACK_COMMAND: .ecxr ; kb ; ** Pseudo Context ** Pseudo ** Value: 207fc8fb430 ** ; kb
FAULTING_SOURCE_LINE: f:\workspace\_work\1\s\src\vm\eepolicy.cpp
FAULTING_SOURCE_FILE: f:\workspace\_work\1\s\src\vm\eepolicy.cpp
FAULTING_SOURCE_LINE_NUMBER: 1177
FAULTING_SOURCE_CODE:
No source found for 'f:\workspace\_work\1\s\src\vm\eepolicy.cpp'
SYMBOL_NAME: coreclr!EEPolicy::HandleFatalError+8b
MODULE_NAME: coreclr
IMAGE_NAME: coreclr.dll
FAILURE_BUCKET_ID: NULL_POINTER_READ_c0000005_coreclr.dll!EEPolicy::HandleFatalError
OS_VERSION: 10.0.17763.1
BUILDLAB_STR: rs5_release
OSPLATFORM_TYPE: x64
OSNAME: Windows 10
IMAGE_VERSION: 4.700.20.41105
FAILURE_ID_HASH: {14f01c38-e318-9c7b-c758-0ddeb36df05a}
Followup: MachineOwner
---------
DBGHELP: coreclr is not source indexed
And also with dotnet-dump.
> clrstack
OS Thread Id: 0xfd4c (27)
Child SP IP Call Site
00000022D55BE150 00007ffccc46789f [FaultingExceptionFrame: 00000022d55be150]
00000022D55BE650 00007FFC6D7A49D4 System.IO.FileSystemWatcher.ParseEventBufferAndNotifyForEach(Byte[]) [/_/src/System.IO.FileSystem.Watcher/src/System/IO/FileSystemWatcher.Win32.cs @ 249]
00000022D55BE6F0 00007FFC6D7A48E6 System.IO.FileSystemWatcher.ReadDirectoryChangesCallback(UInt32, UInt32, System.Threading.NativeOverlapped*) [/_/src/System.IO.FileSystem.Watcher/src/System/IO/FileSystemWatcher.Win32.cs @ 242]
00000022D55BE750 00007FFC6D6F189C System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) [/_/src/System.Private.CoreLib/shared/System/Threading/ExecutionContext.cs @ 201]
00000022D55BE7C0 00007FFC6D7359B5 System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32, UInt32, System.Threading.NativeOverlapped*) [/_/src/System.Private.CoreLib/src/System/Threading/Overlapped.cs @ 59]
00000022D55BE8F0 00007ffccc336ba3 [GCFrame: 00000022d55be8f0]
00000022D55BEAB0 00007ffccc336ba3 [DebuggerU2MCatchHandlerFrame: 00000022d55beab0]
> pe
Exception object: 000001e580001198
Exception type: System.ExecutionEngineException
Message: <none>
InnerException: <none>
StackTrace (generated):
<none>
StackTraceString: <none>
HResult: 80131506
For the moment, I have no small repro steps. Any suggestion?
User contributions licensed under CC BY-SA 3.0