Windows Debugger(Windbg) unable to load Symbols for SQL Dump File

I am troubleshooting am issue with an automated job that rebuilds our indexes and text search. The Specific error:

2020-04-08 02:46:16.29 spid56 Timeout occurred while waiting for latch: class 'FCB', id 0000000629898238, type 3, Task 0x0000000633017C28 : 0, waittime 300 seconds, flags 0x2000000019, owning task 0x0000000624023C28. Continuing to wait.

My plan was to find and examine the thread connected to the Owning Task. My guide is this article at: https://mssqlwiki.com/2012/09/07/latch-timeout-and-sql-server-latch/

The problem is that I cant get the symbols to load. I have tried the Microsoft source as well as the NuGet source:

.sympath srv*c:\Websymbols*https://symbols.nuget.org/download/symbols

.sympath cache*c:\MyCachedSymbols;srv*C:\websymbols*https://msdl.microsoft.com/download/symbols

When I do a .reload /f, I get these messages:

0:060> .reload /f
.
SYMSRV:  BYINDEX: 0x395
         c:\symbols*https://msdl.microsoft.com/download/symbols
         sqlservr.exe
         5DF306DC60000

SYMSRV:  UNC: c:\symbols\sqlservr.exe\5DF306DC60000\sqlservr.exe - path not found
SYMSRV:  UNC: c:\symbols\sqlservr.exe\5DF306DC60000\sqlservr.ex_ - path not found
SYMSRV:  UNC: c:\symbols\sqlservr.exe\5DF306DC60000\file.ptr - path not found
SYMSRV:  HTTPGET: /download/symbols/sqlservr.exe/5DF306DC60000/sqlservr.exe
SYMSRV:  HttpQueryInfo: 80190194 - HTTP_STATUS_NOT_FOUND
SYMSRV:  HTTPGET: /download/symbols/sqlservr.exe/5DF306DC60000/sqlservr.ex_
SYMSRV:  HttpQueryInfo: 80190194 - HTTP_STATUS_NOT_FOUND
SYMSRV:  HTTPGET: /download/symbols/sqlservr.exe/5DF306DC60000/file.ptr
SYMSRV:  HttpQueryInfo: 80190194 - HTTP_STATUS_NOT_FOUND
SYMSRV:  RESULT: 0x80190194
DBGHELP: C:\Program Files (x86)\Windows Kits\10\Debuggers\sqlservr.exe - file not found

SYMSRV:  BYINDEX: 0x396
         https://msdl.microsoft.com/download/symbols
         sqlservr.exe
         5DF306DC60000

SYMSRV:  UNC: C:\ProgramData\dbg\sym\sqlservr.exe\5DF306DC60000\sqlservr.exe - path not found

SYMSRV:  UNC: C:\ProgramData\dbg\sym\sqlservr.exe\5DF306DC60000\sqlservr.ex_ - path not found

SYMSRV:  UNC: C:\ProgramData\dbg\sym\sqlservr.exe\5DF306DC60000\file.ptr - path not found

SYMSRV:  HTTPGET: /download/symbols/sqlservr.exe/5DF306DC60000/sqlservr.exe
SYMSRV:  HttpQueryInfo: 80190194 - HTTP_STATUS_NOT_FOUND
SYMSRV:  HTTPGET: /download/symbols/sqlservr.exe/5DF306DC60000/sqlservr.ex_
SYMSRV:  HttpQueryInfo: 80190194 - HTTP_STATUS_NOT_FOUND
SYMSRV:  HTTPGET: /download/symbols/sqlservr.exe/5DF306DC60000/file.ptr
SYMSRV:  HttpQueryInfo: 80190194 - HTTP_STATUS_NOT_FOUND
SYMSRV:  RESULT: 0x80190194
DBGHELP: C:\Program Files (x86)\Windows Kits\10\Debuggers\sqlservr.exe - file not found

SYMSRV:  BYINDEX: 0x397
         c:\websymbols*https://symbols.nuget.org/download/symbols
         sqlservr.exe
         5DF306DC60000

SYMSRV:  UNC: c:\websymbols\sqlservr.exe\5DF306DC60000\sqlservr.exe - path not found

SYMSRV:  UNC: c:\websymbols\sqlservr.exe\5DF306DC60000\sqlservr.ex_ - path not found

SYMSRV:  UNC: c:\websymbols\sqlservr.exe\5DF306DC60000\file.ptr - path not found
SYMSRV:  HTTPGET: /download/symbols/sqlservr.exe/5DF306DC60000/sqlservr.exe
SYMSRV:  HttpQueryInfo: 80190193 - HTTP_STATUS_FORBIDDEN
SYMSRV:  RESULT: 0x80190193
DBGHELP: C:\Program Files (x86)\Windows Kits\10\Debuggers\sqlservr.exe - file not found

DBGENG:  C:\Program Files\Microsoft SQL Server\MSSQL12.ERPODBVMI\MSSQL\Binn\sqlservr.exe - Couldn't map image from disk.

Unable to load image C:\Program Files\Microsoft SQL Server\MSSQL12.ERPODBVMI\MSSQL\Binn\sqlservr.exe, Win32 error 0n2

DBGENG:  sqlservr.exe - Partial symbol image load missing image info
DBGHELP: Module is not fully loaded into memory.
DBGHELP: Searching for symbols using debugger-provided data.
SYMSRV:  BYINDEX: 0x398
         c:\websymbols*https://symbols.nuget.org/download/symbols
         sqlservr.pdb
         7B78390C11144A34B1C95415F29068192

SYMSRV:  PATH: c:\websymbols\sqlservr.pdb\7B78390C11144A34B1C95415F29068192\sqlservr.pdb

SYMSRV:  RESULT: 0x00000000
*** WARNING: Unable to verify timestamp for sqlservr.exe
DBGHELP: sqlservr - public symbols  
        c:\websymbols\sqlservr.pdb\7B78390C11144A34B1C95415F29068192\sqlservr.pdb

Can anyone tell me why I get these path error ? When I look at the actual symbol folders, I can see that they were populated over the web, but for some reason, they don't seem to be loading.

Thank You !

Added: Stack Text
STACK_TEXT:  
KERNELBASE!RaiseException+0x68
sqlservr!CDmpDump::Dump+0x65
sqllang!SQLDumperLibraryInvoke+0x1e4
sqllang!SQLLangDumperLibraryInvoke+0x15e
sqllang!CImageHelper::DoMiniDump+0x426
sqllang!stackTrace+0x9d5
sqlmin!LatchBase::DumpOnTimeoutIfNeeded+0x261
sqlmin!LatchBase::PrintWarning+0x43d
sqlmin!LatchBase::Suspend+0xb95
sqlmin!LatchBase::AcquireInternal+0x415
sqlmin!FCB::GetAutoLatch+0x3b
sqlmin!FileMgr::GetFileOperationFCB+0xe0
sqlmin!FGCB::GrowAFile+0x38a
sqlmin!MakeSpaceInFileGroup+0x4a
sqlmin!AllocationReq::AllocatePages+0x2597
sqlmin!AllocationReq::Allocate+0xf3
sqlmin!AllocateHoBtDataPage+0x5fd
sqlmin!IndexPageManager::AllocatePage+0x1b8
sqlmin!SplitPage+0xb81
sqlmin!IndexDataSetSession::InsertSmallRecord+0x1a1d
sqlmin!IndexDataSetSession::InsertRowInternal+0x2de1
sqlmin!DatasetSession::InsertRow+0x163
sqlmin!RowsetNewSS::InsertRow+0x26
sqlmin!CValRow::SetDataX+0x5b
0x00007ffd`36b320df


ADDING INFO FROM OWNING THREAD

0:060> ~*e .echo ThreadId:; ?? @$tid; r? @$t1 = ((ntdll!_NT_TIB *)@$teb)->StackLimit; r? @$t2 = ((ntdll!_NT_TIB *)@$teb)->StackBase; s -d @$t1 @$t2  0x0000000624023C28
ThreadId:
unsigned int 0x1238
ThreadId:
unsigned int 0x1160
ThreadId:
unsigned int 0x1a8c
ThreadId:
unsigned int 0x1a94
ThreadId:
unsigned int 0x1a98
ThreadId:
unsigned int 0x1a9c
ThreadId:
unsigned int 0x1aa0
ThreadId:
unsigned int 0x1aa4
ThreadId:
unsigned int 0x1aa8
ThreadId:
unsigned int 0x1aac
ThreadId:
unsigned int 0x1ab0
ThreadId:
unsigned int 0x1abc
ThreadId:
unsigned int 0x1ac0
ThreadId:
unsigned int 0x1ac4
ThreadId:
unsigned int 0x1ac8
ThreadId:
unsigned int 0x1acc
ThreadId:
unsigned int 0x1b08
ThreadId:
unsigned int 0x1b0c
ThreadId:
unsigned int 0x1848
ThreadId:
unsigned int 0x18dc
ThreadId:
unsigned int 0x18cc
ThreadId:
unsigned int 0x1964
ThreadId:
unsigned int 0x19c4
ThreadId:
unsigned int 0x19cc
ThreadId:
unsigned int 0x19dc
ThreadId:
unsigned int 0x19e0
ThreadId:
unsigned int 0x1a04
ThreadId:
unsigned int 0x63c
ThreadId:
unsigned int 0x1a5c
ThreadId:
unsigned int 0x19f0
ThreadId:
unsigned int 0x1a58
ThreadId:
unsigned int 0x1a4c
ThreadId:
unsigned int 0x1a60
ThreadId:
unsigned int 0x1a48
ThreadId:
unsigned int 0xd10
ThreadId:
unsigned int 0x18c8
ThreadId:
unsigned int 0x1d24
ThreadId:
unsigned int 0x1d34
ThreadId:
unsigned int 0x1d78
ThreadId:
unsigned int 0x1ddc
ThreadId:
unsigned int 0x15b8
ThreadId:
unsigned int 0x1510
ThreadId:
unsigned int 0x14f0
ThreadId:
unsigned int 0x3ec
ThreadId:
unsigned int 0x1f00
ThreadId:
unsigned int 0x1420
ThreadId:
unsigned int 0x910
ThreadId:
unsigned int 0xb4c
ThreadId:
unsigned int 0x8f8
ThreadId:
unsigned int 0x144c
ThreadId:
unsigned int 0x620
ThreadId:
unsigned int 0x708
ThreadId:
unsigned int 0xe48
ThreadId:
unsigned int 0x1498
ThreadId:
unsigned int 0x132c
ThreadId:
unsigned int 0xa58
ThreadId:
unsigned int 0x18a4
ThreadId:
unsigned int 0x3158
ThreadId:
unsigned int 0x3614
ThreadId:
unsigned int 0x34f0
ThreadId:
unsigned int 0x27b4
ThreadId:
unsigned int 0x3fe8
ThreadId:
unsigned int 0x43c0
ThreadId:
unsigned int 0x38f8
ThreadId:
unsigned int 0x42c0
ThreadId:
unsigned int 0x34f4
ThreadId:
unsigned int 0x43ec
ThreadId:
unsigned int 0x42f8
ThreadId:
unsigned int 0x23ac
ThreadId:
unsigned int 0x3894
ThreadId:
unsigned int 0x3ca0
ThreadId:
unsigned int 0x3efc
ThreadId:
unsigned int 0x3944
ThreadId:
unsigned int 0x31cc
ThreadId:
unsigned int 0x397c
ThreadId:
unsigned int 0x3a64
ThreadId:
unsigned int 0x4080
ThreadId:
unsigned int 0x3cd4
ThreadId:
unsigned int 0x2c90
ThreadId:
unsigned int 0x3f54
ThreadId:
unsigned int 0x4390
ThreadId:
unsigned int 0x4398
ThreadId:
unsigned int 0x4040
ThreadId:
unsigned int 0x4540
ThreadId:
unsigned int 0x46f0
ThreadId:
unsigned int 0x45b8
ThreadId:
unsigned int 0x39f4
ThreadId:
unsigned int 0x462c
ThreadId:
unsigned int 0x57c
ThreadId:
unsigned int 0x38d8

comment 1

it is looking for exe not pdb try .chkimg on sqlserver.exe try using symchk

Reply1

Why is it looking for an .exe file

  comment 2

from the output you posted sqlserver.pdb has been loaded properly using partial information provided by debugger the paths all point to .exe extension not .pdb extension symsrv often downloads the official binary also along with pdb if it deems that the app might be corrupted hooked inmemory patched whatever the command .chkimg i posted in earlier comment will also download an official binary if it exists for comparing it with in memory image

comment 3

this is especially true when debugging a dump it might be possible that the sqlserver.exe you might have installed in your pc is different from the sqlserver.exe from which the dump was created

comment 4,5

first do lm and see if your pdb is loaded then run !analyze -v to get a preliminary understand then worry about copying exe if it was a single exe problem i would have said go ahead copy it but sql might have a ton of dependencis

lm is windbg command to load modules it normally outputs if the symbols have been properly or not and also dusplays the path of loaded pdb

the stack text you posted is the stack of the thread that wrote the minidump it says that it wrote a minidump because some timeout was exceded

sqllang!CImageHelper::DoMiniDump+0x426
sqllang!stackTrace+0x9d5
sqlmin!LatchBase::DumpOnTimeoutIfNeeded+0x261

you have >= 60 threads you may need to find the thread that owns the latch and find out why it is owning the latch for such a long time so that other thread quit waiting and write minidumps

I just glanced through the linked article in your query it shows you how to scrap a dword in all threads that are running and look at that threads stack follow it

basically all it does is finds the stack base and stack top of each thread in the process and searches for a dword that is the owning thread between these area

changes to that thread and dumps the stack