What is the proper way to investigate memory leaks on .net core linux kubernetes container

Question

What is the proper way to investigate memory leaks on .net core linux kubernetes container

I have the .net core app running on linux docker container, and while taking dumps (core 2.2 or 3.0) I can't open them in the PerfView,

taking dumps according to this instruction: https://github.com/dotnet/diagnostics/blob/master/documentation/dotnet-dump-instructions.md

PerfView shows this error in the logs:

Creating heap dump C:\temp\dumps\dump\dump-1.gcdump from process dump C:\temp\dumps\dump\dump-1.dmp.
HeapDump Error: Microsoft.Diagnostics.Runtime.ClrDiagnosticsException: Could not load crash dump 'C:\temp\dumps\dump\dump-1.dmp', HRESULT: 0x80070057
   at Microsoft.Diagnostics.Runtime.DbgEngDataReader..ctor(String dumpFile)
   at Microsoft.Diagnostics.Runtime.DataTarget.LoadCrashDump(String fileName)
   at GCHeapDumper.InitializeClrRuntime(String processDumpFile, DataTarget& target, ClrRuntime& runtime)
   at GCHeapDumper.DumpHeapFromProcessDump(String processDumpFile)
   at Program.MainWorker(String[] args)

c#

linux

kubernetes

memory-leaks

dump

asked on Stack Overflow Oct 21, 2019 by

sunseeker • edited Oct 21, 2019 by

N3R4ZZuRR0

3 Answers

The dump file is created inside the container and therefore not directly accessible from your machine. (If you are running Windows and Docker for Windows there is even a virtual machine inbetween.)

What you need to do is to copy the dumb file from the container to your host and open it afterwards. This can be achieved using docker cp command, for example: docker cp <container name>:<path in container>dump-1.gcdump C:\temp\dumps\dump\dump-1.gcdump

answered on Stack Overflow Oct 21, 2019 by

Thomas

I believe you need to use Linux debugger to open Linux dumps. Afaik PerfView supports only Windows dump.

CoreClr team provides SOS debugger extension that can be utilized from lldb debugger. https://github.com/dotnet/coreclr/blob/master/Documentation/building/debugging-instructions.md

answered on Stack Overflow Oct 22, 2019 by

Vladimir Dronov

Dumps on .NET core are not cross-platform compatible due to cross-platform DAC (For more info, see https://github.com/dotnet/runtime/blob/master/docs/design/coreclr/botr/dac-notes.md). There have been discussions/plans on supporting this but it hasn't happened yet.

You can use dotnet-gcdump tool and it should be cross-platform compatible. Here is a doc on how to use it: https://github.com/dotnet/diagnostics/blob/master/documentation/dotnet-gcdump-instructions.md

A dump taken from dotnet-gcdump can be viewed on PerfView.

answered on Stack Overflow Mar 5, 2020 by

Sung Yoon Whang

User contributions licensed under CC BY-SA 3.0