How to detect Python memory leaks
We are trying to tune some parameters in Python on Windows 10. We have many parameters and the range is wide so we are trying to calculate around 1000000 possibilities at least. We are using multiprocessing.
When we run the code, we get some undescriptive errors at some time and the IDE is dead. The errors are mostly Runtime errors. We have to restart the pc in order for the IDE to run again.
We are trying to detect this memory leak. We tried pympler and this is the sample output:
completed process count: 25 / 64
types | # objects | total size
=========================================================== | =========== | ============
<class 'pandas.core.series.Series | 6 | 16.50 MB
<class 'numpy.float64 | 286312 | 8.74 MB
<class 'str | 108625 | 7.95 MB
<class 'list | 8640 | 7.84 MB
<class 'float | 100209 | 2.29 MB
<class 'int | 1879 | 51.38 KB
<class 'dict | 16 | 2.64 KB
<class 'numpy.ndarray | 8 | 816 B
<class 'pandas.core.internals.managers.SingleBlockManager | 6 | 720 B
<class 'tuple | 9 | 560 B
<class 'pandas._libs.internals.BlockPlacement | 6 | 480 B
<class 'pandas.core.internals.blocks.FloatBlock | 5 | 400 B
<class 'slice | 6 | 384 B
<class 'code | 2 | 288 B
function (<lambda>) | 2 | 272 B
completed process count: 26 / 64
types | # objects | total size
=========================================================== | =========== | ============
<class 'pandas.core.series.Series | 6 | 16.50 MB
<class 'numpy.float64 | 286312 | 8.74 MB
<class 'str | 108625 | 7.95 MB
<class 'list | 8640 | 7.84 MB
<class 'float | 100209 | 2.29 MB
<class 'int | 1879 | 51.38 KB
<class 'dict | 16 | 2.64 KB
<class 'numpy.ndarray | 8 | 816 B
<class 'pandas.core.internals.managers.SingleBlockManager | 6 | 720 B
<class 'tuple | 9 | 560 B
<class 'pandas._libs.internals.BlockPlacement | 6 | 480 B
<class 'pandas.core.internals.blocks.FloatBlock | 5 | 400 B
<class 'slice | 6 | 384 B
<class 'code | 2 | 288 B
It seems the sizes and object counts do not change in time.
Any hints how to detect the culprit?
EDIT:
I have found a log file:
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 268435456 bytes for Failed to commit area from 0x0000000701600000 to 0x0000000711600000 of length 268435456.
# Possible reasons:
# The system is out of physical RAM or swap space
# The process is running with CompressedOops enabled, and the Java Heap may be blocking the growth of the native heap
# Possible solutions:
# Reduce memory load on the system
# Increase physical memory or swap space
# Check if swap backing store is full
# Decrease Java heap size (-Xmx/-Xms)
# Decrease number of Java threads
# Decrease Java thread stack sizes (-Xss)
# Set larger code cache with -XX:ReservedCodeCacheSize=
# This output file may be truncated or incomplete.
#
# Out of Memory Error (c:/BuildAgent/work/dd4f663211fab7c/src/hotspot/os/windows/os_windows.cpp:3272), pid=12164, tid=1876
#
# JRE version: (11.0.2+9) (build )
# Java VM: OpenJDK 64-Bit Server VM (11.0.2+9-b159.34, mixed mode, aot, sharing, tiered, compressed oops, g1 gc, windows-amd64)
# No core dump will be written. Minidumps are not enabled by default on client versions of Windows
#
--------------- S U M M A R Y ------------
Command Line: org.jetbrains.git4idea.http.GitAskPassApp Username for 'https://gitlab.com':
Host: Intel(R) Core(TM) i7-4710HQ CPU @ 2.50GHz, 8 cores, 15G, Windows 10 , 64 bit Build 17134 (10.0.17134.706)
Time: Fri May 3 09:19:13 2019 Turkey Standard Time elapsed time: 0 seconds (0d 0h 0m 0s)
--------------- T H R E A D ---------------
Current thread (0x000002cd5e5dd000): JavaThread "Unknown thread" [_thread_in_vm, id=1876, stack(0x000000d062400000,0x000000d062500000)]
Stack: [0x000000d062400000,0x000000d062500000]
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [jvm.dll+0x5da82a]
V [jvm.dll+0x70b6c9]
V [jvm.dll+0x70cbe8]
V [jvm.dll+0x70d2b3]
V [jvm.dll+0x240f18]
V [jvm.dll+0x5d7fc4]
V [jvm.dll+0x5cd115]
V [jvm.dll+0x2fa5eb]
V [jvm.dll+0x2fa55a]
V [jvm.dll+0x2fa432]
V [jvm.dll+0x2ff316]
V [jvm.dll+0x347993]
V [jvm.dll+0x348096]
V [jvm.dll+0x347a93]
V [jvm.dll+0x2d5588]
V [jvm.dll+0x2d66c7]
V [jvm.dll+0x6eaac7]
V [jvm.dll+0x6ec22c]
V [jvm.dll+0x3551c9]
V [jvm.dll+0x6d274e]
V [jvm.dll+0x3bd953]
V [jvm.dll+0x3bfcc1]
C [java.exe+0x3763]
C [java.exe+0x17a0c]
C [KERNEL32.DLL+0x14034]
C [ntdll.dll+0x73691]
--------------- P R O C E S S ---------------
Threads class SMR info:
_java_thread_list=0x000002cd5c2518b0, length=0, elements={
}
Java Threads: ( => current thread )
Other Threads:
0x000002cd5e5f3800 GCTaskThread "GC Thread#0" [stack: 0x000000d062500000,0x000000d062600000] [id=7888]
0x000002cd5e653000 ConcurrentGCThread "G1 Main Marker" [stack: 0x000000d062600000,0x000000d062700000] [id=720]
0x000002cd5e654800 ConcurrentGCThread "G1 Conc#0" [stack: 0x000000d062700000,0x000000d062800000] [id=2832]
[error occurred during error reporting (printing all threads), id 0xc0000005, EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x00007ffd9d305687]
VM state:not at safepoint (not fully initialized)
VM Mutex/Monitor currently owned by a thread: ([mutex/lock_event])
[0x000002cd5e5d97c0] Heap_lock - owner thread: 0x000002cd5e5dd000
Heap address: 0x0000000701600000, size: 4074 MB, Compressed Oops mode: Non-zero based: 0x0000000701600000
Narrow klass base: 0x0000000000000000, Narrow klass shift: 0
GC Heap History (0 events):
No events
Deoptimization events (0 events):
No events
Classes redefined (0 events):
No events
Internal exceptions (0 events):
No events
Events (0 events):
No events
Dynamic libraries:
0x00007ff77b370000 - 0x00007ff77b3b7000 C:\Program Files\JetBrains\PyCharm 2019.1\jre64\bin\java.exe
0x00007ffdd37a0000 - 0x00007ffdd3981000 C:\WINDOWS\SYSTEM32\ntdll.dll
0x00007ffdd3050000 - 0x00007ffdd3101000 C:\WINDOWS\System32\KERNEL32.DLL
0x00007ffdd01d0000 - 0x00007ffdd0443000 C:\WINDOWS\System32\KERNELBASE.dll
0x00007ffdd2c80000 - 0x00007ffdd2e10000 C:\WINDOWS\System32\USER32.dll
0x00007ffdd0450000 - 0x00007ffdd0470000 C:\WINDOWS\System32\win32u.dll
0x00007ffdd2670000 - 0x00007ffdd2698000 C:\WINDOWS\System32\GDI32.dll
0x00007ffdcff50000 - 0x00007ffdd00e2000 C:\WINDOWS\System32\gdi32full.dll
0x00007ffdcfeb0000 - 0x00007ffdcff4f000 C:\WINDOWS\System32\msvcp_win.dll
0x00007ffdcfbc0000 - 0x00007ffdcfcb8000 C:\WINDOWS\System32\ucrtbase.dll
0x00007ffdc01f0000 - 0x00007ffdc0459000 C:\WINDOWS\WinSxS\amd64_microsoft.windows.common-controls_6595b64144ccf1df_6.0.17134.706_none_fb43a2cb30647007\COMCTL32.dll
0x00007ffdd2e10000 - 0x00007ffdd2eae000 C:\WINDOWS\System32\msvcrt.dll
0x00007ffdd2930000 - 0x00007ffdd2c52000 C:\WINDOWS\System32\combase.dll
0x00007ffdd2f20000 - 0x00007ffdd3044000 C:\WINDOWS\System32\RPCRT4.dll
0x00007ffdd0150000 - 0x00007ffdd01ca000 C:\WINDOWS\System32\bcryptPrimitives.dll
0x00007ffdd3110000 - 0x00007ffdd313d000 C:\WINDOWS\System32\IMM32.DLL
0x0000000000400000 - 0x00000000005bb000 C:\WINDOWS\SysWOW64\winselfprotect.dll
0x00007ffdd3480000 - 0x00007ffdd3542000 C:\WINDOWS\System32\oleaut32.dll
0x00007ffdd3220000 - 0x00007ffdd32c1000 C:\WINDOWS\System32\advapi32.dll
0x00007ffdce810000 - 0x00007ffdce81a000 C:\WINDOWS\SYSTEM32\version.dll
0x00007ffdd26b0000 - 0x00007ffdd270b000 C:\WINDOWS\System32\sechost.dll
0x00007ffdce7f0000 - 0x00007ffdce807000 C:\WINDOWS\SYSTEM32\netapi32.dll
0x00007ffdcf1b0000 - 0x00007ffdcf1be000 C:\WINDOWS\SYSTEM32\NETUTILS.DLL
0x00007ffdba980000 - 0x00007ffdba997000 C:\Program Files\JetBrains\PyCharm 2019.1\jre64\bin\vcruntime140.dll
0x00007ffdbf1f0000 - 0x00007ffdbf28d000 C:\Program Files\JetBrains\PyCharm 2019.1\jre64\bin\msvcp140.dll
0x00007ffd9d020000 - 0x00007ffd9dad6000 C:\Program Files\JetBrains\PyCharm 2019.1\jre64\bin\server\jvm.dll
0x00007ffdd26a0000 - 0x00007ffdd26a8000 C:\WINDOWS\System32\PSAPI.DLL
0x00007ffdcb0d0000 - 0x00007ffdcb0d9000 C:\WINDOWS\SYSTEM32\WSOCK32.dll
0x00007ffdd2eb0000 - 0x00007ffdd2f1c000 C:\WINDOWS\System32\WS2_32.dll
0x00007ffdcc900000 - 0x00007ffdcc923000 C:\WINDOWS\SYSTEM32\WINMM.dll
0x00007ffdcc8d0000 - 0x00007ffdcc8fa000 C:\WINDOWS\SYSTEM32\WINMMBASE.dll
0x00007ffdcfb70000 - 0x00007ffdcfbb9000 C:\WINDOWS\System32\cfgmgr32.dll
0x00007ffdcfb50000 - 0x00007ffdcfb61000 C:\WINDOWS\System32\kernel.appcore.dll
0x00007ffdb96e0000 - 0x00007ffdb96f1000 C:\Program Files\JetBrains\PyCharm 2019.1\jre64\bin\verify.dll
0x00007ffdbde40000 - 0x00007ffdbe009000 C:\WINDOWS\SYSTEM32\DBGHELP.DLL
0x00007ffdbc9a0000 - 0x00007ffdbc9c9000 C:\WINDOWS\SYSTEM32\dbgcore.DLL
0x00007ffdb8df0000 - 0x00007ffdb8e19000 C:\Program Files\JetBrains\PyCharm 2019.1\jre64\bin\java.dll
0x00007ffdb8dd0000 - 0x00007ffdb8de7000 C:\Program Files\JetBrains\PyCharm 2019.1\jre64\bin\zip.dll
0x00007ffdc9fa0000 - 0x00007ffdc9fab000 C:\Program Files\JetBrains\PyCharm 2019.1\jre64\bin\jimage.dll
dbghelp: loaded successfully - version: 4.0.5 - missing functions: none
symbol engine: initialized successfully - sym options: 0x614 - pdb path: .;C:\Program Files\JetBrains\PyCharm 2019.1\jre64\bin;C:\WINDOWS\SYSTEM32;C:\WINDOWS\WinSxS\amd64_microsoft.windows.common-controls_6595b64144ccf1df_6.0.17134.706_none_fb43a2cb30647007;C:\WINDOWS\SysWOW64;C:\Program Files\JetBrains\PyCharm 2019.1\jre64\bin\server
VM Arguments:
java_command: org.jetbrains.git4idea.http.GitAskPassApp Username for 'https://gitlab.com':
java_class_path (initial): C:/Program Files/JetBrains/PyCharm 2019.1/plugins/git4idea/lib/git4idea-rt.jar;C:/Program Files/JetBrains/PyCharm 2019.1/lib/xmlrpc-2.0.1.jar;C:/Program Files/JetBrains/PyCharm 2019.1/lib/commons-codec-1.10.jar;C:/Program Files/JetBrains/PyCharm 2019.1/lib/util.jar
Launcher Type: SUN_STANDARD
[Global flags]
intx CICompilerCount = 4 {product} {ergonomic}
uint ConcGCThreads = 2 {product} {ergonomic}
uint G1ConcRefinementThreads = 8 {product} {ergonomic}
size_t G1HeapRegionSize = 1048576 {product} {ergonomic}
uintx GCDrainStackTargetSize = 64 {product} {ergonomic}
size_t InitialHeapSize = 268435456 {product} {ergonomic}
size_t MarkStackSize = 4194304 {product} {ergonomic}
size_t MaxHeapSize = 4271898624 {product} {ergonomic}
size_t MinHeapDeltaBytes = 1048576 {product} {ergonomic}
uintx NonNMethodCodeHeapSize = 5835340 {pd product} {ergonomic}
uintx NonProfiledCodeHeapSize = 122911450 {pd product} {ergonomic}
uintx ProfiledCodeHeapSize = 122911450 {pd product} {ergonomic}
uintx ReservedCodeCacheSize = 251658240 {pd product} {ergonomic}
bool SegmentedCodeCache = true {product} {ergonomic}
bool UseCompressedClassPointers = true {lp64_product} {ergonomic}
bool UseCompressedOops = true {lp64_product} {ergonomic}
bool UseG1GC = true {product} {ergonomic}
bool UseLargePagesIndividualAllocation = false {pd product} {ergonomic}
Logging:
Log output configuration:
#0: stdout all=warning uptime,level,tags
#1: stderr all=off uptime,level,tags
Environment Variables:
JAVA_HOME=C:\Program Files\Java\jdk1.8.0_171
PATH=C:/Program Files/Git/mingw64/libexec/git-core;C:/Program Files/Git/mingw64/libexec/git-core;C:\Program Files\Git\mingw64\bin;C:\Program Files\Git\usr\bin;C:\Users\ismetb\bin;C:\Program Files (x86)\Common Files\Oracle\Java\javapath;C:\ProgramData\Oracle\Java\javapath;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\Program Files\Java\jdk1.8.0_171\bin;C:\Program Files\Calibre2\;C:\WINDOWS\System32\OpenSSH\;C:\Program Files (x86)\Gpg4win\..\GnuPG\bin;C:\Android;C:\Windows\System32;C:\Program Files (x86)\EasyFrom Inc\;C:\Program Files\Git\cmd;C:\MATRIKS;C:\MATRIKS\System32\;C:\spark\bin;C:\Program Files\TortoiseGit\bin;C:\Users\ismetb\Anaconda3;C:\Users\ismetb\Anaconda3\Scripts;C:\Users\ismetb\AppData\Local\Microsoft\WindowsApps;C:\Program Files\JetBrains\PyCharm 2018.3.1\bin;C:\Program Files\JetBrains\IntelliJ IDEA 2018.3.5\bin;C:\Program Files\JetBrains\PyCharm 2018.3.4\bin;C:\Users\ismetb\AppData\Local\rodeo\app-2.5.2\bin
USERNAME=ismetb
DISPLAY=:0.0
OS=Windows_NT
PROCESSOR_IDENTIFIER=Intel64 Family 6 Model 60 Stepping 3, GenuineIntel
--------------- S Y S T E M ---------------
OS: Windows 10 , 64 bit Build 17134 (10.0.17134.706)
CPU:total 8 (initial active 8) (4 cores per cpu, 2 threads per core) family 6 model 60 stepping 3, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1, sse4.2, popcnt, avx, avx2, aes, clmul, erms, lzcnt, ht, tsc, tscinvbit, bmi1, bmi2, fma
Memory: 4k page, system-wide physical 16294M (478M free)
TotalPageFile size 45926M (AvailPageFile size 36M)
current process WorkingSet (physical memory assigned to process): 10M, peak: 10M
current process commit charge ("private bytes"): 62M, peak: 318M
vm_info: OpenJDK 64-Bit Server VM (11.0.2+9-b159.34) for windows-amd64 JRE (11.0.2+9-b159.34), built on Mar 26 2019 11:55:29 by "" with MS VC++ 14.0 (VS2015)
END.
iso_9001_0 Answers
Nobody has answered this question yet.
User contributions licensed under CC BY-SA 3.0