I've got the following JVM crash:
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x00007f72017cf04d, pid=15602, tid=0x00007f6ec04c8700
#
# JRE version: Java(TM) SE Runtime Environment (8.0_121-b13) (build 1.8.0_121-b13)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.121-b13 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# J 7487 C2 io.druid.segment.incremental.IncrementalIndexAdapter$1$1.apply(Ljava/lang/Object;)Ljava/lang/Object; (9 bytes) @ 0x00007f72017cf04d [0x00007f72017ce0e0+0xf6d]
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# If you would like to submit a bug report, please visit:
# http://bugreport.java.com/bugreport/crash.jsp
#
--------------- T H R E A D ---------------
Current thread (0x00007f6e940a9800): JavaThread "mmx_metrics_kafka-incremental-persist" daemon [_thread_in_Java, id=17025, stack(0x00007f6ec03c8000,0x00007f6ec04c9000)]
siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr: 0x0000000000000023
Registers:
RAX=0x0000000000000000, RBX=0x0000000000043dc4, RCX=0x0000000000043dc0, RDX=0x0000000000000026
RSP=0x00007f6ec04c6c10, RBP=0x0000000000000000, RSI=0x0000000000000026, RDI=0x00000000d24b15b0
R8 =0x0000000000000017, R9 =0x00000000d24b15b0, R10=0x0000000000000017, R11=0x0000000000043dc4
R12=0x0000000000000000, R13=0x00000000d7800000, R14=0x00000000d6c3e008, R15=0x00007f6e940a9800
RIP=0x00007f72017cf04d, EFLAGS=0x0000000000010206, CSGSFS=0x0000000000000033, ERR=0x0000000000000004
TRAPNO=0x000000000000000e
Top of Stack: (sp=0x00007f6ec04c6c10)
0x00007f6ec04c6c10: 00000000d7800000 00043e4000000002
0x00007f6ec04c6c20: 00043dc000000001 0000000000043dc0
0x00007f6ec04c6c30: 00000000d24b1598 00000000d24b1388
0x00007f6ec04c6c40: 00007f71fe4dd000 00000000a20ef0e0
0x00007f6ec04c6c50: 0000000100010c90 0000000100011338
0x00007f6ec04c6c60: 00000000d24b1430 0000000000000025
0x00007f6ec04c6c70: 0000000095f953c8 0000000000000000
0x00007f6ec04c6c80: 00000000d24b1570 0000000100000000
0x00007f6ec04c6c90: 00000000d6c3e198 00007f720002eabb
0x00007f6ec04c6ca0: 00000000d6c3e760 00000000d6c4e7f0
0x00007f6ec04c6cb0: 00000000d6c3dff0 00007f7201a821a4
0x00007f6ec04c6cc0: 00000000d78000e8 00007f7201746b60
0x00007f6ec04c6cd0: 00000000d6c3dff0 00007f7201794fc0
0x00007f6ec04c6ce0: 00000000d33ffd90 00007f72006166c5
0x00007f6ec04c6cf0: 00000000d78000e8 00007f72027caa38
0x00007f6ec04c6d00: 0000015e3225511b 0000000100000026
0x00007f6ec04c6d10: d6c43ea0200f9e06 00007f71fe4dd000
0x00007f6ec04c6d20: 00000000d7800130 0000015e3225511a
0x00007f6ec04c6d30: ffffffff00000000 0000000100011338
0x00007f6ec04c6d40: 00000000d7800140 0000000100011570
0x00007f6ec04c6d50: 00000000d7800158 00000001007d6808
0x00007f6ec04c6d60: 00000001007d7448 d6c4ee400002eabc
0x00007f6ec04c6d70: 00000000d7800178 00000000d7800188
0x00007f6ec04c6d80: 00000000d24b1108 00000000d6c3e148
0x00007f6ec04c6d90: 00000000d6c43e80 0000000000000026
0x00007f6ec04c6da0: 0000000000000000 00000000d7800178
0x00007f6ec04c6db0: 00000000d6546e88 00007f721526a4c3
0x00007f6ec04c6dc0: 0000000000000000 00000000ddf97360
0x00007f6ec04c6dd0: 00007f6ec04c72d8 00007f72022e6584
0x00007f6ec04c6de0: 00000000d7800100 00000000d7800118
0x00007f6ec04c6df0: 0000000000000002 00007f72014db91c
0x00007f6ec04c6e00: 00007f6e00000002 00000000de1665e0
Instructions: (pc=0x00007f72017cf04d)
0x00007f72017cf02d: 01 de fe e8 fb a9 a9 13 85 ed 0f 8c 14 09 00 00
0x00007f72017cf03d: 45 8b 50 18 45 85 d2 0f 84 a7 00 00 00 4d 8b c2
0x00007f72017cf04d: 45 8b 58 0c 85 05 a9 cf d8 14 44 3b db 74 12 45
0x00007f72017cf05d: 8b 58 18 45 85 db 0f 84 88 00 00 00 4d 8b c3 eb
Register to memory mapping:
RAX=0x0000000000000000 is an unknown value
RBX=0x0000000000043dc4 is an unknown value
RCX=0x0000000000043dc0 is an unknown value
RDX=0x0000000000000026 is an unknown value
RSP=0x00007f6ec04c6c10 is pointing into the stack for thread: 0x00007f6e940a9800
RBP=0x0000000000000000 is an unknown value
RSI=0x0000000000000026 is an unknown value
RDI=0x00000000d24b15b0 is an oop
java.lang.Integer
- klass: 'java/lang/Integer'
R8 =0x0000000000000017 is an unknown value
R9 =0x00000000d24b15b0 is an oop
java.lang.Integer
- klass: 'java/lang/Integer'
R10=0x0000000000000017 is an unknown value
R11=0x0000000000043dc4 is an unknown value
R12=0x0000000000000000 is an unknown value
R13=0x00000000d7800000 is an oop
io.druid.segment.incremental.IncrementalIndexAdapter$1$1
- klass: 'io/druid/segment/incremental/IncrementalIndexAdapter$1$1'
R14=0x00000000d6c3e008 is an oop
java.util.concurrent.ConcurrentSkipListMap$KeyIterator
- klass: 'java/util/concurrent/ConcurrentSkipListMap$KeyIterator'
R15=0x00007f6e940a9800 is a thread
Stack: [0x00007f6ec03c8000,0x00007f6ec04c9000], sp=0x00007f6ec04c6c10, free space=1019k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
J 7487 C2 io.druid.segment.incremental.IncrementalIndexAdapter$1$1.apply(Ljava/lang/Object;)Ljava/lang/Object; (9 bytes) @ 0x00007f72017cf04d [0x00007f72017ce0e0+0xf6d]
J 7477 C2 com.google.common.collect.Iterators$8.transform(Ljava/lang/Object;)Ljava/lang/Object; (11 bytes) @ 0x00007f7201a821a4 [0x00007f7201a82160+0x44]
J 7469 C2 com.google.common.collect.TransformedIterator.next()Ljava/lang/Object; (14 bytes) @ 0x00007f7201794fc0 [0x00007f7201794f60+0x60]
J 12883 C2 io.druid.segment.IndexMergerV9.mergeIndexesAndWriteColumns(Ljava/util/List;Lio/druid/segment/ProgressIndicator;Ljava/lang/Iterable;Lio/druid/segment/LongColumnSerializer;Ljava/util/ArrayList;Ljava/util/List;Ljava/util/List;)V (502 bytes) @ 0x00007f72027caa38 [0x00007f72027c9c20+0xe18]
J 11286 C1 io.druid.segment.IndexMergerV9.makeIndexFiles(Ljava/util/List;[Lio/druid/query/aggregation/AggregatorFactory;Ljava/io/File;Lio/druid/segment/ProgressIndicator;Ljava/util/List;Ljava/util/List;Lcom/google/common/base/Function;Lio/druid/segment/IndexSpec;)Ljava/io/File; (787 bytes) @ 0x00007f72022e6584 [0x00007f72022e2480+0x4104]
J 12543 C1 io.druid.segment.IndexMergerV9.merge(Ljava/util/List;Z[Lio/druid/query/aggregation/AggregatorFactory;Ljava/io/File;Lio/druid/segment/IndexSpec;Lio/druid/segment/ProgressIndicator;)Ljava/io/File; (286 bytes) @ 0x00007f7200b0537c [0x00007f7200b03980+0x19fc]
j io.druid.segment.IndexMergerV9.persist(Lio/druid/segment/incremental/IncrementalIndex;Lorg/joda/time/Interval;Ljava/io/File;Lio/druid/segment/IndexSpec;Lio/druid/segment/ProgressIndicator;)Ljava/io/File;+169
J 12613 C1 io.druid.segment.realtime.plumber.RealtimePlumber.persistHydrant(Lio/druid/segment/realtime/FireHydrant;Lio/druid/segment/indexing/DataSchema;Lorg/joda/time/Interval;Ljava/util/Map;)I (240 bytes) @ 0x00007f7202674c34 [0x00007f7202672da0+0x1e94]
J 12715 C1 io.druid.segment.realtime.plumber.RealtimePlumber$1.doRun()V (219 bytes) @ 0x00007f72026e14a4 [0x00007f72026e0aa0+0xa04]
J 12695 C1 io.druid.common.guava.ThreadRenamingRunnable.run()V (38 bytes) @ 0x00007f72026c55e4 [0x00007f72026c5380+0x264]
j java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V+95
j java.util.concurrent.ThreadPoolExecutor$Worker.run()V+5
j java.lang.Thread.run()V+11
v ~StubRoutines::call_stub
V [libjvm.so+0x690dd6] JavaCalls::call_helper(JavaValue*, methodHandle*, JavaCallArguments*, Thread*)+0x1056
V [libjvm.so+0x6912e1] JavaCalls::call_virtual(JavaValue*, KlassHandle, Symbol*, Symbol*, JavaCallArguments*, Thread*)+0x321
V [libjvm.so+0x691787] JavaCalls::call_virtual(JavaValue*, Handle, KlassHandle, Symbol*, Symbol*, Thread*)+0x47
V [libjvm.so+0x72cb00] thread_entry(JavaThread*, Thread*)+0xa0
V [libjvm.so+0xa75543] JavaThread::thread_main_inner()+0x103
V [libjvm.so+0xa7568c] JavaThread::run()+0x11c
V [libjvm.so+0x926268] java_start(Thread*)+0x108
C [libpthread.so.0+0x8184] start_thread+0xc4
[...]
Internal exceptions (10 events):
Event: 5366.762 Thread 0x00007f6d70005800 Implicit null exception at 0x00007f72019944e8 to 0x00007f72019945b1
Event: 5366.762 Thread 0x00007f6d70005800 Implicit null exception at 0x00007f72019aa40c to 0x00007f72019aa441
Event: 5405.905 Thread 0x00007f6e94060000 Implicit null exception at 0x00007f7201d9b4f9 to 0x00007f7201d9bb4d
Event: 5562.987 Thread 0x00007f6d70005800 Exception (0x00000000bf740c90) thrown at [/HUDSON3/workspace/8-2-build-linux-amd64/jdk8u121/8372/hotspot/src/share/vm/prims/jni.cpp, line 735]
Event: 5765.900 Thread 0x00007f6d70005800 Implicit null exception at 0x00007f7201defd8d to 0x00007f7201df3415
Event: 9144.627 Thread 0x00007f6d70005800 Exception (0x00000000bdd904f8) thrown at [/HUDSON3/workspace/8-2-build-linux-amd64/jdk8u121/8372/hotspot/src/share/vm/prims/jni.cpp, line 735]
Event: 9959.905 Thread 0x00007f7210d44000 Implicit null exception at 0x00007f72023d0406 to 0x00007f72023d0431
Event: 12747.633 Thread 0x00007f6d70005800 Exception (0x00000000e142cbb8) thrown at [/HUDSON3/workspace/8-2-build-linux-amd64/jdk8u121/8372/hotspot/src/share/vm/prims/jni.cpp, line 735]
Event: 16357.638 Thread 0x00007f6d70005800 Exception (0x00000000dece28a0) thrown at [/HUDSON3/workspace/8-2-build-linux-amd64/jdk8u121/8372/hotspot/src/share/vm/prims/jni.cpp, line 735]
Event: 18427.549 Thread 0x00007f6e940a9800 Implicit null exception at 0x00007f72017cf04d to 0x0000000000000000
[...]
Memory: 4k page, physical 65974116k(28165236k free), swap 0k(0k free)
vm_info: Java HotSpot(TM) 64-Bit Server VM (25.121-b13) for linux-amd64 JRE (1.8.0_121-b13), built on Dec 12 2016 16:36:53 by "java_re" with gcc 4.3.0 20080428 (Red Hat 4.3.0-8)
time: Wed Aug 30 07:57:40 2017
elapsed time: 18427 seconds (0d 5h 7m 7s)
Disassembly of the code around the crash gives:
add esi,ebx
(bad)
call 0x13a9aa03
test ebp,ebp
jl 0x00000924
mov r10d,DWORD PTR [r8+0x18]
test r10d,r10d
je 0x000000c4 ; NPE check
mov r8,r10
mov r11d,DWORD PTR [r8+0xc] ; Point of crash; r8 value = 0x17
test DWORD PTR [rip+0x14d8cfa9],eax ; safepoint poll
cmp r11d,ebx
je 0x00000041
mov r11d,DWORD PTR [r8+0x18]
test r11d,r11d
je 0x000000c4 ; NPE check
mov r8,r11
.byte 0xeb
It's seems that at the point of crash R8 register should be a pointer to some Java object, but it contains 0x17 value instead.
The compiled method at which the crash has occurred is here: https://github.com/metamx/druid/blob/druid-0.10.1-mmx131/processing/src/main/java/io/druid/segment/incremental/IncrementalIndexAdapter.java#L188-L218
User contributions licensed under CC BY-SA 3.0