Strange JVM crash in compiled code

0

I've got the following JVM crash:


#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007f72017cf04d, pid=15602, tid=0x00007f6ec04c8700
#
# JRE version: Java(TM) SE Runtime Environment (8.0_121-b13) (build 1.8.0_121-b13)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.121-b13 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# J 7487 C2 io.druid.segment.incremental.IncrementalIndexAdapter$1$1.apply(Ljava/lang/Object;)Ljava/lang/Object; (9 bytes) @ 0x00007f72017cf04d [0x00007f72017ce0e0+0xf6d]
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
#

---------------  T H R E A D  ---------------

Current thread (0x00007f6e940a9800):  JavaThread "mmx_metrics_kafka-incremental-persist" daemon [_thread_in_Java, id=17025, stack(0x00007f6ec03c8000,0x00007f6ec04c9000)]

siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr: 0x0000000000000023

Registers:
RAX=0x0000000000000000, RBX=0x0000000000043dc4, RCX=0x0000000000043dc0, RDX=0x0000000000000026
RSP=0x00007f6ec04c6c10, RBP=0x0000000000000000, RSI=0x0000000000000026, RDI=0x00000000d24b15b0
R8 =0x0000000000000017, R9 =0x00000000d24b15b0, R10=0x0000000000000017, R11=0x0000000000043dc4
R12=0x0000000000000000, R13=0x00000000d7800000, R14=0x00000000d6c3e008, R15=0x00007f6e940a9800
RIP=0x00007f72017cf04d, EFLAGS=0x0000000000010206, CSGSFS=0x0000000000000033, ERR=0x0000000000000004
  TRAPNO=0x000000000000000e

Top of Stack: (sp=0x00007f6ec04c6c10)
0x00007f6ec04c6c10:   00000000d7800000 00043e4000000002
0x00007f6ec04c6c20:   00043dc000000001 0000000000043dc0
0x00007f6ec04c6c30:   00000000d24b1598 00000000d24b1388
0x00007f6ec04c6c40:   00007f71fe4dd000 00000000a20ef0e0
0x00007f6ec04c6c50:   0000000100010c90 0000000100011338
0x00007f6ec04c6c60:   00000000d24b1430 0000000000000025
0x00007f6ec04c6c70:   0000000095f953c8 0000000000000000
0x00007f6ec04c6c80:   00000000d24b1570 0000000100000000
0x00007f6ec04c6c90:   00000000d6c3e198 00007f720002eabb
0x00007f6ec04c6ca0:   00000000d6c3e760 00000000d6c4e7f0
0x00007f6ec04c6cb0:   00000000d6c3dff0 00007f7201a821a4
0x00007f6ec04c6cc0:   00000000d78000e8 00007f7201746b60
0x00007f6ec04c6cd0:   00000000d6c3dff0 00007f7201794fc0
0x00007f6ec04c6ce0:   00000000d33ffd90 00007f72006166c5
0x00007f6ec04c6cf0:   00000000d78000e8 00007f72027caa38
0x00007f6ec04c6d00:   0000015e3225511b 0000000100000026
0x00007f6ec04c6d10:   d6c43ea0200f9e06 00007f71fe4dd000
0x00007f6ec04c6d20:   00000000d7800130 0000015e3225511a
0x00007f6ec04c6d30:   ffffffff00000000 0000000100011338
0x00007f6ec04c6d40:   00000000d7800140 0000000100011570
0x00007f6ec04c6d50:   00000000d7800158 00000001007d6808
0x00007f6ec04c6d60:   00000001007d7448 d6c4ee400002eabc
0x00007f6ec04c6d70:   00000000d7800178 00000000d7800188
0x00007f6ec04c6d80:   00000000d24b1108 00000000d6c3e148
0x00007f6ec04c6d90:   00000000d6c43e80 0000000000000026
0x00007f6ec04c6da0:   0000000000000000 00000000d7800178
0x00007f6ec04c6db0:   00000000d6546e88 00007f721526a4c3
0x00007f6ec04c6dc0:   0000000000000000 00000000ddf97360
0x00007f6ec04c6dd0:   00007f6ec04c72d8 00007f72022e6584
0x00007f6ec04c6de0:   00000000d7800100 00000000d7800118
0x00007f6ec04c6df0:   0000000000000002 00007f72014db91c
0x00007f6ec04c6e00:   00007f6e00000002 00000000de1665e0 

Instructions: (pc=0x00007f72017cf04d)
0x00007f72017cf02d:   01 de fe e8 fb a9 a9 13 85 ed 0f 8c 14 09 00 00
0x00007f72017cf03d:   45 8b 50 18 45 85 d2 0f 84 a7 00 00 00 4d 8b c2
0x00007f72017cf04d:   45 8b 58 0c 85 05 a9 cf d8 14 44 3b db 74 12 45
0x00007f72017cf05d:   8b 58 18 45 85 db 0f 84 88 00 00 00 4d 8b c3 eb 

Register to memory mapping:

RAX=0x0000000000000000 is an unknown value
RBX=0x0000000000043dc4 is an unknown value
RCX=0x0000000000043dc0 is an unknown value
RDX=0x0000000000000026 is an unknown value
RSP=0x00007f6ec04c6c10 is pointing into the stack for thread: 0x00007f6e940a9800
RBP=0x0000000000000000 is an unknown value
RSI=0x0000000000000026 is an unknown value
RDI=0x00000000d24b15b0 is an oop
java.lang.Integer 
 - klass: 'java/lang/Integer'
R8 =0x0000000000000017 is an unknown value
R9 =0x00000000d24b15b0 is an oop
java.lang.Integer 
 - klass: 'java/lang/Integer'
R10=0x0000000000000017 is an unknown value
R11=0x0000000000043dc4 is an unknown value
R12=0x0000000000000000 is an unknown value
R13=0x00000000d7800000 is an oop
io.druid.segment.incremental.IncrementalIndexAdapter$1$1 
 - klass: 'io/druid/segment/incremental/IncrementalIndexAdapter$1$1'
R14=0x00000000d6c3e008 is an oop
java.util.concurrent.ConcurrentSkipListMap$KeyIterator 
 - klass: 'java/util/concurrent/ConcurrentSkipListMap$KeyIterator'
R15=0x00007f6e940a9800 is a thread


Stack: [0x00007f6ec03c8000,0x00007f6ec04c9000],  sp=0x00007f6ec04c6c10,  free space=1019k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
J 7487 C2 io.druid.segment.incremental.IncrementalIndexAdapter$1$1.apply(Ljava/lang/Object;)Ljava/lang/Object; (9 bytes) @ 0x00007f72017cf04d [0x00007f72017ce0e0+0xf6d]
J 7477 C2 com.google.common.collect.Iterators$8.transform(Ljava/lang/Object;)Ljava/lang/Object; (11 bytes) @ 0x00007f7201a821a4 [0x00007f7201a82160+0x44]
J 7469 C2 com.google.common.collect.TransformedIterator.next()Ljava/lang/Object; (14 bytes) @ 0x00007f7201794fc0 [0x00007f7201794f60+0x60]
J 12883 C2 io.druid.segment.IndexMergerV9.mergeIndexesAndWriteColumns(Ljava/util/List;Lio/druid/segment/ProgressIndicator;Ljava/lang/Iterable;Lio/druid/segment/LongColumnSerializer;Ljava/util/ArrayList;Ljava/util/List;Ljava/util/List;)V (502 bytes) @ 0x00007f72027caa38 [0x00007f72027c9c20+0xe18]
J 11286 C1 io.druid.segment.IndexMergerV9.makeIndexFiles(Ljava/util/List;[Lio/druid/query/aggregation/AggregatorFactory;Ljava/io/File;Lio/druid/segment/ProgressIndicator;Ljava/util/List;Ljava/util/List;Lcom/google/common/base/Function;Lio/druid/segment/IndexSpec;)Ljava/io/File; (787 bytes) @ 0x00007f72022e6584 [0x00007f72022e2480+0x4104]
J 12543 C1 io.druid.segment.IndexMergerV9.merge(Ljava/util/List;Z[Lio/druid/query/aggregation/AggregatorFactory;Ljava/io/File;Lio/druid/segment/IndexSpec;Lio/druid/segment/ProgressIndicator;)Ljava/io/File; (286 bytes) @ 0x00007f7200b0537c [0x00007f7200b03980+0x19fc]
j  io.druid.segment.IndexMergerV9.persist(Lio/druid/segment/incremental/IncrementalIndex;Lorg/joda/time/Interval;Ljava/io/File;Lio/druid/segment/IndexSpec;Lio/druid/segment/ProgressIndicator;)Ljava/io/File;+169
J 12613 C1 io.druid.segment.realtime.plumber.RealtimePlumber.persistHydrant(Lio/druid/segment/realtime/FireHydrant;Lio/druid/segment/indexing/DataSchema;Lorg/joda/time/Interval;Ljava/util/Map;)I (240 bytes) @ 0x00007f7202674c34 [0x00007f7202672da0+0x1e94]
J 12715 C1 io.druid.segment.realtime.plumber.RealtimePlumber$1.doRun()V (219 bytes) @ 0x00007f72026e14a4 [0x00007f72026e0aa0+0xa04]
J 12695 C1 io.druid.common.guava.ThreadRenamingRunnable.run()V (38 bytes) @ 0x00007f72026c55e4 [0x00007f72026c5380+0x264]
j  java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V+95
j  java.util.concurrent.ThreadPoolExecutor$Worker.run()V+5
j  java.lang.Thread.run()V+11
v  ~StubRoutines::call_stub
V  [libjvm.so+0x690dd6]  JavaCalls::call_helper(JavaValue*, methodHandle*, JavaCallArguments*, Thread*)+0x1056
V  [libjvm.so+0x6912e1]  JavaCalls::call_virtual(JavaValue*, KlassHandle, Symbol*, Symbol*, JavaCallArguments*, Thread*)+0x321
V  [libjvm.so+0x691787]  JavaCalls::call_virtual(JavaValue*, Handle, KlassHandle, Symbol*, Symbol*, Thread*)+0x47
V  [libjvm.so+0x72cb00]  thread_entry(JavaThread*, Thread*)+0xa0
V  [libjvm.so+0xa75543]  JavaThread::thread_main_inner()+0x103
V  [libjvm.so+0xa7568c]  JavaThread::run()+0x11c
V  [libjvm.so+0x926268]  java_start(Thread*)+0x108
C  [libpthread.so.0+0x8184]  start_thread+0xc4

[...]

Internal exceptions (10 events):
Event: 5366.762 Thread 0x00007f6d70005800 Implicit null exception at 0x00007f72019944e8 to 0x00007f72019945b1
Event: 5366.762 Thread 0x00007f6d70005800 Implicit null exception at 0x00007f72019aa40c to 0x00007f72019aa441
Event: 5405.905 Thread 0x00007f6e94060000 Implicit null exception at 0x00007f7201d9b4f9 to 0x00007f7201d9bb4d
Event: 5562.987 Thread 0x00007f6d70005800 Exception  (0x00000000bf740c90) thrown at [/HUDSON3/workspace/8-2-build-linux-amd64/jdk8u121/8372/hotspot/src/share/vm/prims/jni.cpp, line 735]
Event: 5765.900 Thread 0x00007f6d70005800 Implicit null exception at 0x00007f7201defd8d to 0x00007f7201df3415
Event: 9144.627 Thread 0x00007f6d70005800 Exception  (0x00000000bdd904f8) thrown at [/HUDSON3/workspace/8-2-build-linux-amd64/jdk8u121/8372/hotspot/src/share/vm/prims/jni.cpp, line 735]
Event: 9959.905 Thread 0x00007f7210d44000 Implicit null exception at 0x00007f72023d0406 to 0x00007f72023d0431
Event: 12747.633 Thread 0x00007f6d70005800 Exception  (0x00000000e142cbb8) thrown at [/HUDSON3/workspace/8-2-build-linux-amd64/jdk8u121/8372/hotspot/src/share/vm/prims/jni.cpp, line 735]
Event: 16357.638 Thread 0x00007f6d70005800 Exception  (0x00000000dece28a0) thrown at [/HUDSON3/workspace/8-2-build-linux-amd64/jdk8u121/8372/hotspot/src/share/vm/prims/jni.cpp, line 735]
Event: 18427.549 Thread 0x00007f6e940a9800 Implicit null exception at 0x00007f72017cf04d to 0x0000000000000000

[...]

Memory: 4k page, physical 65974116k(28165236k free), swap 0k(0k free)

vm_info: Java HotSpot(TM) 64-Bit Server VM (25.121-b13) for linux-amd64 JRE (1.8.0_121-b13), built on Dec 12 2016 16:36:53 by "java_re" with gcc 4.3.0 20080428 (Red Hat 4.3.0-8)

time: Wed Aug 30 07:57:40 2017
elapsed time: 18427 seconds (0d 5h 7m 7s)

Disassembly of the code around the crash gives:


add    esi,ebx
(bad)
call   0x13a9aa03
test   ebp,ebp
jl     0x00000924
mov    r10d,DWORD PTR [r8+0x18]        
test   r10d,r10d
je     0x000000c4                      ; NPE check
mov    r8,r10
mov    r11d,DWORD PTR [r8+0xc]         ; Point of crash; r8 value = 0x17
test   DWORD PTR [rip+0x14d8cfa9],eax  ; safepoint poll
cmp    r11d,ebx
je     0x00000041
mov    r11d,DWORD PTR [r8+0x18]
test   r11d,r11d
je     0x000000c4                      ; NPE check
mov    r8,r11
.byte 0xeb

It's seems that at the point of crash R8 register should be a pointer to some Java object, but it contains 0x17 value instead.

The compiled method at which the crash has occurred is here: https://github.com/metamx/druid/blob/druid-0.10.1-mmx131/processing/src/main/java/io/druid/segment/incremental/IncrementalIndexAdapter.java#L188-L218

jvm-hotspot
jvm-crash
asked on Stack Overflow Aug 31, 2017 by leventov

0 Answers

Nobody has answered this question yet.


User contributions licensed under CC BY-SA 3.0