Java 9.0.1 OpenJdk crashed the JVM with problem with AdaptiveWeightedAverage

My application failed for a customer crashing the JVM

I have these details:

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0xb6b48b12, pid=31447, tid=31508
#
# JRE version: OpenJDK Runtime Environment (9.0+11) (build 9.0.1+11)
# Java VM: OpenJDK Client VM (9.0.1+11, mixed mode, serial gc, linux-)
# Problematic frame:
# V  [libjvm.so+0x1d9b12]  AdaptiveWeightedAverage::sample(float)+0x55
#
# No core dump will be written. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /mnt/system/config/Apps/SongKong/songkong/hs_err_pid31447.log
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
#

Does it look familar to anyone, I have search both https://bugs.java.com/bugdatabase/ and https://bugs.openjdk.java.net/secure/Dashboard.jspa without finding anything.

Here is the start of the hs_err_id log file

    #
    # A fatal error has been detected by the Java Runtime Environment:
    #
    #  SIGSEGV (0xb) at pc=0xb6aacb12, pid=13032, tid=13074
    #
    # JRE version: OpenJDK Runtime Environment (9.0+11) (build 9.0.1+11)
    # Java VM: OpenJDK Client VM (9.0.1+11, mixed mode, serial gc, linux-)
    # Problematic frame:
    # V  [libjvm.so+0x1d9b12]  AdaptiveWeightedAverage::sample(float)+0x55
    #
    # No core dump will be written. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
    #
    # If you would like to submit a bug report, please visit:
    #   http://bugreport.java.com/bugreport/crash.jsp
    #

    ---------------  S U M M A R Y ------------

    Command Line: -Duser.home=/mnt/disk1/share -Xms150m -Xmx900m -Dcom.mchange.v2.log.MLog=com.mchange.v2.log.jdk14logging.Jdk14MLog -Dorg.jboss.logging.provider=jdk -Djava.util.logging.config.class=com.jthink.songkong.logging.StandardLogging --add-opens=java.base/java.lang=ALL-UNNAMED lib/songkong-6.8.jar -r

    Host: Marvell PJ4Bv7 Processor rev 1 (v7l), 1 cores, 1G, Buildroot 2014.11-rc1
    Time: Fri Feb  7 01:50:35 2020 GMT elapsed time: 53621 seconds (0d 14h 53m 41s)

    ---------------  T H R E A D  ---------------

    Current thread (0xb6657d00):  VMThread "VM Thread" [stack: 0x7b7e9000,0x7b869000] [id=13074]

    Stack: [0x7b7e9000,0x7b869000],  sp=0x7b8677a0,  free space=505k
    Native frames: (J=compiled Java code, A=aot compiled Java code, j=interpreted, Vv=VM code, C=native code)
    V  [libjvm.so+0x1d9b12]  AdaptiveWeightedAverage::sample(float)+0x55

    [error occurred during error reporting (printing native stack), id 0xb]


    siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr: 0x0000000a

    Register to memory mapping:

      r0  = 0x00000000
    0x00000000 is an unknown value

      r1  = 0x00000100
    0x00000100 is an unknown value

      r2  = 0x000001fc
    0x000001fc is an unknown value

      r3  = 0x00000023
    0x00000023 is an unknown value

      r4  = 0x783bf970
    0x783bf970 is an unknown value

      r5  = 0x00000000
    0x00000000 is an unknown value

      r6  = 0x0e6a0000
    0x0e6a0000 is an unknown value

      r7  = 0x00000000
    0x00000000 is an unknown value

      r8  = 0x783bf940
    0x783bf940 is an unknown value

      r9  = 0x0e6a0000
    0x0e6a0000 is an unknown value

      r10 = 0x7b8679ec
    0x7b8679ec is an unknown value

      fp  = 0xb6657c38
    0xb6657c38 is an unknown value

      r12 = 0xfffffefe
    0xfffffefe is an unknown value

      sp  = 0x7b8677a0
    0x7b8677a0 is an unknown value

      lr  = 0xb6c62daf
    0xb6c62daf: <offset 0x0038fdaf> in /mnt/system/config/Apps/SongKong/songkong/jre/lib/client/libjvm.so at 0xb68d3000

      pc  = 0xb6aacb12
    0xb6aacb12: <offset 0x001d9b12> in /mnt/system/config/Apps/SongKong/songkong/jre/lib/client/libjvm.so at 0xb68d3000



    Registers:
      r0  = 0x00000000
      r1  = 0x00000100
      r2  = 0x000001fc
      r3  = 0x00000023
      r4  = 0x783bf970
      r5  = 0x00000000
      r6  = 0x0e6a0000
      r7  = 0x00000000
      r8  = 0x783bf940
      r9  = 0x0e6a0000
      r10 = 0x7b8679ec
      fp  = 0xb6657c38
      r12 = 0xfffffefe
      sp  = 0x7b8677a0
      lr  = 0xb6c62daf
      pc  = 0xb6aacb12
      cpsr = 0x80000030

    Top of Stack: (sp=0x7b8677a0)
    0x7b8677a0:   00000000 404e0000 00000001 000b6448
    0x7b8677b0:   7b8677b8 b6c62daf b6d4c8b8 783bf910
    0x7b8677c0:   00000000 b660d800 7b867808 b6d1f000
    0x7b8677d0:   b662f310 b6c62f43 7b8679f6 b6cac9cc
    0x7b8677e0:   7b8677f0 b6d1f000 7b8679ec b6b1d987
    0x7b8677f0:   7b867820 7b8679f0 7b867800 b6a068a3
    0x7b867800:   00000400 7b867820 00000000 00000012
    0x7b867810:   b660d800 00000000 00000012 b660d800 

    Instructions: (pc=0xb6aacb12)
    0xb6aacaf2:   ee064603 eeb83a10 ee377a46 ee286ac7
    0xb6aacb02:   ee667a07 ee876a86 eec67a27 ee777aa7
    0xb6aacb12:   edc47a87 46bd7a00 8b02ecbd 2064bd98
    0xb6aacb22:   f9e1f1fd 6a00edd4 0000e7db 480642c8 

    VM_Operation (0x77c5daa0): GenCollectForAllocation, mode: safepoint, requested by thread 0x77a0d790


    ---------------  P R O C E S S  ---------------

    Java Threads: ( => current thread )
      0x742a7448 JavaThread "Thread-513" daemon [_thread_blocked, id=11706, stack(0x76a70000,0x76ac0000)]
      0x73db7458 JavaThread "Thread-510" daemon [_thread_blocked, id=11027, stack(0x773b6000,0x77406000)]
      0x732b66f0 JavaThread "Thread-509" daemon [_thread_blocked, id=10757, stack(0x77b70000,0x77bc0000)]
      0x762428e0 JavaThread "Thread-508" daemon [_thread_blocked, id=10130, stack(0x79310000,0x79360000)]
      0x74284f48 JavaThread "Thread-503" daemon [_thread_blocked, id=8094, stack(0x78210000,0x78260000)]
      0x76d60e10 JavaThread "qtp9318356-11036" [_thread_blocked, id=4101, stack(0x78410000,0x78460000)]
      0x762dc8f0 JavaThread "qtp9318356-11035" [_thread_blocked, id=4100, stack(0x79bc0000,0x79c10000)]
      0x783bf910 JavaThread "qtp9318356-9925" [_thread_blocked, id=22685, stack(0x78260000,0x782b0000)]
      0x732170a8 JavaThread "qtp9318356-5567" [_thread_blocked, id=9426, stack(0x76810000,0x76860000)]
      0x760e2708 JavaThread "qtp9318356-5564" [_thread_blocked, id=9423, stack(0x768b0000,0x76900000)]
      0x77a0d790 JavaThread "Worker:12" [_thread_blocked, id=18002, stack(0x77c10000,0x77c60000)]
      0x77a0daf0 JavaThread "pool-1639-thread-1" [_thread_blocked, id=17984, stack(0x77186000,0x771d6000)]
      0x76db5e88 JavaThread "Worker:11" [_thread_blocked, id=17983, stack(0x77316000,0x77366000)]
      0x783c9df8 JavaThread "Worker:10" [_thread_blocked, id=17982, stack(0x77406000,0x77456000)]
      0x77a25388 JavaThread "SongLoaderWorker:9" [_thread_blocked, id=17942, stack(0x77366000,0x773b6000)]
      0x76d6bce8 JavaThread "com.jthink.songkong.server.callback.ServerFixSongs:8" [_thread_blocked, id=17922, stack(0x793b0000,0x79400000)]
      0x769c7cb8 JavaThread "pool-1636-thread-2" [_thread_blocked, id=17764, stack(0x77276000,0x772c6000)]
      0x732c50d8 JavaThread "pool-1636-thread-1" [_thread_blocked, id=17763, stack(0x77036000,0x77086000)]
      0x7690e140 JavaThread "pool-1601-thread-2" [_thread_blocked, id=11810, stack(0x771d6000,0x77226000)]
      0x769077c8 JavaThread "pool-1601-thread-1" [_thread_blocked, id=11809, stack(0x77226000,0x77276000)]
      0x769368a8 JavaThread "MappedByteBufferSource Async Flush Thread" daemon [_thread_blocked, id=8920, stack(0x76fe6000,0x77036000)]
      0x77d9d990 JavaThread "com.jthink.songkong.fileloader.LoadFolderWorker:3" [_thread_blocked, id=380, stack(0x77eb0000,0x77f00000)]
      0x77d940d8 JavaThread "pool-429-thread-1" [_thread_blocked, id=374, stack(0x77e60000,0x77eb0000)]
      0x77d9c740 JavaThread "SongLoaderCountFiles:2" [_thread_blocked, id=373, stack(0x77cb0000,0x77d00000)]
      0x7b670218 JavaThread "qtp9318356-91" [_thread_blocked, id=32431, stack(0x79360000,0x793b0000)]
      0x7833f418 JavaThread "qtp9318356-90" [_thread_in_native, id=32430, stack(0x79db0000,0x79e00000)]
      0x7b6e7478 JavaThread "MVStore background writer async:/mnt/disk1/share/SongKong/Prefs/Database/Database.mv.db" daemon [_thread_blocked, id=27227, stack(0x79270000,0x792c0000)]
      0x7b670cf8 JavaThread "H2 File Lock Watchdog (Socket) async:/mnt/disk1/share/SongKong/Prefs/Database/Database.lock.db" daemon [_thread_blocked, id=27222, stack(0x7a61c000,0x7a66c000)]
      0x7b694750 JavaThread "Scheduler-7136558" [_thread_blocked, id=13763, stack(0x78560000,0x785b0000)]
      0x7b6786c0 JavaThread "Scheduler-14033461" [_thread_blocked, id=13428, stack(0x785b0000,0x78600000)]
      0x7b6717d8 JavaThread "qtp9318356-45-acceptor-0@4cfe9c-ServerConnector@8b18a5{HTTP/1.1,[http/1.1]}{0.0.0.0:4567}" [_thread_in_native, id=13427, stack(0x7904c000,0x7909c000)]
      0x789c3e20 JavaThread "Thread-10" [_thread_blocked, id=13405, stack(0x791d0000,0x79220000)]
      0x7a0b4f10 JavaThread "C3P0PooledConnectionPoolManager[identityToken->1hge13oa8796in41o2hlgl|1df6bb7]-HelperThread-#2" daemon [_thread_blocked, id=13275, stack(0x7a66c000,0x7a6bc000)]
      0x7a0b4958 JavaThread "C3P0PooledConnectionPoolManager[identityToken->1hge13oa8796in41o2hlgl|1df6bb7]-HelperThread-#1" daemon [_thread_blocked, id=13274, stack(0x79b70000,0x79bc0000)]
      0x7a0b43c0 JavaThread "C3P0PooledConnectionPoolManager[identityToken->1hge13oa8796in41o2hlgl|1df6bb7]-HelperThread-#0" daemon [_thread_blocked, id=13273, stack(0x79920000,0x79970000)]
      0x7a0feb80 JavaThread "C3P0PooledConnectionPoolManager[identityToken->1hge13oa8796in41o2hlgl|1df6bb7]-AdminTaskTimer" daemon [_thread_blocked, id=13272, stack(0x79c10000,0x79c60000)]
      0x7adb1358 JavaThread "Java2D Disposer" daemon [_thread_blocked, id=13146, stack(0x7a7e3000,0x7a833000)]
      0xb66f12d8 JavaThread "Service Thread" daemon [_thread_blocked, id=13114, stack(0x7b2b6000,0x7b306000)]
      0xb66a1a20 JavaThread "Common-Cleaner" daemon [_thread_blocked, id=13105, stack(0x7b306000,0x7b356000)]
      0xb6675c88 JavaThread "Sweeper thread" daemon [_thread_blocked, id=13104, stack(0x7b356000,0x7b3a6000)]
      0xb6674978 JavaThread "C1 CompilerThread0" daemon [_thread_blocked, id=13103, stack(0x7b3a6000,0x7b426000)]
      0xb666cdc8 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=13102, stack(0x7b426000,0x7b476000)]
      0xb6667688 JavaThread "Finalizer" daemon [_thread_blocked, id=13099, stack(0x7b748000,0x7b798000)]
      0xb665a900 JavaThread "Reference Handler" daemon [_thread_blocked, id=13088, stack(0x7b798000,0x7b7e8000)]
      0xb6609308 JavaThread "main" [_thread_blocked, id=13037, stack(0xb67d9000,0xb6828000)]

    Other Threads:
    =>0xb6657d00 VMThread "VM Thread" [stack: 0x7b7e9000,0x7b869000] [id=13074]
      0xb66f2a90 WatcherThread [stack: 0x7b236000,0x7b2b6000] [id=13115]

    Threads with active compile tasks:

VM state:at safepoint (normal execution)

VM Mutex/Monitor currently owned by a thread:  ([mutex/lock_event])
[0xb6608520] Threads_lock - owner thread: 0xb6657d00
[0xb6608930] Heap_lock - owner thread: 0x77a0d790

Heap:
 def new generation   total 265600K, used 265599K [0x7c000000, 0x8e020000, 0x8ec00000)
  eden space 236160K, 100% used [0x7c000000, 0x8a6a0000, 0x8a6a0000)
  from space 29440K,  99% used [0x8c360000, 0x8e01fff8, 0x8e020000)
  to   space 29440K,   0% used [0x8a6a0000, 0x8a6a0000, 0x8c360000)
 tenured generation   total 589672K, used 298477K [0x8ec00000, 0xb2bda000, 0xb4400000)
   the space 589672K,  50% used [0x8ec00000, 0xa0f7b5c0, 0xa0f7b600, 0xb2bda000)
 Metaspace       used 45545K, capacity 46775K, committed 46976K, reserved 47408K

Card table byte_map: [0x7be3d000,0x7c000000] byte_map_base: 0x7ba5d000

Polling page: 0xb6f30000

But surely the question is what does AdaptiveWeightedAverage do ?

If this was the intended question. Then you might have a look into the source.

hotspot/src/share/vm/gc/shared/gcUtil.cpp

// A weighted average maintains a running, weighted average
// of some float value (templates would be handy here if we
// need different types).
//
// The average is adaptive in that we smooth it for the
// initial samples; we don't use the weight until we have
// enough samples for it to be meaningful.
//
// This serves as our best estimate of a future unknown.
//
class AdaptiveWeightedAverage : public CHeapObj<mtGC> {
 private:
  float            _average;        // The last computed average
  unsigned         _sample_count;   // How often we've sampled this average
  unsigned         _weight;         // The weight used to smooth the averages
                                    //   A higher weight favors the most
                                    //   recent data.
  bool             _is_old;         // Has enough historical data

hotspot/src/share/vm/gc/shared/gcUtil.hpp

void AdaptiveWeightedAverage::sample(float new_sample) {
  increment_count();

  // Compute the new weighted average
  float new_avg = compute_adaptive_average(new_sample, average());
  set_average(new_avg);
  _last_sample = new_sample;
}

and

float AdaptiveWeightedAverage::compute_adaptive_average(float new_sample,
                                                        float average) {
  // We smooth the samples by not using weight() directly until we've
  // had enough data to make it meaningful. We'd like the first weight
  // used to be 1, the second to be 1/2, etc until we have
  // OLD_THRESHOLD/weight samples.

It's used by the garbage collector.

Maybe somehow related to a problem with available memory. (it's a guess, don't have much experience with Java on ARM platform)

This lead me to this guess max heap -Xmx900m vs system memory 1G.