java.lang.OutOfMemoryError: requested 32756 bytes for ChunkPool::allocate. Out of swap space?

I am working with a java application deployed on an HP server via WebLogic 10.3.

Version info:

WebLogic Version 10.3
OS Version B.11.23

$ java -version
java version "1.6.0.03"
Java(TM) SE Runtime Environment (build 1.6.0.03-jinteg_05_feb_2009_11_19-b00)
Java HotSpot(TM) Server VM (build 11.0-b16-jre1.6.0.03-rc2, mixed mode)

I am getting this java.lang.OutOfMemoryError and the weblogic managed server is shutting down. It's always on the compiler thread. The error message says to check swap and MAXDSIZ. I looked at both and they seemed ok (see below.) I've gone through the forums and looked at various things but none seem to fully apply. I've changed the heap and stack both up and down with no luck. Also, I am pretty sure this is the first 1.6 port of the application on HP.

# kctune -a -v -q maxdsiz  
Tunable maxdsiz  
Description Maximum size of the data segment of a 32-bit process (bytes)  
Module vm  
Current Value 3221225472  
Value at Next Boot 3221225472  
Value at Last Boot 3221225472  
Default Value 1073741824  
Constraints maxdsiz GT 262144  
maxdsiz LT 4294963200  
Can Change Immediately or at Next Boot  






# swapinfo -tam  
Mb Mb Mb PCT START/ Mb  
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME  
dev 4096 261 3835 6% 0 - 1 /dev/vg00/lvol2  
dev 4096 263 3833 6% 0 - 1 /dev/vg00/lvol9  
reserve - 1195 -1195  
memory 4085 1598 2487 39%  
total 12277 3317 8960 27% - 0 -  

I get a core and a hs.log file. This question seems similar but not sire of the resolution.

JVM errors after updating to Java 6 Update 14

I am posting the log file. None of the Heap spaces show near 100%. I am not used to looking at these and was hoping someone could help guide me to a solution. I see physical memory goes low but not 100%.

#
# An unexpected error has been detected by Java Runtime Environment:
#
# java.lang.OutOfMemoryError: requested 32756 bytes for ChunkPool::allocate. Out of swap space?
Possible causes:
- not enough swap space left, or
- kernel parameter MAXDSIZ is very small.
#
# Internal Error (allocation.cpp:128), pid=11593, tid=9
# Error: ChunkPool::allocate
#
# Java VM: Java HotSpot(TM) Server VM (11.0-b16-jre1.6.0.03-rc2 mixed mode hp-ux-ia64)
# Please report this error to HP customer support.
#

T H R E A D
Current thread (0059e400): JavaThread "CompilerThread0" daemon [_thread_in_native, id=9, lwp_id=3000034, stack(21e01000,22001
000)]

Stack: 21e01000,22001000

Current CompileTask:
C3:154% ! weblogic.timers.internal.TimerThread$Thread.run()V @ 0 (196 bytes)

P R O C E S S
Java Threads: (current thread )
01156000 JavaThread "STANDBY ExecuteThread: '4' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon [_thread_blocke
d, id=58, lwp_id=3000185, stack(1fa01000,1fa41000)]
04fa1c00 JavaThread "STANDBY ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon [_thread_blocke
d, id=57, lwp_id=3000183, stack(1fac1000,1fb01000)]
04bb1600 JavaThread "ACTIVE ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon [_thread_blocked
, id=56, lwp_id=3000182, stack(1fb81000,1fbc1000)]
04f81200 JavaThread "DynamicSSLListenThreadDefaultSecure" daemon [_thread_in_native, id=55, lwp_id=3000181, stack(1fc41000
,1fc81000)]
043d6e00 JavaThread "DynamicListenThreadDefault" daemon [_thread_in_native, id=54, lwp_id=3000180, stack(1fd01000,1fd41000
)]
043d3400 JavaThread "weblogic.GCMonitor" daemon _thread_blocked, id=53, lwp_id=3000178, stack(1fdc1000,1fe01000)
03167a00 JavaThread "Thread-14" daemon _thread_in_native, id=52, lwp_id=3000177, stack(1fe81000,1fec1000)
03e5a600 JavaThread "RecyclableThread - waiting" daemon _thread_blocked, id=51, lwp_id=3000155, stack(1ff41000,1ff81000)
03e56e00 JavaThread "RecyclableThread - waiting" daemon _thread_blocked, id=50, lwp_id=3000154, stack(20001000,20041000)

03e45000 JavaThread "RecyclableThread - waiting" daemon _thread_blocked, id=49, lwp_id=3000153, stack(200c1000,20101000)
03e2e200 JavaThread "RecyclableThread - waiting" daemon _thread_blocked, id=48, lwp_id=3000152, stack(20181000,201c1000)
03e29000 JavaThread "RecyclableThread - waiting" daemon _thread_blocked, id=47, lwp_id=3000151, stack(20241000,20281000)
03e27a00 JavaThread "RecyclableThread - waiting" daemon _thread_blocked, id=46, lwp_id=3000150, stack(20301000,20341000)
03e15a00 JavaThread "RecyclableThread - waiting" daemon _thread_blocked, id=45, lwp_id=3000149, stack(203c1000,20401000)
0073de00 JavaThread "RecyclableThread - waiting" daemon _thread_blocked, id=44, lwp_id=3000148, stack(20481000,204c1000)
0073da00 JavaThread "RecyclableThread - waiting" daemon _thread_blocked, id=43, lwp_id=3000147, stack(20c01000,20c41000)
0073d600 JavaThread "RecyclableThread - waiting" daemon _thread_blocked, id=42, lwp_id=3000146, stack(20cc1000,20d01000)
03e11800 JavaThread "SimpleQuartzScheduler_QuartzSchedulerThread" [_thread_blocked, id=41, lwp_id=3000139, stack(20541000,20
581000)]
03e0f400 JavaThread "ThreadPoolManager" daemon _thread_blocked, id=39, lwp_id=3000137, stack(206c1000,20701000)
03e0d000 JavaThread "DataCacheManager" daemon _thread_blocked, id=38, lwp_id=3000136, stack(20781000,207c1000)
03e0b400 JavaThread "TriggerManager" daemon _thread_blocked, id=37, lwp_id=3000135, stack(20841000,20881000)
03e09c00 JavaThread "RequestEventManager" daemon _thread_blocked, id=36, lwp_id=3000134, stack(20901000,20941000)
034b9200 JavaThread "HyperServiceCacheManager" daemon _thread_blocked, id=35, lwp_id=3000133, stack(209c1000,20a01000)
034b7800 JavaThread "LogManager" daemon _thread_blocked, id=34, lwp_id=3000132, stack(20a81000,20ac1000)
034b7400 JavaThread "ServiceTaskManager" daemon _thread_blocked, id=33, lwp_id=3000131, stack(21141000,21181000)
0073e800 JavaThread "NXAHSProcessMgr::MasterThread" daemon _thread_blocked, id=32, lwp_id=3000130, stack(20b41000,20b81000)
03ae6800 JavaThread "EventBroker" daemon _thread_blocked, id=29, lwp_id=3000102, stack(20d81000,20dc1000)
03aaec00 JavaThread "I/O Monitor" daemon _thread_blocked, id=28, lwp_id=3000101, stack(20e41000,20e81000)
033bf600 JavaThread "Timer-2" daemon _thread_blocked, id=27, lwp_id=3000100, stack(20f01000,20f41000)
0312be00 JavaThread "Thread-11" daemon _thread_blocked, id=26, lwp_id=3000057, stack(20fc1000,21001000)
0079cc00 JavaThread "DoSManager" daemon _thread_blocked, id=25, lwp_id=3000056, stack(21081000,210c1000)
030ca400 JavaThread "VDE Transaction Processor Thread" daemon [_thread_blocked, id=23, lwp_id=3000054, stack(21201000,212410
00)]
03185c00 JavaThread "ExecuteThread: '2' for queue: 'weblogic.socket.Muxer'" daemon [_thread_blocked, id=22, lwp_id=3000053,
stack(212c1000,21301000)]
0312d200 JavaThread "ExecuteThread: '1' for queue: 'weblogic.socket.Muxer'" daemon [_thread_blocked, id=21, lwp_id=3000052,
stack(21381000,213c1000)]
030cba00 JavaThread "ExecuteThread: '0' for queue: 'weblogic.socket.Muxer'" daemon [_thread_in_native, id=20, lwp_id=3000051
, stack(21881000,218c1000)]
00a1c800 JavaThread "ACTIVE ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon [_thread_blocked
, id=19, lwp_id=3000050, stack(21941000,21981000)]
030ab600 JavaThread "Thread-7" daemon _thread_blocked, id=18, lwp_id=3000049, stack(21a01000,21a41000)
00689600 JavaThread "weblogic.timers.TimerThread" daemon _thread_blocked, id=17, lwp_id=3000048, stack(21ac1000,21b01000)
00683e00 JavaThread "weblogic.time.TimeEventGenerator" daemon [_thread_blocked, id=16, lwp_id=3000047, stack(21b81000,21bc10
00)]
00677800 JavaThread "ACTIVE ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon [_thread_blocked
, id=15, lwp_id=3000046, stack(22081000,220c1000)]
00677000 JavaThread "Timer-1" daemon _thread_blocked, id=14, lwp_id=3000045, stack(22281000,222c1000)
02094e00 JavaThread "Timer-0" daemon _thread_blocked, id=13, lwp_id=3000038, stack(22481000,224c1000)
005a9e00 JavaThread "Low Memory Detector" daemon _thread_blocked, id=12, lwp_id=3000037, stack(22681000,226c1000)
005a1400 JavaThread "CompilerThread1" daemon _thread_blocked, id=10, lwp_id=3000035, stack(21601000,21801000)
=>0059e400 JavaThread "CompilerThread0" daemon _thread_in_native, id=9, lwp_id=3000034, stack(21e01000,22001000)
0059c200 JavaThread "Signal Dispatcher" daemon _thread_blocked, id=8, lwp_id=3000033, stack(22881000,228c1000)
0057cc00 JavaThread "Finalizer" daemon _thread_blocked, id=7, lwp_id=3000032, stack(231c1000,23201000)
00535600 JavaThread "Reference Handler" daemon _thread_blocked, id=6, lwp_id=3000031, stack(23281000,232c1000)
00465800 JavaThread "main" _thread_blocked, id=1, lwp_id=-1, stack(7fdfe000,7ffff000)

Other Threads:
0051aa00 VMThread stack: 22381000,22401000 id=5, lwp_id=3000030
0048a800 WatcherThread stack: 22181000,22201000 id=11, lwp_id=3000036

VM state: not at safepoint (normal execution)

VM Mutex/Monitor currently owned by a thread: None

Heap  
PSYoungGen total 308544K, used 132524K [5deb0000, 72790000, 73400000)  
eden space 307328K, 43% used [5deb0000,6601b248,70ad0000)  
from space 1216K, 0% used [72660000,72660000,72790000)  
to space 13888K, 0% used [70c70000,70c70000,71a00000)  
PSOldGen total 174784K, used 90916K [33400000, 3deb0000, 5deb0000)  
object space 174784K, 52% used [33400000,38cc9350,3deb0000)  
PSPermGen total 226304K, used 113097K [23400000, 31100000, 33400000)  
object space 226304K, 49% used [23400000,2a272528,31100000)  

Dynamic libraries:
/opt/java1.6/jre/bin/IA64N/java
text:0x00400000-0x0041b040 data:0x0041c000-0x0041c4f8
/usr/lib/hpux32/libpthread.so.1
text:0xc008d000-0xc011fcb0 data:0x777f2000-0x777f4508
/opt/java1.6/jre/bin/IA64N/../../lib/IA64N/jli/libjli.so
text:0xc1794000-0xc17b15e0 data:0x777f0000-0x777f1498
/usr/lib/hpux32/libdl.so.1
text:0xc1953000-0xc1956ed0 data:0x777ed000-0x777ed150
/usr/lib/hpux32/libc.so.1
text:0xc0120000-0xc039e300 data:0x777d9000-0x777ea0f8
/opt/java1.6/jre/lib/IA64N/server/libjvm.so
text:0xcb8f6000-0xcd783450 data:0x775ae000-0x777bace8
/usr/lib/hpux32/libm.so.1
text:0xc052f000-0xc0691c30 data:0x777cc000-0x777cf818
/usr/lib/hpux32/librt.so.1
text:0xc4ebd000-0xc4ec31f0 data:0x777d4000-0x777d41a8
/usr/lib/hpux32/libuca.so.1

text:0xc0c1c000-0xc0c23af0 data:0x777c9000-0x777c9150
/usr/lib/hpux32/libelf.so.1
text:0xc17b2000-0xc17d9fe0 data:0x777c8000-0x777c88c8
/usr/lib/hpux32/libunwind.so.1
text:0xc0bd3000-0xc0c1b910 data:0x777c7000-0x777c7628
/usr/lib/hpux32/libstd_v2.so.1
text:0xc09e2000-0xc0b82fc0 data:0x775a0000-0x775adc28
/usr/lib/hpux32/libCsup.so.1
text:0xc0b83000-0xc0bd2190 data:0x7759a000-0x7759ff20
/opt/java1.6/jre/lib/IA64N/native_threads/libhpi.so
text:0xc17da000-0xc17f0a60 data:0x777bc000-0x777bd6a0
/opt/java1.6/jre/lib/IA64N/libverify.so
text:0xc1957000-0xc1976b20 data:0x777bb000-0x777bbaf0
/opt/java1.6/jre/lib/IA64N/libjava.so
text:0xcb79e000-0xcb801070 data:0x77591000-0x77593208
/opt/java1.6/jre/lib/IA64N/libzip.so
text:0xc1977000-0xc19ab810 data:0x7758d000-0x7758e800
/opt/java1.6/jre/lib/IA64N/libmanagement.so
text:0xc19b4000-0xc19befa0 data:0x7758a000-0x7758a300
/opt/java1.6/jre/lib/IA64N/libnet.so
text:0xcd784000-0xcd7b2b30 data:0x77589000-0x77589908
/usr/lib/hpux32/libnss_files.so.1
text:0xc0518000-0xc052ea70 data:0x77586000-0x775868e0
/opt/java1.6/jre/lib/IA64N/libnio.so
text:0xc17f9000-0xc1808160 data:0x77585000-0x77585588
/d1/app/bea/wlserver_10.3/server/native/hpux11/IPF32/libwlfileio2.so
text:0xc18ab000-0xc18afb50 data:0x77582000-0x77582218
/d1/app/bea/wlserver_10.3/server/native/hpux11/IPF32/libmuxer.so
text:0xc5f0e000-0xc5f14f10 data:0x77581000-0x775818a0
/usr/lib/hpux32/libnss_nisplus.so.1
text:0xc7756000-0xc776fb40 data:0x7753e000-0x7753f580
/usr/lib/hpux32/libsec.so.1
text:0xc04d5000-0xc0517f50 data:0x7751c000-0x77526db0
/usr/lib/hpux32/libnsl.so.1
text:0xc03b2000-0xc04a5410 data:0x7752c000-0x77539da0
/usr/lib/hpux32/libxti.so.1
text:0xc04a6000-0xc04d4230 data:0x77529000-0x7752b420
/usr/lib/hpux32/libnss_dns.so.1
text:0xc0ceb000-0xc0cf8f00 data:0x7751b000-0x7751b370

VM Arguments:
jvm_args: -Xms256m -Xmx1024m -XX:PermSize=128m -XX:MaxPermSize=256m -Dbea.home=/d1/app/bea -Dweblogic.RootDirectory=/d1/app/bea/user_projects/domains/dev1 -Dweblogic.Name=server1 -Dweblogic.ListenAddress=myserver -Dweblogic.ListenPort=7003 -Dweblogic.management.server=t3://myserver:7001 -Dweblogic.system.BootIdentityFile=/d1/app/bea/user_projects/domains/dev1/servers/server1/security/boot.properties -Djavax.xml.soap.MessageFactory=weblogic.xml.saaj.MessageFactoryImpl -Dcom.sun.xml.namespace.QName.useCompatibleSerialVersionUID=1.0 -Djava.endorsed.dirs=/d1/home/wls/greencm/myapplication/core-4.0.0/blueEndorsed -Djava.awt.headless=true -Djava.security.auth.login.config=/d1/home/wls/greencm/myapplication/jaas.config -Djava.security.policy=/d1/app/bea/wlserver_10.3/server/lib/weblogic.policy

java_command: weblogic.Server
Launcher Type: SUN_STANDARD

Environment Variables:
JAVA_HOME=/opt/java1.6
CLASSPATH=/d1/app/bea/patch_wls1030/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/d1/app/bea/patch_cie660/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/opt/java1.6/lib/tools.jar:/d1/app/bea/wlserver_10.3/server/lib/weblogic_sp.jar:/d1/app/bea/wlserver_10.3/server/lib/weblogic.jar:/d1/app/bea/modules/features/weblogic.server.modules_10.3.0.0.jar:/d1/app/bea/wlserver_10.3/server/lib/webservices.jar:/d1/app/bea/modules/org.apache.ant_1.6.5/lib/ant-all.jar:/d1/app/bea/modules/net.sf.antcontrib_1.0.0.0_1-0b2/lib/ant-contrib.jar:
PATH=/d1/app/bea/wlserver_10.3/server/bin:/d1/app/bea/modules/org.apache.ant_1.6.5/bin:/opt/java1.6/jre/bin:/opt/java1.6/bin:/opt/java1.6/bin:/usr/bin:/usr/local/bin:/usr/bin:/usr/sbin:.
LD_LIBRARY_PATH=/opt/java1.6/jre/lib/IA64N:/opt/java1.6/jre/lib/IA64N/server:/opt/java1.6/jre/../lib/IA64N:/opt/java1.6/jre/lib/IA64N:/opt/java1.6/jre/lib/IA64N/server:/opt/java1.6/jre/../lib/IA64N
SHELL=/usr/bin/sh
SHLIB_PATH=:/d1/app/bea/wlserver_10.3/server/native/hpux11/IPF32:/d1/app/bea/wlserver_10.3/server/native/hpux11/IPF32/oci920_8
PTHREAD_COMPAT_MODE=0

Signal Handlers:
SIGSEGV: htp://libjvm.so+0xffed64a8, sa_mask[0]=0xbf7ffeff, sa_flags=0x00000051
SIGBUS : htp://libjvm.so+0xffed64a8, sa_mask[0]=0xbf7ffeff, sa_flags=0x00000051
SIGFPE : htp://libjvm.so+0xffed64a8, sa_mask[0]=0xbf7ffeff, sa_flags=0x00000050
SIGPIPE: htp://libjvm.so+0xffed64a8, sa_mask[0]=0xbf7ffeff, sa_flags=0x00000050
SIGXFSZ: htp://libjvm.so+0xffed64a8, sa_mask[0]=0xbf7ffeff, sa_flags=0x00000050
SIGILL : htp://libjvm.so+0xffed64a8, sa_mask[0]=0xbf7ffeff, sa_flags=0x00000051
SIGUSR1: SIG_DFL, sa_mask[0]=0x00000000, sa_flags=0x00000004
SIGUSR2: htp://libjvm.so+0xffed4dd8, sa_mask[0]=0x00000000, sa_flags=0x00000000
SIGHUP : SIG_IGN, sa_mask[0]=0x00000000, sa_flags=0x00000004
SIGINT : SIG_IGN, sa_mask[0]=0x00000000, sa_flags=0x00000004
SIGQUIT: htp://libjvm.so+0xffed1ee8, sa_mask[0]=0xbf7ffeff, sa_flags=0x00000040
SIGTERM: htp://libjvm.so+0xffed1ee8, sa_mask[0]=0xbf7ffeff, sa_flags=0x00000040
SIGRTMIN: htp://libjvm.so+0xffed4fa8, sa_mask[0]=0x00000000, sa_flags=0x00000010
SIGRTMIN+1: htp://libjvm.so+0xffed64a8, sa_mask[0]=0xbf7ffeff, sa_flags=0x00000050

S Y S T E M
OS:HP-UX
uname:HP-UX B.11.23 U ia64
rlimit: STACK 131072k, CORE 2097151k, NOFILE 4096, AS infinity
load average:0.45 0.14 0.06

CPU:total 2

Memory: 4k page, physical 4182800k(11892k free)

vm_info: Java HotSpot(TM) Server VM (11.0-b16-jre1.6.0.03-rc2) for hp-ux-ia64 JRE (1.6.0), built on Feb 6 2009 04:02:16 by "jinteg" with aCC

time: Tue Dec 1 01:33:12 2009
elapsed time: 9486 seconds

This is what is in /var/adm/syslog:

Nov 30 01:36:04 myserver syslog: Java out of memory messages are marked with pid:5031 in /var/adm/syslog/syslog.log.  
Nov 30 01:36:04 myserver syslog: pid: 5031 4Kb 4Kb 4Kb PCT START/ 4Kb  
DEV NO./  
Nov 30 01:36:04 myserver syslog: pid: 5031 TYPE AVAIL USED FREE USED LIMIT RESERVE  

PRI FILE NAME  
Nov 30 01:36:04 myserver syslog: pid: 5031 dev 1048576 169912 878664 16.20% 0 -  
1 64 (major) 2 (minor)  
Nov 30 01:36:04 myserver syslog: pid: 5031 reserve - 579015 -579015  
Nov 30 01:36:04 myserver syslog: pid: 5031 memory 1045700 411902 633798 39.39%  

Sorry for any weird formatting and thanks.

Updating this for future searchers. Solution was to upgrade the JDK.

To recap, this was done due to random OOM JVM crashes. Per HP:

http://docs.hp.com/en/JDKJRE60RN/jdk_rnotes_6.0.05.html

In JDK 6.0.03, occasionally, a Java program can experience a memory leak in the C heap due to QXCR1000920722, resulting in an out-of-memory error. HP recommends that you upgrade from JDK 6.0.03 to JDK 6.0.04 (or greater) to prevent possible C heap memory leaks due to QXCR1000920722.

Thanks all.