Lotus Domino Server Crash
We are running domino server is one of our virtual machine. We also see "allocation failure" and "GC Cycle started" error in our domino console frequently after moving to VM
OSVersion: Windows Server 2003 5.2 (Build 3790), PlatID=2, Service Pack 2 (4 Processors)
Notes Build: Release 6.5 September 26, 2003
It is getting crashed atleast 3 to 4 time a day. Below is the NSD log. Any help is much appreciated?
############################################################
### thread 38/65: [ nHTTP:0f08: 5112]
### FP=0d35ede4, PC=7c82845c, SP=0d35ed74, stkbase=0d260000, stksize=262144
############################################################
[ 1] 0x7c82845c ntdll+164956 (15d0,36ee80,0,d35f268)
[ 2] 0x77e61c8d KERNEL32+138381 (15d0,36ee80,0,0)
@[ 3] 0x601a0993 nnotes._OSRunExternalScript@4+579 (d35f380,c0000005,0,d35fa64)
@[ 4] 0x6019f6a0 nnotes._OSFaultCleanup@12+400 (0,0,0,b2b03e0)
@[ 5] 0x601a9509 nnotes._OSNTUnhandledExceptionFilter@4+185 (d35fa64,0,d35fa64,27df3023)
[ 6] 0x71d1bc74 jvm+1031284 (d35fa64,77e61ac1,d35fa6c,0)
[ 7] 0x77e792a3 KERNEL32+234147 (0,0,0,0)
############################################################
### FATAL THREAD 38/65 [ nHTTP:0f08: 5112]
### FP=0x0d35fe3c, PC=0x002a18af, SP=0x0d35fe38, stksize=4
### EAX=0x00000001, EBX=0x00000000, ECX=0x00010000, EDX=0x0e88e4f4
### ESI=0x00000000, EDI=0x000000c8, CS=0x0000001b, SS=0x00000023
### DS=0x00000023, ES=0x00000023, FS=0x0000003b, GS=0x00000000 Flags=0x00010202
Exception code: c0000005 (ACCESS_VIOLATION)
############################################################
############################################################
### PASS 2 : FATAL THREAD with STACK FRAMES 38/65 [ nHTTP:0f08: 5112]
### FP=0d35fe3c, PC=002a18af, SP=0d35fe38, stksize=4
Exception code: c0000005 (ACCESS_VIOLATION)
############################################################
# ---------- Top of the Stack ----------
# 0d35fe38 00000000 0d35fe4c 002a1992 00000000 |....L.5...*.....|
@[ 1] 0x002a18af nservlet._Remove+15 (0,2,d35fe78,2a127a)
# 0d35fe3c 0d35fe4c 002a1992 00000000 00000002 |L.5...*.........|
@[ 2] 0x002a1992 nservlet._ListMgr_RemoveNode+34 (2,b2b03e0,63d58ec,a9b8bf4)
# 0d35fe4c 0d35fe78 002a127a 00000002 0b2b03e0 |x.5.z.*.......+.|
# 0d35fe5c 063d58ec 0a9b8bf4 00000002 0d35fe90 |.X=...........5.|
# 0d35fe6c 05fdbb28 05fdc0b0 0b2b03e0 0d35feac |(.........+...5.|
@[ 3] 0x002a127a nservlet._ServletProcessRequest+106 (d35fe90,5fdba90,0,3)
# 0d35fe78 0d35feac 100254ab 0d35fe90 05fdba90 |..5..T....5.....|
# 0d35fe88 00000000 00000003 05fdbb28 10025530 |........(...0U..|
# 0d35fe98 10025850 100255c0 10025580 10025680 |PX...U...U...V..|
# 0d35fea8 10025780 0d35fec4 1000fea9 05fdba90 |.W....5.........|
[ 4] 0x100254ab nhttpstack (5fdba90,5fdb92c,0,3)
# 0d35feac 0d35fec4 1000fea9 05fdba90 05fdb92c |..5.........,...|
# 0d35febc 00000000 00000003 0d35fef4 1001cd31 |..........5.1...|
[ 5] 0x1000fea9 nhttpstack (2,5fda910,b66dfe0,0)
# 0d35fec4 0d35fef4 1001cd31 00000002 05fda910 |..5.1...........|
# 0d35fed4 0b66dfe0 00000000 1002a75c 05fdb92c |..f.....\...,...|
# 0d35fee4 00000000 0d35ff24 10033dec ffffffff |....$.5..=......|
[ 6] 0x1001cd31 nhttpstack (0,b5fe87e,0,b66e034)
# 0d35fef4 0d35ff30 10021246 00000000 0b5fe87e |0.5.F.......~._.|
# 0d35ff04 00000000 0b66e034 0d35ff20 100249a2 |....4.f. .5..I..|
# 0d35ff14 0b66dfe0 00000001 00000002 0d35fefc |..f...........5.|
# 0d35ff24 0d35ff70 100340a0 00000000 0d35ff7c |p.5..@......|.5.|
[ 7] 0x10021246 nhttpstack (b5fe88a,b5fe87e,0,60091b21)
# 0d35ff30 0d35ff7c 1002a51d 0b5fe88a 0b5fe87e ||.5......._.~._.|
# 0d35ff40 00000000 60091b21 0b5feaf0 00000001 |....!..`.._.....|
# 0d35ff50 00000000 0b5fe87e 0b5feaf8 00000043 |....~._..._.C...|
# 0d35ff60 00000008 00006f95 00000000 0d35ff38 |.....o......8.5.|
# 0d35ff70 0d35ffdc 100344b0 00000000 0d35ff98 |..5..D........5.|
[ 8] 0x1002a51d nhttpstack (3,b5fe87e,10027650,1002767a)
# 0d35ff7c 0d35ff98 1002a3a7 00000003 0b5fe87e |..5.........~._.|
# 0d35ff8c 10027650 1002767a 00000000 0d35ffb8 |Pv..zv........5.|
[ 9] 0x1002a3a7 nhttpstack (b5fe87e,0,0,0)
# 0d35ff98 0d35ffb8 60114924 0b5fe87e 00000000 |..5.$I.`~._.....|
# 0d35ffa8 00000000 00000000 00000000 0b5fe87e |............~._.|
@[10] 0x60114924 nnotes._ThreadWrapper@4+212 (0,0,0,0)
# 0d35ffb8 0d35ffec 77e6482f 00000000 00000000 |..5./H.w........|
# 0d35ffc8 00000000 00000000 c0000005 0d35ffc4 |..............5.|
# 0d35ffd8 0d35fa64 ffffffff 77e61a60 77e64838 |d.5.....`..w8H.w|
# 0d35ffe8 00000000 00000000 00000000 60114850 |............PH.`|
[11] 0x77e6482f KERNEL32
another crash
############################################################
### FATAL THREAD 4/7 [ ewweb:1090: 5724]
### FP=0x1281fe64, PC=0x77e4bef7, SP=0x1281fe10, stksize=84
### EAX=0x1281fe14, EBX=0x7c81a1f3, ECX=0x00000000, EDX=0xe06d7363
### ESI=0x1281fea4, EDI=0x1281fea4, CS=0x3ee8001b, SS=0x50fd0023
### DS=0x03b20023, ES=0x60030023, FS=0x03b2003b, GS=0x60000000 Flags=0x00000206
Exception code: e06d7363 ()
############################################################
[ 1] 0x77e4bef7 KERNEL32 (e06d7363,1,3,1281fe98)
[ 2] 0x1025d5bf lcppn201 (1281fefc,1252cca0,0,7c81a1a8)
[ 3] 0x10080757 lcppn201 (1281fefc,4481f8,10286578,0)
[ 4] 0x00401d6c ewweb (3,0,1252cd90,1281ff84)
############################################################
### PASS 2 : FATAL THREAD with STACK FRAMES 4/7 [ ewweb:1090: 5724]
### FP=1281fe64, PC=77e4bef7, SP=1281fe10, stksize=84
Exception code: e06d7363 ()
############################################################
# ---------- Top of the Stack ----------
# 1281fe10 10286780 e06d7363 00000001 00000000 |.g(.csm.........|
# 1281fe20 77e4bef7 00000003 19930520 1281feb8 |...w.... .......|
# 1281fe30 10287ce8 03b2ff29 1281fe54 60004757 |.|(.)...T...WG.`|
# 1281fe40 03b2f0e8 00000001 7c81a1a8 00000003 |...........|....|
# 1281fe50 7c81a1f3 1281fe68 60004a42 03b2f0e8 |...|h...BJ.`....|
# 1281fe60 00000027 1281fed4 1025d5bf e06d7363 |'.........%.csm.|
[ 1] 0x77e4bef7 KERNEL32 (e06d7363,1,3,1281fe98)
# 1281fe64 1281fed4 1025d5bf e06d7363 00000001 |......%.csm.....|
# 1281fe74 00000003 1281fe98 7c81a1a8 00000003 |...........|....|
# 1281fe84 e06d7363 00000001 00000000 00000000 |csm.............|
# 1281fe94 00000003 19930520 1281feb8 10287ce8 |.... ........|(.|
# 1281fea4 10096091 1281feb8 10287ce8 10aa0054 |.`.......|(.T...|
# 1281feb4 00000000 000013f9 100a378e 128113f9 |.........7......|
# 1281fec4 1252c9ec 85257b52 00000000 85257b4e |..R.R{%.....N{%.|
[ 2] 0x1025d5bf lcppn201 (1281fefc,1252cca0,0,7c81a1a8)
# 1281fed4 1281feec 10080757 1281fefc 1252cca0 |....W.........R.|
# 1281fee4 00000000 7c81a1a8 1281ff4c 00401d6c |.......|L...l.@.|
[ 3] 0x10080757 lcppn201 (1281fefc,4481f8,10286578,0)
# 1281feec 1281ff4c 00401d6c 1281fefc 004481f8 |L...l.@.......D.|
# 1281fefc 10286578 00000000 00000000 000007dd |xe(.............|
# 1281ff0c 00000004 00000012 00000005 00000017 |................|
# 1281ff1c 0000000b 00000039 00000031 00000001 |....9...1.......|
# 1281ff2c 00000005 00119305 85257b52 00010001 |........R{%.....|
# 1281ff3c 00401d4c 1281ff78 00433ea8 ffffffff |L.@.x....>C.....|
[ 4] 0x00401d6c ewweb (3,0,1252cd90,1281ff84)
# 1281ff4c 00442134 004011bb 00000003 00000000 |4!D...@.........|
# 1281ff5c 1252cd90 1281ff84 1252cd90 00000003 |..R.......R.....|
# 1281ff6c 0036ee80 00188940 004484e8 1281ffa8 |..6.@.....D.....|
# 1281ff7c 00433d2a ffffffff 1281ffb8 0042457e |*=C.........~EB.|
# 1281ff8c 00000000 00000000 00000000 1252cd90 |..............R.|
# 1281ff9c e06d7363 1281ff90 1281fa50 1281ffdc |csm.....P.......|
# 1281ffac 00428cd0 00438968 00000000 1281ffec |..B.h.C.........|
# 1281ffbc 77e6482f 1252cd90 00000000 00000000 |/H.w..R.........|
# 1281ffcc 1252cd90 00000000 1281ffc4 808331cc |..R..........1..|
# 1281ffdc ffffffff 77e61a60 77e64838 00000000 |....`..w8H.w....|
# 1281ffec 00000000 00000000 00424527 1252cd90 |........'EB...R.|
# 1281fffc 00000000 00000000 00000000 ffffffff |................|
# 1282000c 00000170 0f773abc 0f773abc 746e6553 |p....:w..:w.Sent|
# 1282001c 00000000 00000000 00000000 00000020 |............ ...|
# 1282002c 00000101 004484e8 004484e8 004484e8 |......D...D...D.|
# 1282003c 004484e8 004484e8 004484e8 004484e8 |..D...D...D...D.|
# 1282004c 004484e8 004484e8 004484e8 004484e8 |..D...D...D...D.|
# 1282005c 004484e8 004484e8 004484e8 10286578 |..D...D...D.xe(.|
# 1282006c 00000000 00000000 000007dd 00000004 |................|
# 1282007c 00000012 00000005 0000000a 0000002c |............,...|
# 1282008c 0000002d 00000004 00000001 00000000 |-...............|
# 1282009c 00358918 80257b51 00010001 65780003 |..5.Q{%.......xe|
# 128200ac 00001028 00000000 07dd0000 00040000 |(...............|
# 128200bc 00120000 00050000 000a0000 002f0000 |............../.|
# 128200cc 001b0000 001f0000 00010000 00000000 |................|
# 128200dc c87b0000 7b510035 00018025 00030001 |..{.5.Q{%.......|
# 128200ec ffffffff 004484e8 004484e8 00000000 |......D...D.....|
# 128200fc ffffffff ffffffff ffffffff 00000000 |................|
# 1282010c 00000003 00000005 00000000 00000000 |................|
# 1282011c 00000001 00000000 00000000 00000101 |................|
# 1282012c 00000050 0f773a2c 0f773a2c 1252d990 |P...,:w.,:w...R.|
# 1282013c 00000000 00000000 00000000 00000020 |............ ...|
# 1282014c 00000031 00000011 00000010 35323538 |1...........8525|
# 1282015c 64613636 36353030 63653961 64610000 |66ad0056a9ec..ad|
# 1282016c 656e696c 676e7c73 00000061 00000170 |lines|nga...p...|
# 1282017c 000000b1 1261e7f0 12823b14 00000000 |......a..;......|
# 1282018c 004484e8 12820490 12823b74 00000008 |..D.....t;......|
# 1282019c 004484e8 12820030 12866fe4 00000001 |..D.0....o......|
# 128201ac 004484e8 12821c50 128255a4 00000005 |..D.P....U......|
# 128201bc 004484e8 128218d0 129d0c34 00000010 |..D.....4.......|
# 128201cc 12cf7fe8 12cf0180 128255c4 00000002 |.........U......|
# 128201dc 004484e8 12821160 12d49f84 0000000f |..D.`...........|
# 128201ec 12cb3568 12cb35f0 12d78f74 00000002 |h5...5..t.......|
# 128201fc 12cbf028 12cbeb60 126187d4 00000003 |(...`.....a.....|
# 1282020c 004484e8 12822600 12d8fec4 0000000a |..D..&..........|
# 1282021c 12cdb028 12cd9610 00000000 000000b1 |(...............|
# 1282022c 00000bc0 1282500c 12827a3c 35323538 |.....P..<z..8525|
# 1282023c 64613636 36353030 63653961 6564242f |66ad0056a9ec/$de|
# 1282024c 6c756166 65697674 38342f77 38304333 |faultview/483C08|
# 1282025c 32323544 39303733 35383141 42373532 |D5223709A185257B|
# 1282026c 30303135 35383533 4f3f4433 446e6570 |510035853D?OpenD|
# 1282027c 6d75636f 00746e65 00000000 00000060 |ocument.....`...|
# 1282028c 00000021 0000000f 0000000e 33313032 |!...........2013|
# 1282029c 38313430 3a393020 00003434 00000021 |0418 09:44..!...|
# 128202ac 00000060 1282022c 0f773a34 0000004e |`...,...4:w.N...|
# 128202bc 00000031 00000019 00000019 00000021 |1...........!...|
# 128202cc 00000020 1282012c 1282042c 2c41474e | ...,...,...NGA,|
# 128202dc 00414750 3a390000 0f003435 00000040 |PGA...9:54..@...|
# 128202ec 00000021 0000000f 0000000e 33313032 |!...........2013|
# 128202fc 38313430 3a393020 00003435 000000e0 |0418 09:54......|
# 1282030c 00000021 00000004 00000003 00415045 |!...........EPA.|
# 1282031c 6e697200 656d0067 3a3a746e 00000021 |.ring.ment::!...|
# 1282032c 00000120 0f773a94 0f773a94 31305045 | ....:w..:w.EP01|
# 1282033c 00003739 00000019 00000019 00000020 |97.......... ...|
# 1282034c 00000021 00000006 00000005 6c65676d |!...........mgel|
# 1282035c 6e00006c 00000067 00000039 00000040 |l..ng...9...@...|
# 1282036c 00000051 0000003a 00000039 20275355 |Q...:...9...US' |
# 1282037c 6d736f4b 6620736f 736d7261 206e6920 |Kosmos farms in |
# 1282038c 70206f74 70736f72 69746365 6f206576 |to prospective o|
# 1282039c 62206c69 6b636f6c 666f2073 72492066 |il blocks off Ir|
# 128203ac 6e616c65 12000064 0f773a0c 00000051 |eland....:w.Q...|
# 128203bc 00000090 0f773a4c 0f773a4c 00000010 |....L:w.L:w.....|
# 128203cc 00000021 00000007 00000007 72657072 |!...........rper|
# 128203dc 0020206b 12820067 0f773a0c 00000021 |k .g....:w.!...|
# 128203ec 00000060 1282022c 0f773a34 34373033 |`...,...4:w.3074|
# 128203fc 38333631 31323334 00303132 00000020 |16384321210. ...|
# 1282040c 00000021 00000005 00000004 746e6553 |!...........Sent|
# 1282041c 00000000 00000019 00000019 00000021 |............!...|
# 1282042c 00000020 1282012c 0f773a14 726f7453 | ...,....:w.Stor|
# 1282043c 6e000079 00000067 00000000 00000220 |y..ng....... ...|
# 1282044c 00000041 00000021 00000020 43333834 |A...!... ...483C|
# 1282045c 35443830 37333232 31413930 35323538 |08D5223709A18525|
# 1282046c 31354237 35333030 44333538 00000000 |7B510035853D....|
# 1282047c 00000000 00000000 00000000 00000260 |............`...|
# 1282048c 00000101 004484e8 004484e8 004484e8 |......D...D...D.|
# 1282049c 004484e8 004484e8 004484e8 004484e8 |..D...D...D...D.|
# 128204ac 004484e8 004484e8 004484e8 004484e8 |..D...D...D...D.|
# 128204bc 004484e8 004484e8 004484e8 10286578 |..D...D...D.xe(.|
# 128204cc 00000000 00000000 000007dd 00000004 |................|
# 128204dc 00000012 00000005 0000000a 0000002c |............,...|
# 128204ec 00000023 00000011 00000001 00000000 |#...............|
# 128204fc 0035853d 80257b51 00010001 65780003 |=.5.Q{%.......xe|
# 1282050c 00001028 00000000 07dd0000 00040000 |(...............|
# 1282051c 00120000 00050000 000a0000 00360000 |..............6.|
# 1282052c 001d0000 00080000 00010000 00000000 |................|
# 1282053c 6d3c0000 7b510036 00018025 00030001 |..<m6.Q{%.......|
# 1282054c ffffffff 004484e8 004484e8 00000000 |......D...D.....|
# 1282055c ffffffff ffffffff ffffffff 00000000 |................|
# 1282056c 00000003 00000005 00000000 00000000 |................|
# 1282057c 00000001 00000012 00000011 00000360 |............`...|
# 1282058c 00000021 12539c60 12538370 12534b20 |!...`.S.p.S. KS.|
# 1282059c 12533190 0f770067 0f773a2c 00000380 |.1S.g.w.,:w.....|
# 128205ac 00000021 00000004 00000003 00414750 |!...........PGA.|
# 128205bc 00000000 00000019 00000019 000003a0 |................|
# 128205cc 00000021 00000001 00000001 0041004e |!...........N.A.|
# 128205dc 6e697200 00000067 00000019 00000021 |.ring.......!...|
# 128205ec 00000040 0f773a24 0f773a24 00000010 |@...$:w.$:w.....|
# 128205fc 00000031 00000011 00000010 35323538 |1...........8525|
# 1282060c 64613636 36353030 63653961 64610000 |66ad0056a9ec..ad|
# 1282061c 656e696c 676e7c73 00000061 00000400 |lines|nga.......|
# 1282062c 00000021 0000000f 0000000e 33313032 |!...........2013|
# 1282063c 38313430 3a303120 00003931 00000420 |0418 10:19.. ...|
# 1282064c 00000021 0000000f 0000000e 33313032 |!...........2013|
# 1282065c 38313430 3a393020 00003034 00000021 |0418 09:40..!...|
# 1282066c 00000060 1282693c 0f773a34 35323538 |`...<i..4:w.8525|
# 1282067c 64613636 36353030 63653961 6564242f |66ad0056a9ec/$de|
# 1282068c 6c756166 65697674 37442f77 32373339 |faultview/D79372|
# 1282069c 44464637 34324336 35384342 42373532 |7FFD6C24BC85257B|
# 128206ac 30303135 39323533 4f3f3044 446e6570 |51003529D0?OpenD|
# 128206bc 6d75636f 00746e65 00000019 000004a0 |ocument.........|
# 128206cc 00000021 00000004 00000003 00414750 |!...........PGA.|
# 128206dc 6e697200 00000067 00000000 00000021 |.ring.......!...|
# 128206ec 00000100 0f773a84 0f773a84 444e4554 |.....:w..:w.TEND|
# 128206fc 44205245 3a415441 646e4920 73276169 |ER DATA: India's|
# 1282070c 4c564f20 66666f20 20737265 6f6b6f53 | OVL offers Soko|
# 1282071c 7263206c 20656475 20726f66 656e754a |l crude for June|
# 1282072c 2d373220 6c203033 6964616f 0000676e | 27-30 loading..|
# 1282073c 00000000 00000000 00000000 00000061 |............a...|
Saravanan1 Answer
Your crash stack matches a known issue in Domino R6. Reported as SPR MIAS5S7F39.
The issue happens due to heavy load on the nServlet process. Reported in 6.5 and should be fixed in the latest version of R6.
However I would strongly recommend upgrading to a more current version, as the last time R6 was updated was 7 years ago.
answered on Server Fault Apr 19, 2013 by 
Simon O'Doherty
Simon O'DohertyUser contributions licensed under CC BY-SA 3.0