server crash please help urgent (php sessionclean)?
This morning my server went down for about an hour I'm using ubuntu 16.04LTS on AWS EC2.
some info: it's a t2.medium instance, I recently upgraded from ubuntu 14.04 to 16.04, updated lots of packages, this is a production instance, I'm using PHP Version 5.6.40-10+ubuntu16.04.1+deb.sury.org+1 and for some sites I'm using the latest php via fast-cgi php modules, my device memory /xvda1 is at 49% storage remaining.
From syslog:
Oct 11 05:09:10 ip-10-0-x-xxx systemd[1]: Starting Clean php session files...
Oct 11 05:09:10 ip-10-0-x-xxx systemd[1]: Stopping User Manager for UID 110...
Oct 11 05:09:10 ip-10-0-x-xxx systemd[19573]: Reached target Shutdown.
Oct 11 05:09:10 ip-10-0-x-xxx systemd[19573]: Starting Exit the Session...
Oct 11 05:09:10 ip-10-0-x-xxx systemd[19573]: Stopped target Default.
Oct 11 05:09:10 ip-10-0-x-xxx systemd[19573]: Stopped target Basic System.
Oct 11 05:09:10 ip-10-0-x-xxx systemd[19573]: Stopped target Timers.
Oct 11 05:09:10 ip-10-0-x-xxx systemd[19573]: Stopped target Sockets.
Oct 11 05:09:10 ip-10-0-x-xxx systemd[19573]: Stopped target Paths.
Oct 11 05:09:10 ip-10-0-x-xxx CRON[28971]: (root) CMD ( [ -
x/usr/lib/php5/maxlifetime ] && [ -x /usr/lib/php5/sessionclean ] && [ -d /var/lib/php5 ] && /usr/lib/php5/sessionclean /var/lib/php5 $(/usr/lib/php5/maxlifetime))
Oct 11 05:09:10 ip-10-0-x-xxx CRON[28972]: (root) CMD ( [ -x /usr/lib/php/sessionclean ] && if [ ! -d /run/systemd/system ]; then /usr/lib/php/sessionclean; fi)
Oct 11 05:09:10 ip-10-0-x-xxx systemd[19573]: Received SIGRTMIN+24 from PID 28967 (kill).
Oct 11 05:09:10 ip-10-0-x-xxx systemd[1]: Stopped User Manager for UID 110.
Oct 11 05:09:10 ip-10-0-x-xxx systemd[1]: Removed slice User Slice of solr.
Oct 11 05:10:32 ip-10-0-x-xxx kernel: [488908.756151] apache2 invoked oom-killer: gfp_mask=0x24201ca, order=0, oom_score_adj=0
Oct 11 05:10:32 ip-10-0-x-xxx kernel: [488908.756155] apache2 cpuset=/ mems_allowed=0
Oct 11 05:10:33 ip-10-0-x-xxx kernel: [488908.756160] CPU: 1 PID: 29346 Comm: apache2 Not tainted 4.4.0-157-generic #185-Ubuntu
Oct 11 05:10:33 ip-10-0-x-xxx kernel: [488908.756162] Hardware name: Xen HVM domU, BIOS 4.2.amazon 08/24/2006
Oct 11 05:10:33 ip-10-0-x-xxx kernel: [488908.756163] 0000000000000286 3f04de0b6dc470df ffff88005a2739e8 ffffffff8140c9a1
Oct 11 05:10:33 ip-10-0-x-xxx kernel: [488908.756166] ffff88005a273ba0 ffff8800e84e4600 ffff88005a273a58 ffffffff81219bde
Oct 11 05:10:33 ip-10-0-x-xxx kernel: [488908.756169] ffff8800eaf61c00 ffff88005a273ba0 ffff88005a273ab0 ffff8800e86a5400
Oct 11 05:10:33 ip-10-0-x-xxx kernel: [488908.756171] Call Trace:
Oct 11 05:10:33 ip-10-0-x-xxx kernel: [488908.756180] [<ffffffff8140c9a1>] dump_stack+0x63/0x82
Oct 11 05:10:33 ip-10-0-x-xxx kernel: [488908.756184] [<ffffffff81219bde>] dump_header+0x5a/0x1c3
Oct 11 05:10:33 ip-10-0-x-xxx kernel: [488908.756189] [<ffffffff8119db2b>] oom_kill_process+0x20b/0x3d0
Oct 11 05:10:33 ip-10-0-x-xxx kernel: [488908.756191] [<ffffffff8119df38>] out_of_memory+0x1f8/0x460
Oct 11 05:10:33 ip-10-0-x-xxx kernel: [488908.756196] [<ffffffff811a3f73>] __alloc_pages_slowpath.constprop.88+0x943/0xaf0
Oct 11 05:10:33 ip-10-0-x-xxx kernel: [488908.756199] [<ffffffff811a43a8>] __alloc_pages_nodemask+0x288/0x2a0
Oct 11 05:10:33 ip-10-0-x-xxx kernel: [488908.756203] [<ffffffff811efc6c>] alloc_pages_current+0x8c/0x110
Oct 11 05:10:33 ip-10-0-x-xxx kernel: [488908.756205] [<ffffffff81199c5b>] __page_cache_alloc+0xab/0xc0
Oct 11 05:10:33 ip-10-0-x-xxx kernel: [488908.756208] [<ffffffff8119c580>] filemap_fault+0x160/0x440
Oct 11 05:10:33 ip-10-0-x-xxx kernel: [488908.756211] [<ffffffff812b3016>] ext4_filemap_fault+0x36/0x50
Oct 11 05:10:33 ip-10-0-x-xxx kernel: [488908.756215] [<ffffffff811c9d37>] __do_fault+0x77/0x110
Oct 11 05:10:33 ip-10-0-x-xxx kernel: [488908.756218] [<ffffffff811cdcdc>] handle_mm_fault+0x125c/0x1b80
Oct 11 05:10:33 ip-10-0-x-xxx kernel: [488908.756222] [<ffffffff81860041>] ? __schedule+0x301/0x810
Oct 11 05:10:33 ip-10-0-x-xxx kernel: [488908.756224] [<ffffffff8186004d>] ? __schedule+0x30d/0x810
Oct 11 05:10:33 ip-10-0-x-xxx kernel: [488908.756225] [<ffffffff81860041>] ? __schedule+0x301/0x810
Oct 11 05:10:33 ip-10-0-x-xxx kernel: [488908.756227] [<ffffffff8186004d>] ? __schedule+0x30d/0x810
Oct 11 05:10:33 ip-10-0-x-xxx kernel: [488908.756229] [<ffffffff81860041>] ? __schedule+0x301/0x810
Oct 11 05:10:33 ip-10-0-x-xxx kernel: [488908.756231] [<ffffffff81860041>] ? __schedule+0x301/0x810
Oct 11 05:10:33 ip-10-0-x-xxx kernel: [488908.756235] [<ffffffff8106edd4>] __do_page_fault+0x1a4/0x410
Oct 11 05:10:33 ip-10-0-x-xxx kernel: [488908.756237] [<ffffffff8106f062>] do_page_fault+0x22/0x30
Oct 11 05:10:33 ip-10-0-x-xxx kernel: [488908.756241] [<ffffffff81867ae8>] page_fault+0x28/0x30
Oct 11 05:10:33 ip-10-0-x-xxx kernel: [488908.756242] Mem-Info:
Oct 11 05:10:33 ip-10-0-x-xxx kernel: [488908.756247] active_anon:919500 inactive_anon:18067 isolated_anon:0
Oct 11 05:10:33 ip-10-0-x-xxx kernel: [488908.756247] active_file:620 inactive_file:764 isolated_file:0
Oct 11 05:10:33 ip-10-0-x-xxx kernel: [488908.756247] unevictable:0 dirty:0 writeback:0 unstable:0
Oct 11 05:10:33 ip-10-0-x-xxx kernel: [488908.756247] slab_reclaimable:8517 slab_unreclaimable:10839
Oct 11 05:10:33 ip-10-0-x-xxx kernel: [488908.756247] mapped:25419 shmem:36238 pagetables:21622 bounce:0
Oct 11 05:10:33 ip-10-0-x-xxx kernel: [488908.756247] free:21367 free_pcp:140 free_cma:0
Oct 11 05:10:33 ip-10-0-x-xxx kernel: [488908.756251] Node 0 DMA free:15820kB min:268kB low:332kB high:400kB active_anon:0kB inactive_anon:0kB active_file:8kB inactive_file:28kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15988kB managed:15900kB mlocked:0kB dirty:0kB writeback:0kB mapped:8kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Oct 11 05:10:33 ip-10-0-x-xxx kernel: [488908.756256] lowmem_reserve[]: 0 3704 3893 3893 3893
Oct 11 05:10:33 ip-10-0-x-xxx kernel: [488908.756260] Node 0 DMA32 free:64756kB min:64040kB low:80048kB high:96060kB active_anon:3513132kB inactive_anon:69560kB active_file:2472kB inactive_file:3028kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3915776kB managed:3834888kB mlocked:0kB dirty:0kB writeback:0kB mapped:95428kB shmem:137276kB slab_reclaimable:32652kB slab_unreclaimable:38928kB kernel_stack:4192kB pagetables:83352kB unstable:0kB bounce:0kB free_pcp:560kB local_pcp:400kB free_cma:0kB writeback_tmp:0kB pages_scanned:37356 all_unreclaimable? yes
Oct 11 05:10:33 ip-10-0-x-xxx kernel: [488908.756265] lowmem_reserve[]: 0 0 189 189 189
Oct 11 05:10:33 ip-10-0-x-xxx kernel: [488908.756268] Node 0 Normal free:4892kB min:3268kB low:4084kB high:4900kB active_anon:164868kB inactive_anon:2708kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:262144kB managed:193808kB mlocked:0kB dirty:0kB writeback:0kB mapped:6240kB shmem:7676kB slab_reclaimable:1416kB slab_unreclaimable:4428kB kernel_stack:720kB pagetables:3136kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Then I get an huge list of this:
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174008] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174009] 37420 total pagecache pages
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174011] 0 pages in swap cache
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174012] Swap cache stats: add 0, delete 0, find 0/0
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174013] Free swap = 0kB
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174014] Total swap = 0kB
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174015] 1048477 pages RAM
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174016] 0 pages HighMem/MovableOnly
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174017] 37328 pages reserved
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174018] 0 pages cma reserved
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174019] 0 pages hwpoisoned
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174020] [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174025] [ 275] 0 275 10189 689 21 3 0 0 systemd-journal
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174028] [ 303] 0 303 11224 309 23 3 0 -1000 systemd-udevd
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174030] [ 532] 0 532 7240 179 18 3 0 0 systemd-logind
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174032] [ 534] 0 534 7470 52 20 3 0 0 cgmanager
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174034] [ 536] 0 536 6512 49 18 3 0 0 atd
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174037] [ 544] 0 544 6933 71 18 3 0 0 cron
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174039] [ 550] 102 550 10762 176 25 3 0 -900 dbus-daemon
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174041] [ 600] 0 600 1099 16 7 3 0 0 acpid
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174043] [ 621] 0 621 68655 244 36 3 0 0 accounts-daemon
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174045] [ 626] 101 626 65157 313 29 3 0 0 rsyslogd
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174047] [ 670] 0 670 4869 65 15 3 0 0 irqbalance
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174050] [ 689] 0 689 13193 2548 29 3 0 0 munin-node
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174052] [ 691] 0 691 4031 216 11 3 0 0 dhclient
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174054] [ 698] 0 698 69861 342 39 3 0 0 polkitd
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174056] [ 735] 0 735 9667 180 22 3 0 0 monit
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174058] [ 1228] 0 1228 133721 6781 85 5 0 -500 dockerd
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174060] [ 1248] 0 1248 109978 1466 139 4 0 0 php-fpm7.2
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174063] [ 1253] 0 1253 43332 2030 52 3 0 0 unattended-upgr
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174065] [ 1255] 0 1255 91709 3642 139 3 0 0 php-fpm7.0
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174067] [ 1321] 107 1321 28735 291 53 3 0 0 opendkim
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174069] [ 1324] 0 1324 16379 177 35 3 0 -1000 sshd
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174072] [ 1327] 107 1327 88165 827 62 3 0 0 opendkim
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174074] [ 1352] 0 1352 43196 733 52 5 0 0 nginx
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174076] [ 1353] 33 1353 43196 733 54 5 0 0 nginx
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174078] [ 1354] 33 1354 43196 733 54 5 0 0 nginx
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174080] [ 1355] 33 1355 43196 733 54 5 0 0 nginx
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174087] [ 1362] 0 1362 3664 33 12 3 0 0 agetty
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174089] [ 1371] 33 1371 109978 1327 125 4 0 0 php-fpm7.2
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174091] [ 1372] 33 1372 109978 1327 125 4 0 0 php-fpm7.2
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174093] [ 1373] 33 1373 112297 9698 150 3 0 0 php-fpm7.0
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174095] [ 1374] 33 1374 112315 10265 151 3 0 0 php-fpm7.0
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174098] [ 1492] 0 1492 93681 2281 40 5 0 -500 docker-containe
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174100] [ 1506] 0 1506 16352 120 24 3 0 0 master
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174102] [ 1974] 111 1974 27509 166 26 3 0 0 ntpd
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174105] [27112] 33 27112 111777 8536 150 3 0 0 php-fpm7.0
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174107] [30935] 0 30935 146764 2750 54 4 0 0 fail2ban-server
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174109] [ 6574] 0 6574 122194 3926 179 4 0 0 apache2
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174111] [27782] 33 27782 44694 2331 70 3 0 0 apache2
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174114] [27811] 108 27811 23037 213 45 3 0 0 zabbix_agentd
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174116] [27812] 108 27812 23037 212 45 3 0 0 zabbix_agentd
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174118] [27813] 108 27813 23037 211 45 3 0 0 zabbix_agentd
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174120] [27814] 108 27814 23037 211 45 3 0 0 zabbix_agentd
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174122] [27815] 108 27815 23037 211 45 3 0 0 zabbix_agentd
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174124] [27816] 108 27816 23037 218 45 3 0 0 zabbix_agentd
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174127] [27291] 106 27291 16911 131 25 3 0 0 qmgr
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174129] [27413] 106 27413 20177 184 30 3 0 0 tlsmgr
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174131] [26417] 106 26417 16869 112 25 3 0 0 pickup
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174133] [27850] 33 27850 163843 26103 224 4 0 0 apache2
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174136] [27856] 33 27856 161347 23458 215 4 0 0 apache2
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174138] [27895] 33 27895 162572 24791 221 4 0 0 apache2
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174140] [28374] 33 28374 163657 33629 236 4 0 0 apache2
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174142] [28377] 33 28377 142967 31279 231 4 0 0 apache2
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174144] [28379] 33 28379 144883 32799 235 4 0 0 apache2
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174146] [28423] 33 28423 162756 24751 222 4 0 0 apache2
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174149] [28429] 33 28429 161587 31448 232 4 0 0 apache2
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174151] [28434] 33 28434 161233 23432 219 4 0 0 apache2
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174153] [28463] 33 28463 163658 33523 236 4 0 0 apache2
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174155] [28464] 33 28464 122401 2826 177 4 0 0 apache2
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174157] [28467] 33 28467 143221 31432 231 4 0 0 apache2
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174159] [28468] 33 28468 122473 2965 177 4 0 0 apache2
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174161] [28469] 33 28469 129756 26267 220 4 0 0 apache2
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174164] [28482] 33 28482 161283 22980 219 4 0 0 apache2
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174166] [28483] 33 28483 163618 25643 223 4 0 0 apache2
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174168] [28484] 33 28484 163367 25141 223 4 0 0 apache2
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174170] [28486] 33 28486 143209 31129 231 4 0 0 apache2
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174172] [28488] 33 28488 153425 24734 220 4 0 0 apache2
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174174] [28492] 33 28492 161351 23526 215 4 0 0 apache2
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174176] [28500] 33 28500 163574 25769 221 4 0 0 apache2
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174178] [28501] 33 28501 163577 26837 224 4 0 0 apache2
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174180] [28503] 33 28503 142863 23211 214 4 0 0 apache2
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174182] [28865] 33 28865 161160 23608 218 4 0 0 apache2
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174184] [28873] 33 28873 161345 23207 215 4 0 0 apache2
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174186] [28890] 33 28890 163621 25344 223 4 0 0 apache2
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174188] [28891] 33 28891 122250 2748 166 4 0 0 apache2
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174190] [28892] 33 28892 162593 24527 217 4 0 0 apache2
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174193] [28894] 33 28894 127346 7830 186 4 0 0 apache2
Oct 11 05:13:34 ip-10-0-x-xxx kernel: [489090.174195] [28899] 33 28899 163557 25511 223 4 0 0 apache2
Oct 11 05:14:05 ip-10-0-x-xxx kernel: [489120.936086] INFO: task apache2:28423 blocked for more than 120 seconds.
Oct 11 05:14:05 ip-10-0-x-xxx kernel: [489120.943206] Not tainted 4.4.0-157-generic #185-Ubuntu
Oct 11 05:14:05 ip-10-0-x-xxx kernel: [489120.948135] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 11 05:14:05 ip-10-0-x-xxx kernel: [489120.955958] apache2 D ffff8800e674fc58 0 28423 6574 0x00000000
Oct 11 05:14:05 ip-10-0-x-xxx kernel: [489120.955963] ffff8800e674fc58 0000000000000001 ffff880021adc600 ffff88001dac8000
Oct 11 05:14:05 ip-10-0-x-xxx kernel: [489120.955966] ffff8800e6750000 ffff880034195984 ffff88001dac8000 00000000ffffffff
Oct 11 05:14:05 ip-10-0-x-xxx kernel: [489120.955969] ffff880034195988 ffff8800e674fc70 ffffffff81860585 ffff880034195980
Oct 11 05:14:05 ip-10-0-x-xxx kernel: [489120.955971] Call Trace:
Oct 11 05:14:05 ip-10-0-x-xxx kernel: [489120.955981] [<ffffffff81860585>] schedule+0x35/0x80
Oct 11 05:14:05 ip-10-0-x-xxx kernel: [489120.955989] [<ffffffff818608de>] schedule_preempt_disabled+0xe/0x10
Oct 11 05:14:05 ip-10-0-x-xxx kernel: [489120.955992] [<ffffffff81862777>] __mutex_lock_slowpath+0xb7/0x130
Oct 11 05:14:05 ip-10-0-x-xxx kernel: [489120.955994] [<ffffffff8186280f>] mutex_lock+0x1f/0x30
Oct 11 05:14:05 ip-10-0-x-xxx kernel: [489120.955998] [<ffffffff8122bbf2>] path_openat+0x512/0x1360
Oct 11 05:14:05 ip-10-0-x-xxx kernel: [489120.956001] [<ffffffff8122e611>] do_filp_open+0x91/0x100
Oct 11 05:14:05 ip-10-0-x-xxx kernel: [489120.956004] [<ffffffff8123c2e7>] ? __alloc_fd+0xc7/0x190
Oct 11 05:14:05 ip-10-0-x-xxx kernel: [489120.956007] [<ffffffff8121c8c8>] do_sys_open+0x138/0x2a0
Oct 11 05:14:05 ip-10-0-x-xxx kernel: [489120.956009] [<ffffffff8121ca4e>] SyS_open+0x1e/0x20
Oct 11 05:14:05 ip-10-0-x-xxx kernel: [489120.956012] [<ffffffff81864f1b>] entry_SYSCALL_64_fastpath+0x22/0xcb
Oct 11 05:14:05 ip-10-0-x-xxx kernel: [489120.956026] INFO: task apache2:29254 blocked for more than 120 seconds.
Oct 11 05:14:05 ip-10-0-x-xxx kernel: [489120.962952] Not tainted 4.4.0-157-generic #185-Ubuntu
Oct 11 05:14:05 ip-10-0-x-xxx kernel: [489120.977383] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 11 05:14:05 ip-10-0-x-xxx kernel: [489120.992125] apache2 D ffff880013db3bf8 0 29254 6574 0x00000000
Oct 11 05:14:05 ip-10-0-x-xxx kernel: [489120.992131] ffff880013db3bf8 0000000000000000 ffffffff81e13500 ffff8800e8701c00
Oct 11 05:14:05 ip-10-0-x-xxx kernel: [489120.992135] ffff880013db4000 ffff880034195984 ffff8800e8701c00 00000000ffffffff
Oct 11 05:14:05 ip-10-0-x-xxx kernel: [489120.992137] ffff880034195988 ffff880013db3c10 ffffffff81860585 ffff880034195980
Oct 11 05:14:05 ip-10-0-x-xxx kernel: [489120.992140] Call Trace:
Possible reasons: Cron is running sessionclean at the same time as systemd starts clean php session files? It's simply not cleaning php sessions correctly? My RAM is too low?
free -m #see memory info
total used free shared buff/cache available
Mem: 3949 1710 1397 141 842 1793
Swap: 0 0 0
Also since I updated the OS my sites are not using php5 anymore but that's an cron job from before the update that migrated over, maybe that could be an issue? Thanks!
asked on Server Fault Oct 11, 2019 by 
paul9494
paul94941 Answer
You might want to double-check your php version. I have php7.0 as the bundled version in my Ubuntu 16.04 installs. IIRC, PHP5 has been deprecated/left-for-dead
Also, this suggests you're running out of RAM:
apache2 invoked oom-killer: gfp_mask=0x24201ca, order=0, oom_score_adj=0
Possibly add more RAM, disk swap, or at least a swap file and see if things improve.
Total swap = 0kB
answered on Server Fault Oct 11, 2019 by 
Server Fault
Server FaultUser contributions licensed under CC BY-SA 3.0