Process is stuck when generating core with gcore

2

I have a multi-threaded process for which I wanted to generate core dump. gcore ran, gdb started, the process went to "t" state according to ps. However, it got stuck there.

As it was already being traced, I could not attach another gdb session to see what was going on. I dumped kernel threads, some process threads were in 'ptrace_stop' while others were working.

Any ideas on why gcore got stuck would be helpful. The system had enough free memory too (free reported 300MB, but also reported swap had kicked in - though not sure when).

The process is running inside a docker container on a virtualbox vm.

I ran:

gcore process_id

That seems to have spawned a gdb child process

root 26540 0.0 0.0 4500 660 ? Ss May20 0:00 /bin/sh /usr/bin/gcore -o /var/cores/core.proc_name 2539

root 26545 0.0 2.9 160896 105144 ? S May20 0:01 /usr/bin/gdb --nx --batch -ex set pagination off -ex set height 0 -ex set width 0 -ex attach 2539 -ex gcore /var/cores/core.proc_name.2539 -ex detach -ex quit

May 22 12:08:24 BaseVM kernel: Call Trace:
May 22 12:08:24 BaseVM kernel:  [<ffffffff8163bb39>] schedule+0x29/0x70
May 22 12:08:24 BaseVM kernel:  [<ffffffff81639829>] schedule_timeout+0x209/0x2d0
May 22 12:08:24 BaseVM kernel:  [<ffffffff811c4268>] ? __kmalloc_node_track_caller+0x58/0x270
May 22 12:08:24 BaseVM kernel:  [<ffffffff8151a7f6>] ? skb_release_data+0xd6/0x110
May 22 12:08:24 BaseVM kernel:  [<ffffffff810a68a6>] ? prepare_to_wait+0x56/0x90
May 22 12:08:24 BaseVM kernel:  [<ffffffff815d372f>] unix_stream_read_generic+0x30f/0x8e0
May 22 12:08:24 BaseVM kernel:  [<ffffffff810a6b80>] ? wake_up_atomic_t+0x30/0x30
May 22 12:08:24 BaseVM kernel:  [<ffffffff8152158d>] ? __scm_destroy+0x4d/0x60
May 22 12:08:24 BaseVM kernel:  [<ffffffff815d29f3>] ? unix_stream_sendmsg+0x413/0x430
May 22 12:08:24 BaseVM kernel:  [<ffffffff815d3df4>] unix_stream_recvmsg+0x54/0x70
May 22 12:08:24 BaseVM kernel:  [<ffffffff815d0d90>] ? unix_state_double_unlock+0x50/0x50
May 22 12:08:24 BaseVM kernel:  [<ffffffff8151199f>] sock_recvmsg+0xbf/0x100
May 22 12:08:24 BaseVM kernel:  [<ffffffff81511cae>] ___sys_recvmsg+0x11e/0x2b0
May 22 12:08:24 BaseVM kernel:  [<ffffffff811f05db>] ? do_filp_open+0x4b/0xb0
May 22 12:08:24 BaseVM kernel:  [<ffffffff81512861>] __sys_recvmsg+0x51/0x90
May 22 12:08:24 BaseVM kernel:  [<ffffffff815128b2>] SyS_recvmsg+0x12/0x20
May 22 12:08:24 BaseVM kernel:  [<ffffffff81646b49>] system_call_fastpath+0x16/0x1b
May 22 12:08:24 BaseVM kernel: proc_name     t ffff880212ce5080     0  8081   8003 0x00000082
May 22 12:08:24 BaseVM kernel:  ffff8800c97e3d50 0000000000000082 ffff880212ce5080 ffff8800c97e3fd8
May 22 12:08:24 BaseVM kernel:  ffff8800c97e3fd8 ffff8800c97e3fd8 ffff880212ce5080 ffff880212ce5080
May 22 12:08:24 BaseVM kernel:  ffff880212ce5080 0000000000000000 ffff880212ce5080 ffff880212ce5080
May 22 12:08:24 BaseVM kernel: Call Trace:
May 22 12:08:24 BaseVM kernel:  [<ffffffff8163bb39>] schedule+0x29/0x70
May 22 12:08:24 BaseVM kernel:  [<ffffffff810912fd>] ptrace_stop+0x16d/0x2b0
May 22 12:08:24 BaseVM kernel:  [<ffffffff81092ebd>] get_signal_to_deliver+0x3dd/0x6d0
May 22 12:08:24 BaseVM kernel:  [<ffffffff81014417>] do_signal+0x57/0x6c0
May 22 12:08:24 BaseVM kernel:  [<ffffffff81014adf>] do_notify_resume+0x5f/0xb0
May 22 12:08:24 BaseVM kernel:  [<ffffffff81646dfd>] int_signal+0x12/0x17
May 22 12:08:24 BaseVM kernel: proc-name-ust  t ffff880213f87300     0  8103   8003 0x00000082
May 22 12:08:24 BaseVM kernel:  ffff8800c1c2fd50 0000000000000082 ffff880213f87300 ffff8800c1c2ffd8
May 22 12:08:24 BaseVM kernel:  ffff8800c1c2ffd8 ffff8800c1c2ffd8 ffff880213f87300 ffff880213f87300
May 22 12:08:24 BaseVM kernel:  ffff880213f87300 0000000000000000 ffff880213f87300 ffff880213f87300
May 22 12:08:24 BaseVM kernel: Call Trace:
May 22 12:08:24 BaseVM kernel:  [<ffffffff8163bb39>] schedule+0x29/0x70
May 22 12:08:24 BaseVM kernel:  [<ffffffff810912fd>] ptrace_stop+0x16d/0x2b0
May 22 12:08:24 BaseVM kernel:  [<ffffffff81092ebd>] get_signal_to_deliver+0x3dd/0x6d0
May 22 12:08:24 BaseVM kernel:  [<ffffffff810e2860>] ? futex_wake+0x80/0x160
May 22 12:08:24 BaseVM kernel:  [<ffffffff81014417>] do_signal+0x57/0x6c0
May 22 12:08:24 BaseVM kernel:  [<ffffffff8108fe7b>] ? recalc_sigpending+0x1b/0x50
May 22 12:08:24 BaseVM kernel:  [<ffffffff81014adf>] do_notify_resume+0x5f/0xb0
May 22 12:08:24 BaseVM kernel:  [<ffffffff81646dfd>] int_signal+0x12/0x17
May 22 12:08:24 BaseVM kernel: proc-name-ust  S ffffc90000dd5c00     0  8104   8003 0x00000080
May 22 12:08:24 BaseVM kernel:  ffff8800c1c03cd0 0000000000000082 ffff880213f85080 ffff8800c1c03fd8
May 22 12:08:24 BaseVM kernel:  ffff8800c1c03fd8 ffff8800c1c03fd8 ffff880213f85080 ffff880213f85080
May 22 12:08:24 BaseVM kernel:  ffff880213f85080 0000000000000000 ffff8800c1c03de0 ffffc90000dd5c00
May 22 12:08:24 BaseVM kernel: Call Trace:
May 22 12:08:24 BaseVM kernel:  [<ffffffff8163bb39>] schedule+0x29/0x70
May 22 12:08:24 BaseVM kernel:  [<ffffffff810e2704>] futex_wait_queue_me+0xc4/0x120
May 22 12:08:24 BaseVM kernel:  [<ffffffff810e3279>] futex_wait+0x179/0x280
May 22 12:08:24 BaseVM kernel:  [<ffffffff8108ffa6>] ? dequeue_signal+0x86/0x180
May 22 12:08:24 BaseVM kernel:  [<ffffffff81092bdf>] ? get_signal_to_deliver+0xff/0x6d0
May 22 12:08:24 BaseVM kernel:  [<ffffffff810e530e>] do_futex+0xfe/0x5b0
May 22 12:08:24 BaseVM kernel:  [<ffffffff8164205d>] ? __do_page_fault+0x16d/0x450
May 22 12:08:24 BaseVM kernel:  [<ffffffff810e5840>] SyS_futex+0x80/0x180
May 22 12:08:24 BaseVM kernel:  [<ffffffff81014afa>] ? do_notify_resume+0x7a/0xb0
May 22 12:08:24 BaseVM kernel:  [<ffffffff81646b49>] system_call_fastpath+0x16/0x1b
May 22 12:08:24 BaseVM kernel: proc-name      t ffff880213f82280     0  8107   8003 0x00000082
May 22 12:08:24 BaseVM kernel:  ffff8800c7dcfd50 0000000000000082 ffff880213f82280 ffff8800c7dcffd8
May 22 12:08:24 BaseVM kernel:  ffff8800c7dcffd8 ffff8800c7dcffd8 ffff880213f82280 ffff880213f82280
May 22 12:08:24 BaseVM kernel:  ffff880213f82280 0000000000000000 ffff880213f82280 ffff880213f82280
linux
linux-kernel
pthreads
virtualbox
gcore
asked on Stack Overflow May 23, 2017 by learner • edited May 23, 2017 by learner

0 Answers

Nobody has answered this question yet.


User contributions licensed under CC BY-SA 3.0