Ubuntu with XFS blocked for more than 120 seconds

0

Im running ubuntu 16.04 with an XFS file system volume. Its running on Google cloud. This Disk is a Local SSD. It is used for MongoDB.

A few days back my Primary and Secondary both are frozen from IO. This is error message.

nvme nvme0: I/O 593 QID 1 timeout, aborting
nvme nvme0: Abort status: 0x0
nvme nvme0: I/O 397 QID 8 timeout, aborting
nvme nvme0: Abort status: 0x0
nvme nvme0: I/O 593 QID 1 timeout, reset controller
nvme nvme0: I/O 593 QID 1 timeout, disable controller
INFO: task xfsaild/nvme0n1:3514 blocked for more 
      Not tainted 4.15.0-1032-gcp #34~16.04.1-Ubuntu
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" 
xfsaild/nvme0n1 D    0  3514      2 0x80000000
Call Trace:
 __schedule+0x3db/0x8b0
 schedule+0x36/0x80
 _xfs_log_force+0x145/0x290 [xfs]
 ? wake_up_q+0x70/0x70
 ? xfsaild+0x1bf/0x7b0 [xfs]
 xfs_log_force+0x2c/0x80 [xfs]
 xfsaild+0x1bf/0x7b0 [xfs]
 kthread+0x105/0x140
 ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
 ? kthread+0x105/0x140
 ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
 ? kthread_destroy_worker+0x50/0x50
 ret_from_fork+0x3a/0x50
INFO: task mongod:8655 blocked for more than 120 
      Not tainted 4.15.0-1032-gcp #34~16.04.1-Ubuntu
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" 
mongod          D    0  8655      1 0x00000000
Call Trace:
 __schedule+0x3db/0x8b0
 schedule+0x36/0x80
 rwsem_down_write_failed+0x214/0x3b0
 ? xfs_file_buffered_aio_write+0x79/0x2a0 [xfs]
 call_rwsem_down_write_failed+0x17/0x30
 ? call_rwsem_down_write_failed+0x17/0x30
 down_write+0x2d/0x40
 xfs_ilock+0xbf/0x130 [xfs]
 xfs_file_buffered_aio_write+0x79/0x2a0 [xfs]
 xfs_file_write_iter+0x103/0x150 [xfs]
 new_sync_write+0xe5/0x140
 __vfs_write+0x29/0x40
 vfs_write+0xb8/0x1b0
 ? __do_page_fault+0x282/0x500
 SyS_pwrite64+0x95/0xb0
 do_syscall_64+0x7b/0x150
 entry_SYSCALL_64_after_hwframe+0x42/0xb7
RSP: 002b:00007fb7068f4240 EFLAGS: 00000293 ORIG_RAX: 0000000000000012
RAX: ffffffffffffffda RBX: 0000000000002000 RCX: 00007fb709cd6da3
[38954361.184215] RDX: 0000000000002000 RSI: 000055b752eb9000 RDI: 0000000000000565
[38954361.184215] RBP: 00007fb7068f4290 R08: 000055b752eb9000 R09: 00000000000f3000
[38954361.184216] R10: 00000000000f3000 R11: 0000000000000293 R12: 0000000000002000
[38954361.184217] R13: 000055b752eb9000 R14: 000055b2105643c0 R15: 00000000000f3000
[38954361.184219] INFO: task mongod:8657 blocked for more than 120 seconds.

I'm not sure how this happened at the same timing.

Also what might be the reason behind this?

ubuntu
mongodb
xfs
asked on Server Fault Aug 27, 2020 by Bhuvanesh

0 Answers

Nobody has answered this question yet.


User contributions licensed under CC BY-SA 3.0