Ubuntu with XFS blocked for more than 120 seconds

Im running ubuntu 16.04 with an XFS file system volume. Its running on Google cloud. This Disk is a Local SSD. It is used for MongoDB.

A few days back my Primary and Secondary both are frozen from IO. This is error message.

nvme nvme0: I/O 593 QID 1 timeout, aborting
nvme nvme0: Abort status: 0x0
nvme nvme0: I/O 397 QID 8 timeout, aborting
nvme nvme0: Abort status: 0x0
nvme nvme0: I/O 593 QID 1 timeout, reset controller
nvme nvme0: I/O 593 QID 1 timeout, disable controller
INFO: task xfsaild/nvme0n1:3514 blocked for more 
      Not tainted 4.15.0-1032-gcp #34~16.04.1-Ubuntu
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" 
xfsaild/nvme0n1 D    0  3514      2 0x80000000
Call Trace:
 __schedule+0x3db/0x8b0
 schedule+0x36/0x80
 _xfs_log_force+0x145/0x290 [xfs]
 ? wake_up_q+0x70/0x70
 ? xfsaild+0x1bf/0x7b0 [xfs]
 xfs_log_force+0x2c/0x80 [xfs]
 xfsaild+0x1bf/0x7b0 [xfs]
 kthread+0x105/0x140
 ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
 ? kthread+0x105/0x140
 ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
 ? kthread_destroy_worker+0x50/0x50
 ret_from_fork+0x3a/0x50
INFO: task mongod:8655 blocked for more than 120 
      Not tainted 4.15.0-1032-gcp #34~16.04.1-Ubuntu
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" 
mongod          D    0  8655      1 0x00000000
Call Trace:
 __schedule+0x3db/0x8b0
 schedule+0x36/0x80
 rwsem_down_write_failed+0x214/0x3b0
 ? xfs_file_buffered_aio_write+0x79/0x2a0 [xfs]
 call_rwsem_down_write_failed+0x17/0x30
 ? call_rwsem_down_write_failed+0x17/0x30
 down_write+0x2d/0x40
 xfs_ilock+0xbf/0x130 [xfs]
 xfs_file_buffered_aio_write+0x79/0x2a0 [xfs]
 xfs_file_write_iter+0x103/0x150 [xfs]
 new_sync_write+0xe5/0x140
 __vfs_write+0x29/0x40
 vfs_write+0xb8/0x1b0
 ? __do_page_fault+0x282/0x500
 SyS_pwrite64+0x95/0xb0
 do_syscall_64+0x7b/0x150
 entry_SYSCALL_64_after_hwframe+0x42/0xb7
RSP: 002b:00007fb7068f4240 EFLAGS: 00000293 ORIG_RAX: 0000000000000012
RAX: ffffffffffffffda RBX: 0000000000002000 RCX: 00007fb709cd6da3
[38954361.184215] RDX: 0000000000002000 RSI: 000055b752eb9000 RDI: 0000000000000565
[38954361.184215] RBP: 00007fb7068f4290 R08: 000055b752eb9000 R09: 00000000000f3000
[38954361.184216] R10: 00000000000f3000 R11: 0000000000000293 R12: 0000000000002000
[38954361.184217] R13: 000055b752eb9000 R14: 000055b2105643c0 R15: 00000000000f3000
[38954361.184219] INFO: task mongod:8657 blocked for more than 120 seconds.

I'm not sure how this happened at the same timing.

Also what might be the reason behind this?