NFS severe slowdown and hung nfsiod threads after KernelCare live patch on RHEL 8–based systems

Issue

After applying the latest KernelCare live patch, NFS performance degrades severely, effectively reducing NFS operations to a crawling speed. 

• The system remains responsive at a basic level, but all NFS-related operations stall or take an extremely long time to complete.

root     3035561  0.0  0.0      0     0 ?        D    14:48   0:03  \_ [kworker/u64:8+nfsiod]
root     3066728  0.0  0.0      0     0 ?        D    15:04   0:03  \_ [kworker/u64:2+nfsiod]
root     3099278  0.0  0.0      0     0 ?        D    15:22   0:03  \_ [kworker/u64:3+nfsiod]
[-----"-----]
root     3237162  0.0  0.0      0     0 ?        D    16:41   0:00  \_ [kworker/u64:254+nfsiod]
root     3240169  0.0  0.0      0     0 ?        D    16:42   0:00  \_ [kworker/u64:256+nfsiod]
root     3240176  0.0  0.0      0     0 ?        D    16:42   0:00  \_ [kworker/u64:257+nfsiod]

• Hung task events are observed (which pointed to kpatch):

Jan 20 12:34:01 ceres kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 20 12:34:01 ceres kernel: task:kworker/u64:1   state:D stack:0     pid:415608 ppid:2      flags:0x80004080
Jan 20 12:34:01 ceres kernel: Workqueue: nfsiod rpc_async_release [sunrpc]
Jan 20 12:34:01 ceres kernel: Call Trace:
Jan 20 12:34:01 ceres kernel: __schedule+0x2d1/0x870
Jan 20 12:34:01 ceres kernel: ? bit_wait_io+0x50/0x50
Jan 20 12:34:01 ceres kernel: schedule+0x55/0xf0
Jan 20 12:34:01 ceres kernel: bit_wait+0xd/0x50
Jan 20 12:34:01 ceres kernel: __wait_on_bit_lock+0x5e/0xb0
Jan 20 12:34:01 ceres kernel: out_of_line_wait_on_bit_lock+0x91/0xb0
Jan 20 12:34:01 ceres kernel: ? var_wake_function+0x30/0x30
Jan 20 12:34:01 ceres kernel: nfs_page_group_lock+0xe/0x30 [nfs]
Jan 20 12:34:01 ceres kernel: nfs_page_group_sync_on_bit+0x11/0x90 [nfs]
Jan 20 12:34:01 ceres kernel: nfs_inode_remove_request.kpatch+0x58/0x11c [nfs]
Jan 20 12:34:01 ceres kernel: ? nfs_write_completion+0xab/0x230 [nfs]
Jan 20 12:34:01 ceres kernel: ? rpc_free_task+0x39/0x60 [sunrpc]
Jan 20 12:34:01 ceres kernel: ? rpc_async_release+0x29/0x40 [sunrpc]
Jan 20 12:34:01 ceres kernel: ? process_one_work+0x1d3/0x390
Jan 20 12:34:01 ceres kernel: ? worker_thread+0x30/0x390
Jan 20 12:34:01 ceres kernel: ? process_one_work+0x390/0x390
Jan 20 12:34:01 ceres kernel: ? kthread+0x134/0x150
Jan 20 12:34:01 ceres kernel: ? set_kthread_struct+0x50/0x50
Jan 20 12:34:01 ceres kernel: ? ret_from_fork+0x1f/0x40

Environment

• KernelCare
• Filesystem NFS
• CloudLinux 8 

Solution

We've identified multiple reports of NFS-related problems affecting CloudLinux 8. As a precautionary measure, the latest CL8 release has been temporarily help back, and the same approach will be applied to other recent EL8-based distributions.

The team is still assessing the overall impact, so additional rollbacks may be necessary. 

In the meantime, please avoid manually reverting patches unless it is absolutely required.

Useful links

For updates, please refer our status page at https://cloudlinux.statuspage.io/incidents/xk8v2d5gvbrt