KernelCare error: kpatch: module 'mpt3sas' srcversion mismatch

Issue

When trying to run kcarectl -u, we get this error:

# kcarectl --update
Updates already downloaded
Unable to apply patch (/var/cache/kcare/patches/none-d1dd14386bea5957aeb5ae8a2b8a0a3be2e4e782-64-default/kpatch.bin 64 22 cloudlinux 4.18.0-513.5.1.lve.el8.x86_64, default, , , False)

Checking the logs, we see:

kernel: kpatch: module 'mpt3sas' srcversion mismatch 37BFFED214873BE24435411/AE9937FCEF105BF369AEB64...

Environment

• KernelCare

Solution

We need to ensure that the module is not loaded from an older kernel. In the error message, the correct srcversion is the first string before the slash ("/") and the one loaded on the server is the second:

...mismatch 37BFFED214873BE24435411/AE9937FCEF105BF369AEB64...

The srcversion we need is "37BFFED214873BE24435411", so we have to check that the module has it:

 # modinfo mpt3sas
filename:       /lib/modules/4.18.0-553.lve.el8.x86_64/kernel/drivers/scsi/mpt3sas/mpt3sas.ko.xz
alias:          mpt2sas
...
srcversion:     37BFFED214873BE24435411 

In some cases, like in the above example, the srcversion is correct, but this is the metadata on disk. We also need to check the value loaded in memory:

# cat /sys/module/mpt3sas/srcversion
AE9937FCEF105BF369AEB64 # Incorrect number

Then, we need to remove and reload the correct module. The best option is to rename/move redundant versions. The list of modules can be found with this command:

find /lib/modules -type f -name 'mpt3sas*.ko*' -printf '%p\n'

For example:

/lib/modules/4.18.0-553.lve.el8.x86_64/kernel/drivers/scsi/mpt3sas/mpt3sas.ko.xz
/lib/modules/4.18.0-372.9.1.el8.x86_64/extra/mpt3sas/mpt3sas.ko

The file to rename/remove is "/lib/modules/4.18.0-372.9.1.el8.x86_64/extra/mpt3sas/mpt3sas.ko" or any module that is not from the running kernel, then we proceed to reload it:

mv /lib/modules/4.18.0-372.9.1.el8.x86_64/extra/mpt3sas/mpt3sas.ko{,.bk}
yum -y reinstall kernel-modules*$(uname -r) 
modprobe -Rr mpt3sas
lsmod | grep mpt3sas # To ensure the module is not loaded. The command should produce no output
depmod -a 
modprobe -v mpt3sas
cat /sys/module/mpt3sas/srcversion # To verify, it should be 37BFFED214873BE24435411

If the module is in use, the only other option would be reboot the server at that point, and it should load the right version of the module.