Issue
Imunify service starts and fails shortly after. Python segfaults can be spotted in log messages like:
python3[12719]: segfault at 0 ip 00007f9d7ea73bc6 sp 00007f9d5d97f920
error 4 in libc-2.28.so[7f9d7e9ee000+1bc000]
Segfaults are sporadic, and there are many processes that fail. For Imunify tracebacks more often than not lead to thread.py than those do to other modules:
Fatal Python error: Segmentation fault
Current thread 0x00007fe23bfff700 (most recent call first):
<no Python frame>
Thread 0x00007fe241483700 (most recent call first):
File "/opt/alt/python38/lib64/python3.8/concurrent/futures/thread.py", line 78 in _worker
File "/opt/alt/python38/lib64/python3.8/threading.py", line 870 in Segmentation fault (core dumped)
Additional troubleshooting.
While the running service can be straced with:
/opt/alt/python38/bin/python3 /usr/bin/imunify-service
It makes sense to look at service uptime:
systemctl status imunify360
If it persists for an hour or more and no Python segfaults are in logs, this article is likely inapplicable. Just bear in mind that segfaults are not only due to bad code but can be caused by the kernel.
Environment
- ImunifyAV
- Imunify360
- Python3
- libc.so
Solution
To fix the error, it is necessary to update the kernel on the server and reboot the server with the new kernel after that. Please, gather more info on this (doctor), the root causes are not well understood.
1. Use the package manager features and the uname -a commands to check the loaded kernel version.
2. Check pending updates with:
needs-restarting -r Pending packages or their absence doesn't automatically mean that the article is not applicable.
3. In regard to how KernelCare manages updates, there is also a simple tool to detect outdated shared libraries still linked to processes in memory. The script run will look like this:
$ curl -s -L https://kernelcare.com/uchecker | sudo python
[*] Process httpd[15516] linked to the `libc-2.17.so` that is not up to date.
We also can quickly unload patches and check whether the issue is related to KCare:
kcarectl --unload
And to switch back on:
kcarectl --update
kcarectl --patch-info
Cause
Segfaults like SEGV_MAPERR can also be caused by the kernel managing the task sizes while kernel update is pended. For example, when "yum update" was executed but the loaded kernel is not yet new.
Useful links
- https://coder-question.com/cq-blog/885128
- https://github.com/cloudlinux/kcare-uchecker
- https://cloudlinux.zendesk.com/agent/tickets/162729
- https://www.kernel.org/doc/gorman/pdf/understand.pdf
Comments
0 comments
Please sign in to leave a comment.