High number of sleeping check-detached processes executed by cron/log size issue

Issue

it was identified that cron task to check the detached scans piles up processes due to Ubuntu nc build peculiarity while handling end of file and no timeout provided. Such processes can't finish on such systems when executed as provided by our cron:

cat /etc/cron.d/imunify-antivirus | grep detached
*/5 * * * * root echo '{"command": ["malware", "on-demand", "check-detached"], "params": {}}' | nc -U /var/run/defence360agent/simple_rpc.sock > /dev/null 2>&1 || :

So that each 5 minutes cron task was executed and hanging after, resulting in high number of sleeping process during the runtime:

ps aux | grep defence360agent | wc -l
1405

Such issue may also lead to garbage logs entries increasing the imunify-error log size.

While there are no hanging for agent command:

time imunify360-agent malware on-demand check-detached
ERROR: imunify360 service is running.

real 0m1.166s
user 0m1.073s
sys 0m0.091s

echo $?
11

Environment

• Imunify360 7.5.0 (fixed release is 7.5.3)
• Ubuntu 20.04/22.04
• cron task with nc -U

Solution

The patch is going to be delivered in 7.5.3 version with DEF-25815.

1. Fix the cron task to use the socat tool instead of nc with the parameters as per:

cat /etc/cron.d/imunify-antivirus | grep detached
*/5 * * * *    root    echo '{"command": ["malware", "on-demand", "check-detached"], "params": {}}' | socat -T 5 - UNIX-CONNECT:/var/run/defence360agent/simple_rpc.sock > /dev/null 2>&1 || :

As there are quoting issues when using sed to make the change, to pursue the same fix with onliner it is also possible to rewrite the file in full as per:

cat > /etc/cron.d/imunify-antivirus <<EOF
PATH=/usr/local/sbin:/usr/sbin:/sbin:/usr/local/bin:/usr/bin:/bin

# Every Saturday at 1:25
25 1 * * 6     root    /usr/bin/tmpwatch 168 /var/imunify360/tmp
# Every 5 minutes. Ignore "ERROR: imunify360 service is running."
*/5 * * * *    root    echo '{"command": ["malware", "on-demand", "check-detached"], "params": {}}' | socat -T 5 - UNIX-CONNECT:/var/run/defence360agent/simple_rpc.sock > /dev/null 2>&1 || :
17 4 * * *     root    /usr/libexec/report-command-error /opt/imunify360/venv/share/imunify360/scripts/update_components_versions.py > /dev/null 2>&1
EOF

2. Also required to kill hanging processes:

ps aux | grep defence360agent | grep -v grep | awk '{print $2}' | xargs kill -9

With this the processes are expected to normalize:

ps aux | grep defence360agent | grep -v grep | wc -l
0

Cause

The culprit is seemingly is in the fact that nc can't handle end of file for the socket and such connections hang. Cron task with direct socket communication seems like speed optimisation that skips Python inits, yet nc command doesn't have a timeout value.