Realtime scanner creates a high i/o load on the disk

Issue

On a server with Imunify360, the imunify-realtime-av process and its corresponding ai-bolit process create a high i/o load in the system.

Total DISK READ : 71.54 M/s | Total DISK WRITE : 585.56 K/s
Actual DISK READ: 72.50 M/s | Actual DISK WRITE: 552.48 K/s
TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND 
3070224 be/4 root 3.86 M/s 0.00 B/s 0.00 % 99.99 % php -n -d short_open_tag=on -d extension=~ss 2224429611020973032 --create-shared-mem

Total DISK READ : 25.34 M/s | Total DISK WRITE : 245.80 K/s
Actual DISK READ: 36.83 M/s | Actual DISK WRITE: 432.91 K/s
TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND 
1569355 be/4 root 9.52 M/s 0.00 B/s 0.00 % 97.35 % imunify-realtime-av

Environment

• Imunify360
• realtime scanner
• ai-bolit

Solution

It is necessary to check whether an abnormally large number of incoming tasks have been created for ai-bolit in the directory /var/imunify360/aibolit/resident/in/notify-jobs

# ls -la /var/imunify360/aibolit/resident/in/
total 1001592
drwxr-xr-x 4 root root 4096 Jan 9 13:39 .
drwxr-xr-x 4 root root 4096 Jan 9 13:39 ..
drwx------ 2 root root 1025257472 Jan 17 03:29 notify-jobs
drwxrwxrwt 2 root root 4096 Jan 17 03:17 upload-jobs

In case the directory is full and the number of files in it cannot be obtained, it is possible to try clearing or renaming it and restarting the realtime services.

# systemctl stop imunify-realtime-av.service
# systemctl stop aibolit-resident.service
# systemctl stop aibolit-resident.socket

# cd /var/imunify360/aibolit/resident/in/
# mkdir notify-jobs.new
# chmod 700 notify-jobs.new
# mv notify-jobs notify-jobs.backup
# mv notify-jobs.new notify-jobs

# systemctl restart aibolit-resident.socket
# systemctl restart aibolit-resident.service
# systemctl restart imunify-realtime-av.service

The situation with the i/o on the server should normalize after these actions.

# ls -la /var/imunify360/aibolit/resident/in/
total 1001592
drwxr-xr-x 4 root root 4096 Jan 9 13:39 .
drwxr-xr-x 4 root root 4096 Jan 9 13:39 ..
drwx------ 2 root root 4096 Jan 17 03:45 notify-jobs
drwx------ 2 root root 1025257472 Jan 17 03:29 notify-jobs.backup
drwxrwxrwt 2 root root 4096 Jan 17 03:17 upload-jobs

The directory /var/imunify360/aibolit/resident/in/notify-jobs.backup can be deleted later.

Useful links

• https://docs.imunify360.com/dashboard/#general-2