Issues with syncing large IP lists between server groups and how to resolve them

Issue

The Imunify's console log may contain the following errors coinciding with attempts to sync IP lists between server groups:

Hash is full, cannot add more elements

Environment

• Imunify 6.12.4-1
• Group synchronization

Solution

This feature was designed to sync manually made changes in lists between the server's fleet and wasn't designed for bulk updates. To effectively manage more than 100,000 IPs in your lists, we recommend switching to external Black/White lists. Such lists support up to 500k elements.

1. To use external files as a source for the list, it is required to place a TXT file into one of the directories as per: 

/etc/imunify360/whitelist/*.txt 
/etc/imunify360/blacklist/*.txt 

2. And use reload command:

imunify360-agent reload-lists

• It is possible to include notes for each IP or a subnet in the list but it is required to follow a format and use a # symbol delimiter to separate the IP addresses and the notes. The provided examples show the correct format and incorrect formats that won't be parsed correctly:

mkdir /etc/imunify360/blacklist/
cat /etc/imunify360/blacklist/external500kipset.txt

192.168.1.1 #1 firehole list, comments with space OK
#one line comment is OK
192.168.1.2#2 multipleple DDoS, #nospace is OK
192.168.99.0/24 #subnets are OK
2001:0db8:85a3:0000:0000:8a2e:0370:7334 #more IP in ipv6 OK

•  While the below examples will not be parsed correctly:

192.168.1.3, will not be in the list #nope
192.168.1.4,192.168.1.5 #nope
192.168.1.6 192.168.1.7 #nope

• As well as it is required to follow CIDR rules:

192.168.1.11-192.168.1.21 #nope
192.168.1.22/24 #incorrectly defined subnet address #nope

Cause

The problem occurs with large lists of IPs when the set limit of a single IP list is exceeded. If the number of IPs surpasses the 100,000 IPs per list (black/white) for the group synchronization the syncing between server groups will not work reliably and fail to process lists longer than the limit.

Useful links

• https://docs.imunify360.com/command_line_interface/#reload-lists
• https://docs.imunify360.com/faq_and_known_issues/#_26-ipset-max-elements-error-hash-is-full-cannot-add-more-elements