How to limit a known crawler bot?

Issue

Imunify360 has security mitigation mechanisms to avoid known crawler bots from an un-legitimate source. We fight hard against fake crawlers with WAF rules. Sometimes it might be needed to limit requests that come from a legitimate source

Environment

• Imunify360

Solution

You can use a proper custom rule to handle such abuse requests.

1. Check the ModSec config files hooked up to the webserver by using the following command:

# apachectl -t -D DUMP_INCLUDES | grep modsec

2. Pick up one file out of the Imunify360 rules directory, so it can avoid overrides on our usual auto-rules update/new im360 ruleset releases rollout process. For example, on cPanel, we recommend using the following file:

/etc/apache2/conf.d/modsec/modsec2.user.conf

 3. Add the following prepared custom rule to it:

SecRule REQUEST_URI "@contains sitemap" "chain,id:88722311,phase:1,t:none,t:lowercase,pass,severity:2,msg:'IM360 WAF: Found bing crawler ||T:{}||',tag:'service_i360_custom'"
SecRule REQUEST_HEADERS:User-Agent "@contains Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)" "t:none"

You can customize it further, maybe to check a file against a list of User-Agent you want to limit. This is a sample rule you can evolve. 

4. Finally, restart your webserver.
 
The custom rule will check the request URI has the "sitemap" string and request headers against the UA bot string. If matched, after two requests, it will add the attacker IP into the ipset/greylist, so that, the traffic is blocked and further requests are redirected to WebShield which will deliver a Captcha challenge. Any requests will be blocked until WebShield allows it either by solving the challenge or by hitting the TTL. This approach is preferable. If you want to loosen up, you can tweak the severity – 2 means after two consecutive requests. You can increase it to 5, for example.

Another option is switching the rule to block mode. It will block the request right away and return a 403 HTTP denied status code.

Additionally, you can also have a mechanism to loosen up and grip on the rule by:

imunify360-agent config update '{"MOD_SEC_BLOCK_BY_CUSTOM_RULE": {"88722311": {"check_period": 60,"max_incidents": 15}}}'