404 bot attack is one of the powerful weapons of hackers.
But, that’s where Fail2ban comes for your help.
We can use Fail2ban to block malicious 404 scans and invalid requests on a web server such as Apache.
At Bobcares, we help server owners setup Fail2ban to block 404 bot attacks on their servers as part of our Server Management Services.
Today, let’s discuss how we setup Fail2ban to block 404 bot attacks on Apache web server.
What’s Apache 404 error and how Fail2ban helps here?
A 404 is an HTTP status code which indicates that the requested web page couldn’t be found on the server. In other words, the requested webpage doesn’t exist or broken.
And, hackers use this as a DDoS tool. They generate thousands of requests in a minute to the non existing web pages which result in 404 errors like this.
xxx.xxx.xx.xx - - [11/Nov/2018:22:25:01 +0100] xxx.xxx.x.x "POST /xx23456.php HTTP/1.1" 404 209 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:57.0) Gecko/20100101 Firefox/57.0" xxx.xxx.xx.xx - - [11/Nov/2018:22:25:03 +0100] xxx.xxx.x.x "POST /xx123.php HTTP/1.1" 404 210 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:57.0) Gecko/20100101 Firefox/57.0" xxx.xxx.xx.xx - - [11/Nov/2018:22:25:05 +0100] xxx.xxx.x.x "POST /xxx.php HTTP/1.1" 404 206 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:57.0) Gecko/20100101 Firefox/57.0"
But, the web server has to process these much requests in a minute putting the server on stress.
That’s where Fail2ban plays it’s role. Fail2ban constantly monitors the Apache logs. Once it identifies an unusual behavior, it blocks such repeating offenders using temporary firewall rules. In other words, there are custom fail2ban jails which monitor the log files for malicious 404 patterns and block those IPs in firewall.
Our Server Experts help customers installing fail2ban and custom jails on their servers. For example, we install fail2ban using the below command on an Ubuntu server.
apt-get install fail2ban
And, we make further modifications in the configuration file /etc/failban/jail.local.
How to setup Fail2ban to detect Apache 404 attacks?
Now, let’s see how our Dedicated Engineers setup Fail2ban to block 404 scans and invalid request methods.
1) Create filter
Firstly, our Support Engineers create a filter in the location /etc/fail2ban/filter.d. Further, we add a set of rules to ban IPs that cause 404 errors.
For example, to monitor the Apache 404 requests, we create a filter file apache-404.conf in the location /etc/fail2ban/filter.d. The filter looks like this.
failregex = ^<HOST> - .* "(GET|POST|HEAD).*HTTP.*" 404 .*$ ignoreregex =.*(robots.txt|favicon.ico|jpg|png)
We define the regular expression to be matched under the failregex parameter. Here, the above regular expression identifies the IP address that is making too many 404 requests. And, the ignoreregex excludes the valid files such as robots.txt, favicon.io and the images.
2) Create a custom jail
Secondly, we add a new jail in the location /etc/fail2ban/jail.conf. This defines the Apache log path to be checked, maxretry, bantime etc.
For example, to create a custom jail that monitors the Apache 404 requests, we add the following code in the file /etc/fail2ban/jail.conf.
[apache-404] enabled = true port = http,https filter = apache-404 logpath = /var/log/httpd/error_log logpath = /var/log/httpd/access_log bantime = 3600 findtime = 600 maxretry = 5
Here, we update the apache log file under logpath parameter. Similarly, the bantime, species how many seconds an offending IP is banned for. We always set this value to an optimum level, so that it’s not short to affect the legitimate users, while not long enough favoring malicious users.
Further, the maxretry parameter specifies the total number of connection attempts. So, if a client makes retry attempts more than maxretry value within the time specified in findtime parameter, they will be banned.
Similarly, on Plesk servers, we create new jails from Tools & Settings > IP address Banning > Jails > Add Jail.
Finally, we restart fail2ban for the changes to take effect.
3) Test Fail2ban rules
Further, our Server Experts test the new Fail2ban rules on the server to ensure that they work as expected. For example, here we test the fail2ban filter apache-404.conf using the below command.
fail2ban-regex /var/log/httpd/error_log /etc/fail2ban/filter.d/apache-404.conf fail2ban-regex /var/log/httpd/access_log /etc/fail2ban/filter.d/apache-404.conf
In addition to that, we verify the status of the newly created jail using the fail2ban-client command. For example, we use the following command to check the status of the newly created jail apache-404.
fail2ban-client status apache-404
The output will be like this.
Status for the jail: apache-404 |- filter | |- File list: /var/log/httpd/access.log | |- Currently failed: 0 | `- Total failed: 0 `- action |- Currently banned: 0 | `- IP list: `- Total banned: 0
4) Verify firewall rules
Finally, we verify that the rules were added by fail2ban to the server firewall. For example, we confirm whether fail2ban blocks IPs in iptables using the below command.
Moreover, we check the fail2ban logs /var/log/fail2ban.log to confirm the IPs are banned.
2013-02-28 02:17:21,388 fail2ban.actions: WARNING [apache-404] Ban xxx.xxx.xx.xx 2019-02-28 02:37:21,195 fail2ban.actions: WARNING [apache-404] Unban xxx.xxx.xx.xx
Done! Fail2ban will now protect Apache from 404 bot attacks.
In short, it’s quite easy to setup fail2ban to protect Apache web server from 404 bot attacks. Today, we’ve discussed how our Dedicated Engineers setup fail2ban to protect Apache from 404 attacks.