Modern bots frequently probe websites by “feeling around” for potential vulnerabilities. They attempt various known exploits targeting popular web platforms such as WordPress and other common CMS solutions. During these automated scans, they often trigger numerous 404 Not Found errors when trying to access non-existent files or directories.
One of the simplest and most effective ways to protect your server against such malicious traffic is to automatically block any IP address that repeatedly generates 404 errors within a short period of time.
The following guide demonstrates how to implement this protection on Ubuntu, using Fail2ban to detect suspicious behavior and UFW (Uncomplicated Firewall) to block offending IP addresses.
Step by step guide
This guide shows a practical, copy-pasteable setup to detect repeated 404 Not Found hits in your web logs and automatically block offending IPs with UFW using Fail2ban. Adjust logpath and thresholds to suit your environment.
1. Update system and install packages
sudo apt update
sudo apt install -y ufw fail2ban2. Enable basic UFW rules and the firewall
Allow management access (SSH) and common web ports, then enable UFW:
sudo ufw allow OpenSSH
sudo ufw allow http
sudo ufw allow https
sudo ufw enable
sudo ufw status verboseIf you use a non-standard SSH port, replace
OpenSSHwith the port number (e.g.sudo ufw allow 2222/tcp).
3. Create the Fail2ban filter for 404s
Create a filter file that matches 404 lines in your access log. Use the correct log format for your web server.
Example for Nginx (create /etc/fail2ban/filter.d/nginx-404.conf):
[Definition]
# datepattern not always required; adjust if you need precise matching
datepattern = %%d/%%b/%%Y:%%H:%%M:%%S %%z
# Basic regex: matches common Nginx/Apache access log 404 lines where the IP is at the start
failregex = ^<HOST> - - \[.*\] ".*" 404
ignoreregex =
If your logs differ, adapt the regex. You can also create
apache-404.confwith the same or adjusted regex.
4. Create a jail that uses the filter and UFW action
Create a jail file /etc/fail2ban/jail.d/nginx-404.conf (or .local) and point it at your actual log:
[nginx-404]
enabled = true
port = http,https
filter = nginx-404
# IMPORTANT: set logpath to the log your site actually writes to
# e.g. /var/log/nginx/access.log or /var/log/apache/access.log
logpath = /var/log/server/access.log
maxretry = 10
findtime = 600 # in seconds (10 minutes)
bantime = 86400 # in seconds (24 hours)
backend = auto
action = ufw # other options - iptables-multiport, firewallcmdFields explained:
logpath— change to the exact file Fail2ban should monitor.maxretry— how many 404s withinfindtimetrigger a ban.findtime— time window in seconds (600s = 10 minutes).bantime— ban length in seconds (86400s = 24 hours).action = ufw— use UFW to block the offending IP.
Optional: add ignoreip = 127.0.0.1/8 192.0.2.0/24 in /etc/fail2ban/jail.local to whitelist trusted IPs.
5. Validate the filter (recommended)
Use fail2ban-regex to test your filter against current logs:
sudo fail2ban-regex /var/log/nginx/service-access.log /etc/fail2ban/filter.d/nginx-404.confThis shows which lines would match and how many hits would be counted.
6. Restart Fail2ban and enable it at boot
sudo systemctl restart fail2ban
sudo systemctl enable fail2banCheck the new jail:
sudo fail2ban-client status
sudo fail2ban-client status nginx-404 You should see the jail listed and the logpath from your config.
7. Test the setup (simulate 404s)
From a test host (or local machine), request a non-existent URL repeatedly until you exceed maxretry:
# example using curl in a loop
for i in {1..12}; do curl -s -o /dev/null -w "%{http_code}\n" http://your.server.example/nonexistent_$i; done
Then check if the IP was banned:
sudo fail2ban-client status nginx-404
# or view UFW rules
sudo ufw status numberedTo unban a test IP:
sudo fail2ban-client set nginx-404 unbanip <IP_ADDRESS>
# or remove via ufw:
sudo ufw delete allow from <IP_ADDRESS>8. Troubleshooting & tips
- Wrong logfile: ensure
logpathpoints to the exact file your virtual host writes to (e.g. custom site logs like/var/log/nginx/navigal-redmine.log). - Regex doesn’t match: run
fail2ban-regexand tweakfailregexuntil it matches expected log lines. - Permissions: Fail2ban runs as root, but ensure logs are readable by the service.
- Log rotation: Fail2ban handles rotated logs if
logpathuses the main filename; ensure rotated files keep the same naming pattern or adjustlogpathwith wildcards (Fail2ban supports globbing). - Whitelist internal services: set
ignoreipto avoid accidentally banning internal monitoring or load balancer IPs. - Tune thresholds: set
maxretry,findtime,bantimeto balance blocking attackers vs false positives. - Logging: view Fail2ban logs at
/var/log/fail2ban.log.
9. Example full file list (quick reference)
- Filter:
/etc/fail2ban/filter.d/nginx-404.conf - Jail config:
/etc/fail2ban/jail.d/nginx-404.conf(or/etc/fail2ban/jail.local) - Fail2ban log:
/var/log/fail2ban.log - Web log (example):
/var/log/nginx/navigal-redmine.log
Summary
This is a basic, effective mitigation against simple scanning bots. It helps reduce noise and automated attack surface but is not a replacement for:
- keeping CMS and plugins up to date,
- proper web application hardening,
- WAF (when needed) for sophisticated attacks,
- and monitoring/alerting.
I hope you find this guide helpful Cheers!