Block AI Scrapers and Bad Bots at the Edge with Nginx and Fail2Ban

AI Crawlers Are Eating Your Server Resources#

If you run shared hosting, your servers are being scraped right now. AI training crawlers — GPTBot, CCBot, Bytespider, Amazonbot — hit sites aggressively, often requesting hundreds of pages per minute per IP. They frequently ignore robots.txt, burn bandwidth, spike PHP-FPM worker counts, and occasionally trigger false DDoS alarms at your upstream provider.

A 2024 report from Cloudflare estimated that AI crawlers now account for nearly 5% of all web traffic, with ByteDance's Bytespider alone responsible for a disproportionate share. For shared hosting providers running tight margins, that traffic isn't just annoying — it's expensive.

Traditional rate-limiting with Nginx helps, but it's blunt. You need a layered approach: block known bad user-agents at the edge, detect repeat offenders by request pattern, and automatically ban IPs that cross thresholds. Fail2Ban handles the banning. Nginx handles the logging and initial filtering. Together, they're a lightweight, effective defense that won't require CloudLinux or a commercial WAF.

This guide walks through a complete setup you can deploy on any Nginx-based shared hosting server in about 15 minutes.

Step 1: Block Known Bot User-Agents with an Nginx Map#

Before Fail2Ban even gets involved, reject the worst offenders at the HTTP level. Add this to your nginx.conf inside the http block:

map $http_user_agent $bad_bot {
    default 0;
    "~*GPTBot"         1;
    "~*CCBot"          1;
    "~*Bytespider"     1;
    "~*Amazonbot"      1;
    "~*anthropic-ai"   1;
    "~*ClaudeBot"      1;
    "~*Omgilibot"      1;
    "~*Diffbot"        1;
    "~*Applebot-Extended" 1;
    "~*FacebookBot"    1;
    "~*meta-externalagent" 1;
    "~*Google-Extended" 1;
    "~*Timpibot"       1;
    "~*DataForSeoBot"  1;
    "~*MJ12bot"        1;
    "~*AhrefsBot"      1;
    "~*SemrushBot"     1;
    "~*DotBot"         1;
    "~*BLEXBot"        1;
}

Then, in each server block (or in a shared include file):

if ($bad_bot) {
    return 444;
}

HTTP 444 is an Nginx-specific status that drops the connection with no response body. It's cheaper than returning a 403 and wastes zero bandwidth.

Why a map instead of if chains?#

The map directive compiles into a hash lookup — O(1) per request. Chaining if ($http_user_agent ~* "...") statements is slower and, per Nginx's own documentation, error-prone in location blocks.

Step 2: Log Blocked Requests for Fail2Ban#

The map above silently drops connections, so Fail2Ban never sees them. We need a second layer: log requests that aren't caught by the map but still look suspicious — aggressive request rates, missing referrers on resource-heavy paths, or repeated 403/444 responses.

Create a custom log format in nginx.conf:

log_format botwatch '$remote_addr - $remote_user [$time_local] '
                    '"$request" $status $body_bytes_sent '
                    '"$http_referer" "$http_user_agent"';

Add a dedicated access log inside your server block:

access_log /var/log/nginx/botwatch.log botwatch;

This log captures everything, but Fail2Ban will only act on lines matching specific patterns.

Step 3: Write Fail2Ban Filters for Bot-Like Behavior#

Create /etc/fail2ban/filter.d/nginx-botsearch.conf:

[Definition]
failregex = ^<HOST> .* "(GET|POST|HEAD) /.*" (403|444) .*$
ignoreregex = \.(?:css|js|jpg|jpeg|png|gif|ico|svg|woff2?)$

This matches IPs generating repeated 403 or 444 responses. The ignoreregex line excludes static asset requests — you don't want to ban a legitimate visitor whose browser triggered a 403 on a missing image.

For a tighter filter targeting high-frequency scrapers regardless of status code, create /etc/fail2ban/filter.d/nginx-aggressive-bot.conf:

[Definition]
failregex = ^<HOST> .* "(GET|POST) /.*" 200 .* "(?:GPTBot|CCBot|Bytespider|Amazonbot|anthropic|ClaudeBot)"$
ignoreregex =

This catches AI bots that slipped past the map (maybe they changed their user-agent string) but still self-identify.

Step 4: Configure the Fail2Ban Jails#

Add to /etc/fail2ban/jail.local:

[nginx-botsearch]
enabled  = true
port     = http,https
filter   = nginx-botsearch
logpath  = /var/log/nginx/botwatch.log
maxretry = 10
findtime = 60
bantime  = 3600

[nginx-aggressive-bot]
enabled  = true
port     = http,https
filter   = nginx-aggressive-bot
logpath  = /var/log/nginx/botwatch.log
maxretry = 5
findtime = 120
bantime  = 86400

The first jail bans IPs after 10 blocked requests in 60 seconds (one-hour ban). The second jail is stricter: 5 requests from a known AI bot user-agent in two minutes earns a 24-hour ban.

Restart Fail2Ban:

sudo systemctl restart fail2ban
sudo fail2ban-client status nginx-botsearch

Check the status output to confirm the jail is active and parsing the log.

Step 5: Persist Bans Efficiently#

By default, Fail2Ban uses iptables to drop traffic at the network layer. On servers with many jailed IPs, consider switching to the nftables action for better performance:

[DEFAULT]
banaction = nftables-multiport

For shared hosting boxes running 50+ sites, nftables handles large rule sets more efficiently than legacy iptables. If you're running a modern kernel (5.x+), nftables is the better default.

Step 6: Monitor and Tune#

After 24 hours, check what's being caught:

sudo fail2ban-client status nginx-botsearch
sudo fail2ban-client status nginx-aggressive-bot
sudo zgrep 'Ban' /var/log/fail2ban.log | tail -20

If legitimate IPs are getting banned, increase maxretry or extend findtime. If known bots are still hammering the server, add their user-agent strings to the Nginx map in Step 1.

Rotate the botwatch log#

Add a logrotate entry at /etc/logrotate.d/nginx-botwatch:

/var/log/nginx/botwatch.log {
    daily
    rotate 7
    compress
    missingok
    notifempty
    sharedscripts
    postrotate
        [ -f /var/run/nginx.pid ] && kill -USR1 $(cat /var/run/nginx.pid)
    endscript
}

Without this, the botwatch log will grow fast — especially on busy shared servers.

What This Won't Catch#

Sophisticated bots rotate user-agents and use residential proxies. For those, you need behavioral analysis — Cloudflare's bot management, CrowdSec's community-driven threat intelligence, or application-layer challenges. This setup handles the bulk of the noise: the blunt-force scrapers that burn your CPU and bandwidth without contributing anything useful.

For hosting resellers managing dozens of shared accounts on a single box, reducing bot traffic by even 15–20% translates directly into lower load averages, fewer PHP-FPM restarts, and fewer support tickets about slow sites. That's margin you can measure.

The Bottom Line#

Layered bot defense doesn't require expensive hardware or commercial subscriptions. A well-tuned Nginx map stops the worst offenders at zero cost, Fail2Ban catches the stragglers, and a few logrotate entries keep the whole thing from filling your disk. Deploy this on your next server build, revisit the user-agent list monthly — the bot landscape shifts fast, but the infrastructure to handle it doesn't have to.