AI bots are hammering your WordPress REST API — how I rate limit /wp-json/ with Nginx
· 15 min read
A client's WooCommerce store had been getting progressively slower over a three-week period. Page loads crept from 1.6 seconds to 5+, and checkout timeouts were starting to affect conversions. The usual suspects — plugin updates, database bloat, PHP-FPM misconfiguration — were all clean. The answer was hiding in the access logs.
It's an answer I've now found on three separate servers in a matter of months: bots hammering the WordPress REST API at /wp-json/, eating PHP-FPM workers and CPU while every monitoring dashboard reports "healthy". This post is the full diagnostic process and the fix I now deploy as standard.
The Symptom
PHP-FPM workers were consistently saturated. On a server tuned for 20 workers, 18-19 were active during what should have been quiet mid-afternoon hours. But the site's analytics showed only 40-50 concurrent human visitors — nowhere near enough to consume that many workers.
curl -s http://127.0.0.1/status?full | grep -c "state: Running"I had the PHP-FPM status page enabled on this server (listening on /status), so I could see active versus idle workers directly. Active workers were hovering at 18-19 regardless of real traffic. Something was consuming PHP resources that had nothing to do with actual customers.
The Investigation
I pulled the top request paths from the past hour:
awk '{print $7}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head -20The output was immediately telling:
4281 /wp-json/wp/v2/posts?per_page=100&page=1
3744 /wp-json/wp/v2/posts?per_page=100&page=2
2198 /wp-json/wp/v2/pages
1856 /wp-json/wp/v2/categories
1203 /wp-json/wp/v2/tags
947 /wp-json/wp/v2/users
891 /wp-json/wc/store/v1/products
342 /wp-json/wp/v2/comments
156 /
89 /shop/Over 15,000 REST API requests per hour. The site only had around 120 posts and 30 pages — these bots were re-scraping the entire content catalogue in a loop. If your endpoint list is noisy with pagination parameters, strip the query strings before counting:
grep "wp-json" /var/log/nginx/access.log \
| awk '{print $7}' \
| sed 's/\?.*//' \
| sort | uniq -c | sort -rn | head -15I checked the user agents:
grep "wp-json" /var/log/nginx/access.log | awk -F'"' '{print $6}' | sort | uniq -c | sort -rn | head -10 5823 GPTBot/1.2 (+https://openai.com/gptbot)
3912 ClaudeBot/1.0 ([email protected])
2741 PetalBot; +https://webmaster.petalsearch.com/
1488 Bytespider; bytedance.com/bot
987 CCBot/2.0 (https://commoncrawl.org/faq/)
412 Mozilla/5.0 (compatible; DataForSeoBot/1.0)
203 python-requests/2.31.0AI training crawlers, search crawlers, and generic scrapers — all hitting the REST API directly. Every single one of those requests bypassed the Nginx FastCGI page cache because /wp-json/ endpoints return dynamic JSON from PHP. Each request loaded WordPress core, ran the query, serialised the response, and released the worker. At 15,000+ per hour, that's 4+ uncached PHP requests per second on top of real traffic.
The third useful breakdown is by source IP:
grep "wp-json" /var/log/nginx/access.log | awk '{print $1}' | sort | uniq -c | sort -rn | head -10On this server, the traffic came from a wide spread of addresses. That matters for the fix: when no single IP dominates, a firewall deny rule gets you nowhere and you need rate limiting instead.
The /wp-json/wp/v2/users requests were also leaking usernames — a known enumeration vector I've written about separately. But the bigger problem here wasn't security. It was resource exhaustion.
Why REST API Abuse Is Worse Than Page Scraping
When a bot crawls your normal pages, Nginx can serve a cached HTML response in microseconds without touching PHP. But REST API endpoints are uncacheable by default — they return dynamic JSON, often with pagination parameters, and WordPress treats every request as a fresh PHP execution: spawn a PHP-FPM worker, bootstrap core with all active plugins and the theme, run the database queries, serialise the JSON, return it.
A single bot requesting /wp-json/wp/v2/posts?per_page=100 triggers a WP_Query that loads 100 posts from the database, serialises them into JSON with all their metadata, and returns a response that can be 200-500KB. Multiply that by several bots running in parallel, paginating through every endpoint, and you've got a silent DDoS that looks like legitimate API traffic. The damage mechanism is the same worker exhaustion I've covered for XML-RPC and login brute force — it's the PHP-FPM pool that dies, not the content being accessed.
The Same Pattern, Two More Servers
This wasn't a one-off. On another client's WooCommerce store — a 4-core VPS that normally idled at 15-20% CPU — monitoring alerts fired when CPU climbed above 90% and stayed there for three hours. Pages that loaded in 800ms were taking 6-7 seconds. The access log showed over 23,000 requests to /wp-json/ in four hours, from a mix of cloud provider IPs and residential proxies with no single address dominant. I timed the requests: each one consumed 180-250ms of PHP-FPM worker time. Hundreds of concurrent bot requests kept the pool permanently saturated, and real customers queued behind them.
A third incident, on a 4GB VPS, was subtler: two scraper IPs making 7,200 REST API requests per day against a site with 800 legitimate daily visits, keeping 11 of 12 PHP-FPM workers busy. It went unnoticed for three weeks for reasons worth spelling out. The requests all returned 200s, so nothing hit the error logs. One bot masqueraded as Googlebot — a reverse DNS check confirmed it wasn't. And the ramp-up was gradual, from around 50 requests a day to thousands, so no single day looked like an attack.
Three servers, one root cause. Here's the fix I now apply everywhere.
The Fix: Nginx Rate Limiting for /wp-json/
Rate limiting has to happen at the Nginx level, not in a WordPress plugin — by the time a plugin runs, the request has already consumed a PHP-FPM worker, which is the exact resource under attack.
Add to the http block in /etc/nginx/nginx.conf:
limit_req_zone $binary_remote_addr zone=wp_rest:10m rate=5r/s;
limit_req_status 429;The 10m shared memory zone tracks roughly 160,000 client IPs. rate=5r/s is deliberately generous: legitimate frontend features (WooCommerce cart fragments, the block editor, contact forms) call /wp-json/ in short bursts, and 5 requests per second per IP covers all of them while stopping a bot from paginating through your entire content library at speed.
If your site sits behind Cloudflare or another reverse proxy, $binary_remote_addr will be the proxy's IP, not the visitor's. You need to restore the real client IP first — otherwise all traffic shares one rate-limit bucket:
set_real_ip_from 173.245.48.0/20;
set_real_ip_from 103.21.244.0/22;
# ... remaining Cloudflare ranges from https://www.cloudflare.com/ips/
real_ip_header CF-Connecting-IP;Then in the site's server block:
location /wp-json/ {
limit_req zone=wp_rest burst=15 nodelay;
try_files $uri $uri/ /index.php?$args;
}
# WordPress also serves REST responses at /?rest_route=/wp/v2/posts,
# which bypasses the /wp-json/ location block entirely. Pretty
# permalinks have been default for years — no legitimate client
# needs this fallback on a properly configured site.
if ($arg_rest_route != "") {
return 403;
}burst=15 nodelay allows short spikes of up to 15 requests beyond the sustained rate before throttling — enough headroom for a checkout flow or an editor session. Once the limit is exceeded, Nginx returns 429 Too Many Requests without spawning a PHP process. The bot gets throttled; the server stays healthy. Test and reload:
nginx -t && systemctl reload nginxOne variation worth knowing: if your editors find Gutenberg sluggish under the limit (it fires dozens of API calls per editing session), exempt logged-in users by keying the zone on a cookie map instead of the raw IP:
map $http_cookie $rate_limit_key {
default $binary_remote_addr;
"~wordpress_logged_in_" "";
}
limit_req_zone $rate_limit_key zone=wp_rest:10m rate=5r/s;An empty key exempts the request from the zone, so authenticated sessions skip rate limiting entirely while anonymous traffic is limited per IP as before.
Blocking AI Crawlers Outright
GPTBot and ClaudeBot technically respect robots.txt, but by the time they've parsed it and backed off, they've already consumed resources. Blocking at the Nginx level is instant and costs nothing:
map $http_user_agent $is_ai_crawler {
default 0;
"~*GPTBot" 1;
"~*ClaudeBot" 1;
"~*Bytespider" 1;
"~*PetalBot" 1;
"~*CCBot" 1;
"~*DataForSeoBot" 1;
"~*anthropic-ai" 1;
"~*Google-Extended" 1;
}
server {
# ... existing config ...
if ($is_ai_crawler) {
return 444;
}
}The 444 response is Nginx-specific — it drops the connection immediately with no response body. Zero overhead. I also update robots.txt with Disallow: / entries for the same user agents as a courtesy signal. It won't stop badly-behaved bots, but it stops the legitimate ones from indexing your content for AI training — a reasonable default for a commercial WooCommerce store.
Fail2Ban for Persistent Crawlers
Rate limiting caps the throughput, but a persistent bot will happily hammer the endpoint at the allowed rate forever. For IPs that trigger 429s repeatedly, I ban them at the firewall with Fail2Ban.
Filter at /etc/fail2ban/filter.d/nginx-restapi.conf:
[Definition]
failregex = ^<HOST> .* "(GET|POST) /wp-json/.* HTTP/.*" 429
ignoreregex =Jail at /etc/fail2ban/jail.d/nginx-restapi.conf:
[nginx-restapi]
enabled = true
port = http,https
filter = nginx-restapi
logpath = /var/log/nginx/access.log
maxretry = 30
findtime = 60
bantime = 3600Thirty rate-limit responses within 60 seconds earns a one-hour ban. Legitimate users never get near that threshold; automated crawlers hit it within minutes. On the 4-core VPS incident, this jail banned 87 IPs in the first 24 hours.
Restricting the REST API at the WordPress Level
Even with Nginx rate limiting, there's no reason for unauthenticated visitors to access most REST API endpoints. But blanket-blocking the API is a common recommendation I disagree with: the block editor, Contact Form 7, and WooCommerce's Store API (/wp-json/wc/store/ — used by guest checkout and block-based Cart/Checkout pages) all depend on it, and a blanket restriction breaks things that surface days later when a customer can't complete checkout.
Whitelist the routes that genuinely need public access instead. Add to a site-specific plugin or functions.php:
add_filter( 'rest_authentication_errors', function ( $result ) {
if ( true === $result || is_wp_error( $result ) ) {
return $result;
}
$allowed_routes = array(
'/wc/store/',
'/contact-form-7/',
);
$rest_route = $GLOBALS['wp']->query_vars['rest_route'] ?? '';
foreach ( $allowed_routes as $route ) {
if ( 0 === strpos( $rest_route, $route ) ) {
return $result;
}
}
if ( ! is_user_logged_in() ) {
return new WP_Error(
'rest_forbidden',
'REST API access restricted.',
array( 'status' => 403 )
);
}
return $result;
} );This checks the parsed REST route path rather than the raw REQUEST_URI, so it can't be bypassed by stuffing a whitelisted substring into a query parameter. It keeps WooCommerce block checkout and contact forms functional while locking down everything else. Authenticated sessions — Gutenberg, WP-Admin, WooCommerce admin — are unaffected. Test on staging first.
The Results
Within 24 hours of deploying all layers on the lead incident:
- REST API requests dropped from 15,000/hour to under 200/hour (legitimate authenticated requests only)
- Active PHP-FPM workers during off-peak dropped from 18-19 to 4-6
- Average page load time returned to 1.7 seconds
- Checkout timeout errors stopped completely
- Server CPU load average dropped from 3.8 to 0.9
The AI crawlers moved on. The scrapers hit 429s and 444s and stopped retrying. The generic Python bots got 403s from WordPress and had nothing left to scrape. On the 4-core VPS, the effect was even faster: CPU dropped from 92% to 18% within ten minutes of the Nginx reload, with 3,400 requests answered by a cheap 429 instead of a PHP-FPM worker over the following day.
Monitoring So It Doesn't Come Back
I now run a small cron job across managed servers to flag REST API traffic spikes early:
#!/bin/bash
# /usr/local/bin/check-restapi-traffic.sh
THRESHOLD=2000
LOG="/var/log/nginx/access.log"
COUNT=$(grep -cE "(wp-json|rest_route)" "$LOG")
if [ "$COUNT" -gt "$THRESHOLD" ]; then
echo "REST API request count: $COUNT (threshold: $THRESHOLD)" | \
mail -s "High REST API traffic on $(hostname)" [email protected]
fi# /etc/cron.d/restapi-monitor
0 */6 * * * root /usr/local/bin/check-restapi-traffic.shOn a site with 800 daily visitors, anything above 2,000 REST API requests per day is suspicious, and any single IP making more than 200 REST API requests a day on a small-to-medium site warrants a look:
grep "wp-json" /var/log/nginx/access.log | \
awk '{print $1}' | sort | uniq -c | sort -rn | \
awk '$1 > 200 {print $1, $2}'What I Now Check on Every Server
REST API abuse has overtaken XML-RPC as the most common source of unexplained PHP-FPM saturation on the servers I manage. The REST API ships enabled and fully open by default on every WordPress installation, most hosting providers don't rate-limit it, and security plugins don't flag it — Wordfence tracks failed logins and file changes, not a scraper making thousands of valid 200-status API requests a day.
If your server is running hot and you've already ruled out wp-login.php and xmlrpc.php, grep your access logs for wp-json. You might be surprised by what you find.
This is part of the standard hardening I apply during server management onboarding and security monitoring setup. If your WooCommerce store is slow and you're not sure why, have a look at my maintenance plans.
