How to stop bots from hammering api endpoints

by

Summary

A surge of automated clients was overwhelming the /api/books endpoint, causing NGINX to buffer massive upstream responses to disk and degrading performance. The bot‑blocking logic was incomplete, and the API returned payloads too large to serve efficiently under load. The result was excessive I/O, slow responses, and ineffective rate limiting.

Root Cause

The incident stemmed from a combination of configuration and architectural issues:

  • Large JSON responses forced NGINX to buffer upstream data to disk when memory limits were exceeded.
  • User‑Agent–based bot detection is unreliable, allowing most bots to bypass filters.
  • The map logic incorrectly blocks or allows traffic, causing inconsistent behavior.
  • Rate limiting was too permissive, allowing bursts that overwhelmed the Node backend.
  • Proxy buffering was disabled, which paradoxically increases disk writes when responses exceed memory thresholds.
  • No CDN or caching layer existed to absorb repeated heavy requests.

Why This Happens in Real Systems

Even well‑designed systems get hammered by bots because:

  • Bots rarely send honest User‑Agents.
  • Public endpoints with large responses are irresistible targets for scraping.
  • Attackers use distributed IPs, making IP‑based blocking ineffective.
  • APIs without caching or pagination become easy denial‑of‑service vectors.
  • NGINX defaults assume moderate payload sizes, not multi‑MB JSON blobs.

Real-World Impact

Systems hit by this pattern typically experience:

  • High disk I/O due to proxy buffering to temp files.
  • Increased latency as Node waits for NGINX to flush buffers.
  • Memory pressure on both NGINX and Node.
  • Rate‑limit bypasses because bots distribute requests across many IPs.
  • User‑visible slowdowns or outright API failures.

Example or Code (if necessary and relevant)

Below is an example of a safer NGINX configuration using proper buffering, caching, and rate limiting:

proxy_buffering on;
proxy_buffers 16 32k;
proxy_buffer_size 64k;

limit_req_zone $binary_remote_addr zone=api_limit:10m rate=2r/s;

location /api/books {
    limit_req zone=api_limit burst=5 nodelay;
    proxy_cache api_cache;
    proxy_cache_valid 200 10s;
    proxy_pass http://127.0.0.1:4000;
}

How Senior Engineers Fix It

Experienced engineers approach this holistically:

  • Stop relying on User‑Agent filtering; use behavioral detection instead.
  • Introduce pagination so no endpoint returns multi‑MB payloads.
  • Enable proxy buffering with tuned limits to avoid disk thrashing.
  • Add a CDN or edge cache to absorb repeated heavy requests.
  • Implement token‑based access control (HMAC, JWT, signed URLs).
  • Use dynamic rate limiting based on IP reputation or request patterns.
  • Instrument the API to understand which clients are abusive.
  • Deploy WAF rules to block scraper‑like behavior.

Why Juniors Miss It

Less‑experienced engineers often overlook:

  • How NGINX buffering actually works, especially when disabled.
  • That User‑Agent filtering is nearly useless against real bots.
  • The cost of returning huge JSON payloads repeatedly.
  • The difference between burst rate and sustained rate limiting.
  • The need for caching layers in front of heavy endpoints.
  • The fact that bots evolve faster than static rules.

They focus on blocking specific clients instead of designing the system to be resilient regardless of who calls it.

Leave a Comment