3 ms·
In the past month, I've had to block LLM bots attacking my poor little VPSs — not once, but twice. First, it was Facebook https://news.ycombinator.com/item?id=
by napolux 2y ago
In the past month, I've had to block LLM bots attacking my poor little VPSs — not once, but twice.
First, it was Facebook https://news.ycombinator.com/item?id=23490367 https://news.ycombinator.com/item?id=23490367 and now it's these other companies.
What's worse? They completely ignore a simple HTTP 429 status.
- DamonHD 2y agoAlmost no one pays attention to 429 for general Web pages. Eg on an RSS file Googlebot speeds up repolling. Amazon RSS feed puller does pay some attention to 429. 503 is at least apparently understood by more crawlers/bots, but they still like to blame the victim: YouTube sends me a condescending (and inaccurate) email when it gets a 503 for ignoring cache headers and other basics it seems...