4 ms·

Yeah, I've had to mitigate this problem too, on a tiny forum serving a very niche audience. It was brought to its knees by a handful of LLM bots that completely

Yeah, I've had to mitigate this problem too, on a tiny forum serving a very niche audience. It was brought to its knees by a handful of LLM bots that completely ignored robots.txt. Thankfully, these bots were easy enough to block at the firewall level, but that may not work forever.

diggan 2y ago
Were these bots also using random IPs and random User-Agents and if so, how do you know they are LLM agents?
- bovermyer 2y ago
  The User-Agent was not random; that's how I was able to block them this time.
  - szszrk 2y ago
    But they were far from honest, if, as you say, crawlers impersonate real users and their browsers.