5 ms·

I got DoSed by ClaudeBot (Anthropic) just last week. Hitting a website I manage 700,000 times in one month and tripping our bandwidth limit with our hosting pro

I got DoSed by ClaudeBot (Anthropic) just last week. Hitting a website I manage 700,000 times in one month and tripping our bandwidth limit with our hosting provider. What a PITA to have to investigate that, figure it out, block the user agent, and work with hosting provider support to get the limit lifted as a courtesy. Noticed that the ChatGPT bot was 2nd in traffic to this site, just not enough to cause trouble.

throwaway2037 2y ago
robots.txt did not work?
- lemper 2y ago
  bro, since when vc funded ai companies have the courtesy to respect robots.txt?
- seabird 2y ago
  Of course it didn't work. At best, the dorks doing this think there's a gamechanging LLM application to justify the insane valuations right around the corner if they just scrape every backwater site they can find. At worst, they're doing it because it's paying good money. Either way, they don't care, they're just going to ignore robots.txt.
- hsbauauvhabzb 2y ago
  I have not monitored traffic in this way, but I imagine most AI companies would explicitly follow links listed in robots, even if not mentioned elsewhere on the site.
- epc 2y ago
  I’ve been doing web sites for thirty years, robots.txt is at best a request to polite user agents to respect the server’s desires. None of the malicious crawlers respect it. None of the AI crawlers respect it. I’ve resorted to returning xml and zip bombs in canary pages. At best it slows them down until I block their network.
Neil44 2y ago
I've got claude bot blocked too. It regularly took sites offline and ignored robots.txt. Claude bot is an asshole.
i5heu 2y ago
At which level of DDos one can claim damages from them?
- brookst 2y ago
  You can claim whatever you want, but actually litigation is expensive and it’s not at all a sure thing that “I made a publicly available resource and they used it too much” is going to win damages. Maybe? Maybe not?
  - i5heu 2y ago
    So I could ddos anyone legally as long I have a some reason?
    - brookst 2y ago
      Sure. Courts tend to care about intent, but your are welcome to try to change that.

throwaway2037 2y ago

robots.txt did not work?

lemper 2y ago
bro, since when vc funded ai companies have the courtesy to respect robots.txt?
seabird 2y ago
Of course it didn't work. At best, the dorks doing this think there's a gamechanging LLM application to justify the insane valuations right around the corner if they just scrape every backwater site they can find. At worst, they're doing it because it's paying good money. Either way, they don't care, they're just going to ignore robots.txt.
hsbauauvhabzb 2y ago
I have not monitored traffic in this way, but I imagine most AI companies would explicitly follow links listed in robots, even if not mentioned elsewhere on the site.
epc 2y ago
I’ve been doing web sites for thirty years, robots.txt is at best a request to polite user agents to respect the server’s desires. None of the malicious crawlers respect it. None of the AI crawlers respect it. I’ve resorted to returning xml and zip bombs in canary pages. At best it slows them down until I block their network.

Neil44 2y ago

I've got claude bot blocked too. It regularly took sites offline and ignored robots.txt. Claude bot is an asshole.

i5heu 2y ago

At which level of DDos one can claim damages from them?

brookst 2y ago
You can claim whatever you want, but actually litigation is expensive and it’s not at all a sure thing that “I made a publicly available resource and they used it too much” is going to win damages. Maybe? Maybe not?
- i5heu 2y ago
  So I could ddos anyone legally as long I have a some reason?
  - brookst 2y ago
    Sure. Courts tend to care about intent, but your are welcome to try to change that.