4 ms·

So these companies broke the internet

• haiku2077 1y ago
  Which companies? OpenAI, Anthropic, Google? No, their bots are pretty well behaved. The smaller AI companies deploying bots that don't respect any reasonable rate limits and are scraping the same static pages thousands of times an hour? Yup
  • e3bc54b2 1y ago
    Anecdote, but at least for tiny little server hosting single public repository, none of these companies had 'well behaved' bots. It may be possible that they learned to behave better but I wouldn't know since my only possible recourse was to blacklist them all AND take the repo private.
    • haiku2077 1y ago
      Those are the small companies spoofing their user agent as the big companies to dodge countermeasures.