3 ms·

Archive.org was knocked offline the other day due to some AI startup scraping it to death. It’s not a good thing.

• moneywoes 3y ago
  Source, they don’t rate limit
  • Kon-Peki 3y ago
    https://news.ycombinator.com/item?id=36110527 https://news.ycombinator.com/item?id=36110527
  • piperswe 3y ago
    True - and their lack of rate limiting ended up letting someone overwhelm their servers, knocking them offline.
  • edgyquant 3y ago
    They put out a blog asking people not to scrape afterwards. A simple google will be much fast than asking for sources.