4 ms·
Can't they pull the data from archive.org?
by amelius 3y ago
Can't they pull the data from archive.org?
- notacoward 3y agoThat would be worse.
- KuiN 3y agoArchive.org was knocked offline the other day due to some AI startup scraping it to death. It’s not a good thing.
- moneywoes 3y agoSource, they don’t rate limit
- Kon-Peki 3y agohttps://news.ycombinator.com/item?id=36110527 https://news.ycombinator.com/item?id=36110527
- piperswe 3y agoTrue - and their lack of rate limiting ended up letting someone overwhelm their servers, knocking them offline.
- edgyquant 3y agoThey put out a blog asking people not to scrape afterwards. A simple google will be much fast than asking for sources.
- SllX 3y agoArchive.org is a non-profit without the capacity to serve that many requests. An excellent resource for people to use carefully, but not a treasure trove for bots to scrape down to the last bit.
- notpushkin 3y agoWould be cool if they introduce some reasonably priced access for mass scrapers. Should make some nice income in addition to donations, and a valuable service to community.