3 ms·

It is. They will most likely eventually need to move to a walled model for Wayback due to scraper aggressiveness (like Reddit deprecating anonymous old.reddit.c

It is. They will most likely eventually need to move to a walled model for Wayback due to scraper aggressiveness (like Reddit deprecating anonymous old.reddit.com), or behind Cloudflare for aggressive bot and scraping protection. Hard to defend against abuse of a public resource when its intent is public access with as little restriction as possible. https://en.wikipedia.org/wiki/Tragedy_of_the_commons https://en.wikipedia.org/wiki/Tragedy_of_the_commons (no affiliation)

ronsor 1mo ago
Reddit has no excuses for the anonymous old.reddit.com removal; they're simply greedy. On the other hand, the Internet Archive is a non-profit offering a free public resource.
- toomuchtodo 1mo ago
  Examples provided as technical examples, strong feelings are out of scope for this thread.
  - itintheory 1mo ago
    As someone who operates a large non-profit public data driven website, I have some VERY strong feelings about scrapers. We looked into various commercial solutions (Datadome, HUMAN) and based on our traffic estimates from logs we'd be looking at at least 250k/yr for bot mitigation. Anubis is offering a temporary reprieve, but after reading the recent kernel.org article [0] it's increasingly clear that this is a temporary bandaid. The cheapest solution is to require a login and rate limit by API key. I also have strong feelings about the tragedy of the commons. [0] https://people.kernel.org/monsieuricon/creepy-crawlies https://people.kernel.org/monsieuricon/creepy-crawlies
    - toomuchtodo 1mo ago
      No strong feelings here is what I meant. Certainly, that energy is best directed into aggressive countermeasures and defense in depth of public goods. https://hn.algolia.com/?dateRange=all&page=0&prefix=true&query=author%3Adang%20%E2%80%9Cstrong%20feelings%E2%80%9D&sort=byDate&type=comment https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...
    - userbinator 1mo ago
      I say put your data up in torrents, host a few KB of plain HTML linking to them, and let decentralisation do the rest.
      - itintheory 1mo ago
        The data IS available. You can download it all from several sources in one big dump. And yet we're still scraped.
        userbinator 1mo ago
        Further evidence that they're not actually going after your data, but just DDoS'ing.

ronsor 1mo ago

Reddit has no excuses for the anonymous old.reddit.com removal; they're simply greedy. On the other hand, the Internet Archive is a non-profit offering a free public resource.

toomuchtodo 1mo ago
Examples provided as technical examples, strong feelings are out of scope for this thread.
- itintheory 1mo ago
  As someone who operates a large non-profit public data driven website, I have some VERY strong feelings about scrapers. We looked into various commercial solutions (Datadome, HUMAN) and based on our traffic estimates from logs we'd be looking at at least 250k/yr for bot mitigation. Anubis is offering a temporary reprieve, but after reading the recent kernel.org article [0] it's increasingly clear that this is a temporary bandaid. The cheapest solution is to require a login and rate limit by API key. I also have strong feelings about the tragedy of the commons. [0] https://people.kernel.org/monsieuricon/creepy-crawlies https://people.kernel.org/monsieuricon/creepy-crawlies
  - toomuchtodo 1mo ago
    No strong feelings here is what I meant. Certainly, that energy is best directed into aggressive countermeasures and defense in depth of public goods. https://hn.algolia.com/?dateRange=all&page=0&prefix=true&query=author%3Adang%20%E2%80%9Cstrong%20feelings%E2%80%9D&sort=byDate&type=comment https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...
  - userbinator 1mo ago
    I say put your data up in torrents, host a few KB of plain HTML linking to them, and let decentralisation do the rest.
    - itintheory 1mo ago
      The data IS available. You can download it all from several sources in one big dump. And yet we're still scraped.
      - userbinator 1mo ago
        Further evidence that they're not actually going after your data, but just DDoS'ing.