14 ms·

But it didn't, and it's happening now, because of AI.

by mostlysimilar 1y ago

kjkjadksj 1y ago
People have been complaining about these crawlers for years well before AI
- PaulDavisThe1st 1y ago
  The issue is 1 to 4 orders of magnitude worse than it was just a couple of years ago. This is not "crawlers suck". This is "crawlers are overwhelming us and almost impossible to fully block". It really isn't the same thing.
  - tadfisher 1y ago
    Tragedy of the commons. Before, it was cryptominers eating up all free sources of compute [0]. Now it's AI crawlers eating up all available bandwidth and server resources [1]. Reading SourceHut's struggles against the Once-lers of the world makes me want to introduce a new application layer protocol where consumers pay for abusing shared resources. Which sucks, because the Internet should remain free. [0]: https://drewdevault.com/2021/04/26/Cryptocurrency-is-a-disaster.html https://drewdevault.com/2021/04/26/Cryptocurrency-is-a-disas... [1]: https://drewdevault.com/2025/03/17/2025-03-17-Stop-externalizing-your-costs-on-me.html https://drewdevault.com/2025/03/17/2025-03-17-Stop-externali...
    - PaulDavisThe1st 1y ago
      > Tragedy of the commons. No, because there is no such thing, at least not as understood by Garrett Hardin, who put forward the phrase. Commons fail when selfish, greedy people subvert or destroy the governance structures that help control them. If those governance structures exist (and they do for all historical commons) and continue to exist, the commons suffers no tragedy. This recent slide deck talks about Ostrom's ideas on this, which even Hardin eventually conceded were correct, and that his diagnosis of a "tragedy of the commons" does not actually describe the historical processes by which commons are abused. https://dougwebb.site/slides/commons https://dougwebb.site/slides/commons That said ... arguably there is a problem here with a "commons" that does in fact lack any real governance structure.
      - erlend_sh 1y ago
        No idea why this is getting downvoted; this is a very important correction since the “tragedy of the commons” meme is based on a flawed premise that needs to be amended.
  - p3rls 1y ago
    i am getting almost 500,000 ai scraper requests a day according to cloudflare's ai audit. google requests the same pages 10+ times each an hour. it was never this bad before.

But it didn't, and it's happening now, because of AI.

kjkjadksj 1y ago

People have been complaining about these crawlers for years well before AI

PaulDavisThe1st 1y ago
The issue is 1 to 4 orders of magnitude worse than it was just a couple of years ago. This is not "crawlers suck". This is "crawlers are overwhelming us and almost impossible to fully block". It really isn't the same thing.
- tadfisher 1y ago
  Tragedy of the commons. Before, it was cryptominers eating up all free sources of compute [0]. Now it's AI crawlers eating up all available bandwidth and server resources [1]. Reading SourceHut's struggles against the Once-lers of the world makes me want to introduce a new application layer protocol where consumers pay for abusing shared resources. Which sucks, because the Internet should remain free. [0]: https://drewdevault.com/2021/04/26/Cryptocurrency-is-a-disaster.html https://drewdevault.com/2021/04/26/Cryptocurrency-is-a-disas... [1]: https://drewdevault.com/2025/03/17/2025-03-17-Stop-externalizing-your-costs-on-me.html https://drewdevault.com/2025/03/17/2025-03-17-Stop-externali...
  - PaulDavisThe1st 1y ago
    > Tragedy of the commons. No, because there is no such thing, at least not as understood by Garrett Hardin, who put forward the phrase. Commons fail when selfish, greedy people subvert or destroy the governance structures that help control them. If those governance structures exist (and they do for all historical commons) and continue to exist, the commons suffers no tragedy. This recent slide deck talks about Ostrom's ideas on this, which even Hardin eventually conceded were correct, and that his diagnosis of a "tragedy of the commons" does not actually describe the historical processes by which commons are abused. https://dougwebb.site/slides/commons https://dougwebb.site/slides/commons That said ... arguably there is a problem here with a "commons" that does in fact lack any real governance structure.
    - erlend_sh 1y ago
      No idea why this is getting downvoted; this is a very important correction since the “tragedy of the commons” meme is based on a flawed premise that needs to be amended.
- p3rls 1y ago
  i am getting almost 500,000 ai scraper requests a day according to cloudflare's ai audit. google requests the same pages 10+ times each an hour. it was never this bad before.