3 ms·

But what's the difference between one user making 900k hits and 900k different users making one hit? In both cases you have made a resource available and people

But what's the difference between one user making 900k hits and 900k different users making one hit? In both cases you have made a resource available and people are requesting it, some more than others. If serving traffic for free is a problem, don't. If you are only able to serve N requests per second/minute/day/etc, do that. But don't complain if you give out something for free and people take it. (also, a lot of the numbers people quote during these AI scraper "attacks" are very tame and the fact they are branded as problematic makes me suspect there's substantial incompetence in the solutions deployed to serve them)

latexr 1y ago
> But what's the difference between one user making 900k hits and 900k different users making one hit? What’s the difference between giving 900K meals to one person and feeding 900K people? The former is being abusive, wasteful, and depriving almost 900K other people of food. They are also being deceitful by pretending to be 900K different people. Resources are finite. Web requests aren’t food, but you still pay for them. A spike in traffic may mean your service being down for the rest of the month, which is more acceptable if you helped a bunch of people who have now learned about and can talk about and share what you provided, versus having wasted all your traffic on a single bad actor who didn’t even care because they were just a robot. > makes me suspect there's substantial incompetence in the solutions deployed to serve them So you see bots scraping the Wikipedia webpages instead of downloading their organised dump, or scraping every git service webpage instead of cloning a repo, and think the incompetence is with the website instead of the scraper wasting time and resources to do a worse job?
PaulDavisThe1st 1y ago
There were never 900k users interested in each commit. Never was, never will be. So that's a false comparison. These scrapers have upped both the server load (requests per second) and bandwidth requirements, without me consenting to it. If they were actual human users OR bots that were appropriately designed to minimize their impact on the target sites, that's perfectly OK. Maybe if this was truly the only way to get to our god-like LLM to work in a god-like way (*), it would also be acceptable. But it isn't. And on top of that, they are incompetently designed and they are causing real issues that a huge number of sites need to address. (*) put differently, if all this current scraping activity delivered some notable benefit to humanity

latexr 1y ago

> But what's the difference between one user making 900k hits and 900k different users making one hit? What’s the difference between giving 900K meals to one person and feeding 900K people? The former is being abusive, wasteful, and depriving almost 900K other people of food. They are also being deceitful by pretending to be 900K different people. Resources are finite. Web requests aren’t food, but you still pay for them. A spike in traffic may mean your service being down for the rest of the month, which is more acceptable if you helped a bunch of people who have now learned about and can talk about and share what you provided, versus having wasted all your traffic on a single bad actor who didn’t even care because they were just a robot. > makes me suspect there's substantial incompetence in the solutions deployed to serve them So you see bots scraping the Wikipedia webpages instead of downloading their organised dump, or scraping every git service webpage instead of cloning a repo, and think the incompetence is with the website instead of the scraper wasting time and resources to do a worse job?

PaulDavisThe1st 1y ago

There were never 900k users interested in each commit. Never was, never will be. So that's a false comparison. These scrapers have upped both the server load (requests per second) and bandwidth requirements, without me consenting to it. If they were actual human users OR bots that were appropriately designed to minimize their impact on the target sites, that's perfectly OK. Maybe if this was truly the only way to get to our god-like LLM to work in a god-like way (*), it would also be acceptable. But it isn't. And on top of that, they are incompetently designed and they are causing real issues that a huge number of sites need to address. (*) put differently, if all this current scraping activity delivered some notable benefit to humanity