4 ms·

Fighting the AI Scraperbot Scourge

54 points by pabs3 2y ago

foxes 2y ago
Maybe the botnets can start fighting each other, eliminate the drifters to maximise inputs for slopclips Wonder if meta et al buy these kind of services, they already steal so it’s not a huge step Ye ai/ml was a neat piece of tech; it is intellectually interesting to try figure out what it learns and how it learns; and there are maybe valid uses, but capitalism and worse has utterly consumed it. I don’t have the words to quite describe it all, the farce of it. Then you get people that talk about alignment, how can anything “aligned” be built on such a foundation. What’s the end goal, a techbronocracy worshiping a fake god
aftbit 2y ago
So far I have not seen any of the operators proposing what seems to me to be the obvious solution: sell this data set in a machine readable format for cheap to the biggest scrapers.
- MattDaEskimo 2y ago
  It would have to be aggregated data that's constantly updated if it were to be anywhere competitive to just simply scraping the page.
  - UncleEntity 2y ago
    Plus, the "biggest scrapers" have shown their unwillingness to pay for content.
  - aftbit 2y ago
    Yes, presumably the approach would be to publish both a current dump of all of the databases every week or month, as well as a daily diff. Even better if it were just given away for free. The challenge would be around monetization and licensing. Even if you only published e.g. the kernel mailing list archive in this way, I could see someone like OpenAI turning around and suing to claim that you had given them copyright over that content rather than just providing the content itself.
- deleted 2y ago
  [deleted]
- immibis 2y ago
  Lots of sites have been trying to do this, but they're fighting a losing battle because the AI companies can get the same data for free.
grg0 2y ago
Good post, but it looks like there is no solution to this? At what point do you give up the web and invent a different protocol? Is there actually a solution to this?
immibis 2y ago
They're probably using residential proxy services. You can pay for a proxy service that redirects each individual connection through a random proxy in their pool, and all of them are in netblocks marked as residential. They're not all botnets - people can also get paid for sharing their connection. They're expensive, on the order of $1/GB.
phendrenad2 2y ago
I think that in the next 12 months AI companies will shut these scrapers off. The big AI companies have enough data already, and there won't be funding for any new smaller companies.
ndiddy 2y ago
Interesting how he brings up AI companies paying botnets to scrape sites, I hadn't heard of that before. It's extremely malicious behavior because as he mentions, it's literally a DDoS attack. > But, increasingly, the scraperbot traffic does not fit that pattern. Instead, traffic will come from literally millions of IP addresses, where no specific address is responsible for more than two or three hits over the course of a week. Watching the traffic on the site, one can easily see scraping efforts that are fetching a sorted list of URLs in an obvious sequence, but the same IP address will not appear twice in that sequence. The specific addresses involved come from all over the globe, with no evident pattern. > In other words, this scraping is being done by botnets, quite likely bought in underground markets and consisting of compromised machines. There really is not any other explanation that fits the observed patterns. Once upon a time, compromised systems were put to work mining cryptocurrency; now, it seems, there is more money to be had in repeatedly scraping the same web pages. When one of these botnets goes nuts, the result is indistinguishable from a distributed denial-of-service (DDOS) attack — it is a distributed denial-of-service attack. Should anybody be in doubt about the moral integrity of the people running these systems, a look at the techniques they use should make the situation abundantly clear.

54 points by pabs3 2y ago

Maybe the botnets can start fighting each other, eliminate the drifters to maximise inputs for slopclips Wonder if meta et al buy these kind of services, they already steal so it’s not a huge step Ye ai/ml was a neat piece of tech; it is intellectually interesting to try figure out what it learns and how it learns; and there are maybe valid uses, but capitalism and worse has utterly consumed it. I don’t have the words to quite describe it all, the farce of it. Then you get people that talk about alignment, how can anything “aligned” be built on such a foundation. What’s the end goal, a techbronocracy worshiping a fake god

aftbit 2y ago

So far I have not seen any of the operators proposing what seems to me to be the obvious solution: sell this data set in a machine readable format for cheap to the biggest scrapers.

MattDaEskimo 2y ago
It would have to be aggregated data that's constantly updated if it were to be anywhere competitive to just simply scraping the page.
- UncleEntity 2y ago
  Plus, the "biggest scrapers" have shown their unwillingness to pay for content.
- aftbit 2y ago
  Yes, presumably the approach would be to publish both a current dump of all of the databases every week or month, as well as a daily diff. Even better if it were just given away for free. The challenge would be around monetization and licensing. Even if you only published e.g. the kernel mailing list archive in this way, I could see someone like OpenAI turning around and suing to claim that you had given them copyright over that content rather than just providing the content itself.
deleted 2y ago
[deleted]
immibis 2y ago
Lots of sites have been trying to do this, but they're fighting a losing battle because the AI companies can get the same data for free.

grg0 2y ago

Good post, but it looks like there is no solution to this? At what point do you give up the web and invent a different protocol? Is there actually a solution to this?

immibis 2y ago

They're probably using residential proxy services. You can pay for a proxy service that redirects each individual connection through a random proxy in their pool, and all of them are in netblocks marked as residential. They're not all botnets - people can also get paid for sharing their connection. They're expensive, on the order of $1/GB.

phendrenad2 2y ago

I think that in the next 12 months AI companies will shut these scrapers off. The big AI companies have enough data already, and there won't be funding for any new smaller companies.

ndiddy 2y ago

Interesting how he brings up AI companies paying botnets to scrape sites, I hadn't heard of that before. It's extremely malicious behavior because as he mentions, it's literally a DDoS attack. > But, increasingly, the scraperbot traffic does not fit that pattern. Instead, traffic will come from literally millions of IP addresses, where no specific address is responsible for more than two or three hits over the course of a week. Watching the traffic on the site, one can easily see scraping efforts that are fetching a sorted list of URLs in an obvious sequence, but the same IP address will not appear twice in that sequence. The specific addresses involved come from all over the globe, with no evident pattern. > In other words, this scraping is being done by botnets, quite likely bought in underground markets and consisting of compromised machines. There really is not any other explanation that fits the observed patterns. Once upon a time, compromised systems were put to work mining cryptocurrency; now, it seems, there is more money to be had in repeatedly scraping the same web pages. When one of these botnets goes nuts, the result is indistinguishable from a distributed denial-of-service (DDOS) attack — it is a distributed denial-of-service attack. Should anybody be in doubt about the moral integrity of the people running these systems, a look at the techniques they use should make the situation abundantly clear.