3 ms·
My initial reaction was that running something like this is still a loss, because it probably costs you as much or more than it costs them in terms of both netw
by akoboldfrying 1y ago
My initial reaction was that running something like this is still a loss, because it probably costs you as much or more than it costs them in terms of both network bytes and CPU. But then I realised two things:
1. If they are using residential IPs, each byte of network bandwidth is probably costing them a lot more than it's costing you. Win.
2. More importantly, if this became a thing that a large fraction of all websites do, the economic incentive for AI scrapers would greatly shrink. (They don't care if 0.02% of their scraping is garbage; they care a lot if 80% is.) And the only move I think they would have in this arms race would be... to use an LLM to decide whether a page is garbage or not! And now the cost of scraping a page is really starting to increase for them, even if they only run a local LLM.
- asgerhb 1y agoNot to mention they have to store the data after they download it. In theory storing garbage data is costly to them. However I have a nagging feeling that the attitude of these scrapers is they get paid the same amount per gigabyte whether it's nonsense or not.
- luckylion 1y agoIf they even are AI crawlers. Could be just as well some exploit-scanners that are searching for endpoints they'd try to exploit. That wouldn't require storing the content, only the links.
- m3047 1y agoIf you look at the pages which are hit and how many pages are hit by any one address in a given period of time it's pretty easy to identify features which are reliable proxies for e.g. exploit scanners, trawlers, agents. I publish a feed of what's being hit on my servers, contact me for details (you need to be able to make DNS queries to a particular server directed at a domain which is not reachable from ICANN's root).
- mrweasel 1y agoWe should encourage number 2. So much of the content that the AI companies are scraping is already garbage, and that's a problem. E.g. LLMs are frequently confidently wrong, but so is Reddit, who produce a large volume of trading data. We've seen a study surgesting that you can poison an LLM with very little data. Encouraging the AI companies to care about the quality of the data they are scraping could be beneficial to all. The cost of being critical of source material might make some AI companies tank, but that seems inevitable.
- kelnos 1y ago> it probably costs you as much or more than it costs them in terms of both network bytes and CPU Network bytes, perhaps (though text is small), but the article points out that each garbage page is served using only microseconds of CPU time, and a little over a megabyte of RAM. The goal here isn't to get the bots to go away, it's to feed them garbage forever, in a way that's light on your resources. Certainly the bot, plus the offline process that trains on your garbage data, will be using more CPU (and I/O) time than you will to generate it.