4 ms·
What really confuses me is ... people always say, it's because companies are gathering data for AI training. Then why would they need to scrape the same page th
by andai 4mo ago
What really confuses me is ... people always say, it's because companies are gathering data for AI training. Then why would they need to scrape the same page thousands of times per day?
Edit: the article says millions of times per hour? (!?)
The article is also astonished by this, and speculates it might be some kind of underground AI labs but... millions of them? Or does it only take one with too much money and a badly configured scraping setup?
- mips_avatar 4mo agoI kind of wish the recent Google monopoly court ruling had forced Google to open up their index to anyone, not just Perplexity/other big players.
- catdog 4mo agoThat's really a huge issue right now (to some extent even before the AI hype) that almost everywhere google is effectively the only entity explicitly allowed to scrape.
- cyberax 4mo agoHah. I have a homelab with a couple of sites, including a personal Forgejo installation. Last night my server turned off because it went into thermal protection shutdown. Turns out, my all-in-one cooler has inoperative fans, which I normally never really notice. The passive heat dissipation from the water cooler is more than enough. However, this time they hammered my computer for 12 hours with about 200 requests per _second_ to my Forgejo.
- ezst 4mo agoThe paradox of them selling "intelligence on demand" or "coding agents rivaling the best developers" and yet having dumb as fart bots/scrapers is lost on many. But not all.
- inigyou 4mo agoThere's no paradox, just deception. "AI" is dumb as a bag of rocks, but good at convincing people that it isn't. However, it's also a really good semantic vector search that can find and combine existing answers to many pre-asked questions.
- alightsoul 4mo agoMaybe they are aggressively scanning for updates on the page
- catdog 4mo agoMaybe they have just too much money at hand, would not surprise me, people are still investing into gen AI like there is no tomorrow. Also, for the completely criminal operations, you only have to find a way to infect and distribute your bot to, e.g. some common internet of shit device. Scaling is basically free afterwards as you don't need to ask anyone. The article also hints that those are actually the biggest problem. Then there is probably also a lot of time pressure on the people implementing and operating those scrapers so they have even less incentive to optimize their code.
- axus 4mo agoMaybe someone is paid per scrape, without reduction in payment for duplicates. Maybe every web query for Linux commands in $LARGE_COUNTRY checks all the Linux websites again.
- bigiain 4mo ago> a badly configured scraping setup? Cynical-me assumes every single AI company is vibe-coding everything, and _all_ their scrapers are as badly written as the typical publicly available scraper code and tutorial - mostly written by self promoting spammers and SEO "experts" in the late 2010s. Any they all DGAF about wasting website owners server/network resources, of the CPU and network resources of the "dumb schmucks" who have a free vpn installed or a factory-hacked cheapo media box or mobile game the developer has surreptitiously monetised with a residential proxy sdk. It also wouldn't surprise me at all to find there are dozens of competing training data acquisition teams at every frontier and wannabe frontier AI company - scraping the entire web in parallel to meet internal KPIs. Half of which have lost entire datasets due to vibe coded storage and archive setups.
- pineapplepizza6 4mo ago[dead]
- stavros 4mo agoIt's not AI companies scraping these websites, it's AI companies creating a massively profitable need for data, and every random Joe with vibe coded scrapers tries to make a buck out of it.
- armchairhacker 4mo agoIt should really just be called DDoS, at a certain level of incompetence intent doesn’t matter. You’re right that there’s zero (information gathering) benefit over reasonable scraping which wouldn’t cripple the site. Who’s doing it, are they even using the data?
- procaryote 4mo ago750k items in their content management sysem. N independent labs crawling wanting to check that every day could easily give bursts of millions per hour Millions per hour is tens per second though; perhaps the fix is performance improvements
- Jabbles 4mo ago1,000,000 / 3,600 = ~278
- setupminimal 4mo agoWe have put in a number of performance improvements, yes. The nice thing about those is that they also make the site snappier when it's not under load. Right now we have just over a million items in our CMS, plus our publicly available mailing list archives, which are much larger, even if they're less frequently referenced.
- andai 4mo agoYeah we can just rewrite the web servers in Rust ;) That'll be great until.. they rewrite the scrapers in Rust! Then we're really hosed!
- inigyou 4mo agoIt isn't. AI scraping has nothing to do with it, for the reasons you said. Someone wants the web to go offline, they are DDoSing the entire web, and it's working. For some reason we are tackling the symptom instead of finding out who that person is. Come on, it can't be that hard to subpoena Bright Data. The law enforcement system knows how to track down someone who's trying to be anonymous on the internet.
- andai 4mo agoSo who would benefit from the entire web going offline? Which powerful entities have historically hated a free and open internet? ...all of them??
- dgacmu 4mo agoI had meta's crawler hitting the pi searcher at something like 5qps for days on end, just ... querying for substrings of pi, ignoring robots.txt, etc. it wasn't enough to break anything but it triggered a lot of alerts. I can imagine that sites with dynamic content and potentially unbounded query types or pathnames are in danger from particularly stupid crawlers.
- leros 4mo agoMeta has hit me with over 1,000 requests per second. Luckily it tripped my rate limiting but geez.
- szatkus 4mo agoI always thought it's the web search tool. Grok actually shows a number of sources used for an answer. Once I asked it something simple and it apparently scanned 200 different websites. And it was just a short prompt. Now imagine millions of users asking for something multiple times a day.
- deleted 4mo ago[deleted]