4 ms·
I’ve been a web developer for decades as well as doing scraping, indexing, and analyzing million of sites. Just follow the golden rule: don’t ever load any sit
by griomnib 2y ago
I’ve been a web developer for decades as well as doing scraping, indexing, and analyzing million of sites.
Just follow the golden rule: don’t ever load any site more aggressively than you would want yours to be.
This isn’t hard stuff, and these AI companies have grossly inefficient and obnoxious scrapers.
As a site owner those pisses me off as a matter of decency on the web, but as an engineer doing distributed data collection I’m offended by how shitty and inefficient their crawlers are.
- PaulHoule 2y agoI worked at one place where it probably cost us 100x (in CPU) more to serve content the way we were doing it as opposed to the way most people would do it. We could afford it because it was still cheap, but we deferred the cost reduction work for half a decade and went on a war against webcrawlers instead. (hint: who introduced the robots.txt standard?)
- mingabunga 2y agoWe've had to block a lot of these bots as they slowed our technical forum to a crawl, but new ones appear every now and again. Amazons was the worst
- griomnib 2y agoI really wonder if these dogshit scrapers are wholly built by LLM. Nobody competent codes like this.
- add-sub-mul-div 2y agoThese people think they're on the verge of the most important invention in modern history. Etiquette means nothing to them. They would probably consider an impediment to their work a harm to the human race.
- krapp 2y ago>They would probably consider an impediment to their work a harm to the human race. They do. Marc Andreeson said as much in his "techno-optimist manifesto," that any hesitation or slowdown in AI development or adoption is equivalent to mass murder.
- add-sub-mul-div 2y agoI want to believe he's bullshitting to hype it up for profit because at least that's not as bad as if it was sincere.
- krapp 2y agoI also want to believe AI hype is being fueled just by grifters and not an accelerationist messiah cult (and also grifters.) But they do seem really committed to the bit.
- mikro2nd 2y agoIt's not just for profit, it's to save some future, mythical version of humankind! https://netzpolitik.org/2023/longtermism-an-odd-and-peculiar-ideology/ https://netzpolitik.org/2023/longtermism-an-odd-and-peculiar...
- griomnib 2y agoYeah but it’s just shit engineering. They re-crawl entire sites basically continuously absent any updates or changes. How hard is it to cache a fucking sitemap for a week? It’s a waste of bandwidth and CPU on their end as well, “the bitter lesson” isn’t “keep duplicating the same training data”. I’m glad DeepSeek is showing how inefficient and dogshit most of the frontier model engineer is - how much VC is getting burned literally redownloading a copy of the entire web daily when like <1% of it is new data. I get they have no shame economically, that they are deluded and greedy. But bad engineering is another class of sin!