6 ms·
> Scraping static content from a website at near-zero marginal cost to its server, vs scraping an expensive LLM service provided for free, are different things.
by PunchyHamster 7mo ago
> Scraping static content from a website at near-zero marginal cost to its server, vs scraping an expensive LLM service provided for free, are different things.
I bet people being fucking DDOSed by AI bots disagree
Also the fucking ignorance assuming it's "static content" and not something needing code running
- Den_VR 7mo agoI miss the www where the .html was written in vim or notepad.
- holler 7mo agoahh yes, fresh off reading "Html For Dummies" I made my first tripod.com site
- consp 7mo agoJust did that for a test frontend for a module I needed to build (not my primary job so don't know anything about UI but running in browsers was a requirement), so basic HTML with the bare minimum of JS and all DOM. Colleagues were very surprized. And yes, vim is still the goto editor and will be for a long time now all "IDE" are pushing "AI" slop everywhere.
- mghackerlady 7mo agoIt still can be. Do it. Go make your website in M$ Frontpage, for all I care
- butlike 7mo agoShameless plug: My music homepage follows the HTML 2.0 spec and is written by hand https://sampleoffline.com/ https://sampleoffline.com/
- mghackerlady 7mo agoheck yeah B)
- eru 7mo ago> I bet people being fucking DDOSed by AI bots disagree Are you sure it's a DDoS and not just a DoS?
- SolarNet 7mo agoWhen every AI company does it from multiple data centers... yes it's distributed.
- Bilal_io 7mo agoUncoordinated DDoS, when multiple search and AI companies are hammering your server.
- susupro1 7mo ago[dead]
- troyvit 7mo agoYou should see Cloudflare's control panel for AI bot blocking. There are dozens of different AI bots you can choose to block, and that doesn't even count the different ASNs they might use. So in this case I'd say that a DDoS is a decent description. It's not as bad as every home router on the eastern seaboard or something, but it's pretty bad.
- catoc 7mo ago> Are you sure it's a DDoS and not just a DoS? I think these days it’s ‘DAIS’, as in your site just DAIS - from Distributed/Damned AI Scraping
- MattJ100 7mo agoYes, it is. The worst offenders hammer us (and others) with thousands upon thousands of requests, and each request uses unique IP addresses making all per-IP limits useless. We implemented an anti-bot challenge and it helped for a while. Then our server collapsed again recently. The perf command showed that the actual TLS handshakes inside nginx were using over 50% of our server's CPU, starving other stuff on the machine. It's a DDoS.
- lm411 7mo ago> Also the fucking ignorance assuming it's "static content" and not something needing code running Wild eh. If it's not ai now, it's by default labelled "static content" and "near-zero marginal cost".
- littlestymaar 7mo agoWhat's a database after all.
- eloisius 7mo agoAlso wild that from the tech bro perspective, the cost of journalism is just how much data transfer costs for the finished article. Authors spend their blood, sweat and tears writing and then OpenAI comes to Hoover it up without a care in the world about license, copyright or what constitutes fair use. But don’t you dare scrape their slop.
- lelanthran 7mo ago> Also wild that from the tech bro perspective, the cost of journalism is just how much data transfer costs for the finished article. Exactly. I think the unfairness can be mitigated if models trained on public information, or on data generated by a model trained on public information, or has any of those two in its ancestry, must be made public. Then we don't have to hit (for example) Anthropic, we can download and use the models as we see fit without Anthropic whining that the users are using too much capacity.
- mikkupikku 7mo ago[flagged]
- jazzyjackson 7mo agoThe library's archive is not a service provided by the newspaper
- mikkupikku 7mo agoSo? If the newspaper's website is willing to serve the documents, what's the problem? The point is, if you're pleading with others to respect ""intellectual property"" then you're a worm serving corporate interests against your own.
- jazzyjackson 7mo agoI may be a worm but at least I respect that others might have a different take on how best to make creative work an attainable way of life since before copyright law it was basically "have a wealthy patron who steered if not outright commissioned what you would produce"
- remus 7mo agoI think the parent is just pointing out that these things lie on a spectrum. I have a website that consists largely of static content and the (significant) scraping which occurs doesn't impact the site for general users so I don't mind (and means I get good, up to date answers from LLMs on the niche topic my site covers). If it did have an impact on real users, or cost me significant money, I would feel pretty differently.
- 0xEF 7mo agoPutting everything on a spectrum is what got us into this mess of zero regulation and moving goal posts. It's slippery slope thinking no matter which way we cut it, because every time someone calls for a stop sign to be put up after giving an inch, the very people who would have to stop will argue tirelessly for the extra mile.
- daveidol 7mo agoI’d argue putting everything in terms of black and white is the bigger issue than understanding nuance
- instig007 7mo agoGeneralizing with "everything", "all", etc exclusive markers is exactly the kind of black/white divide you're arguing against. What happened to your nuanced reality within a single sentence? Not everything is black and white, but some situations are.
- fc417fc802 7mo agoThe person he's replying to argued against putting things on a spectrum. Does that not imply painting everything in black and white? Thus his response seems perfectly sensible to me.
- instig007 7mo agoHe argued against putting things in a spectrum in many instances where that would be wrong, including the case under the question. What's your argument against that idea? LLM'ed too much lately?
- 1718627440 7mo agoOff topic, but why is a DoS something considered to act on, often by just shutting down the service altogether? That results in the same DoS just by the operator than due to congestion. Actually it's worse, because now the requests will never actually be responded rather then after some delay. Why is the default not to just don't do anything?
- echoangle 7mo agoI think some people use hosting that is paid per request/load, so having crawlers make unwanted requests costs them money.
- ImPostingOnHN 7mo ago*> Why is the default not to just don't do anything? Because ingress and compute costs often increase with every request, to the point where AI bot requests rack up bills of hundreds or thousands of dollars more than the hobbyist operator was expecting to send.
- pocksuppet 7mo agoIt keeps the other projects hosted on the same server or network online. Blackhole routes are pushed upstream to the really big networks and they push them to their edge routers, so traffic to the affected IPs is dropped near the sender's ISP and doesn't cause network congestion. DDoSers who really want to cause damage now target random IPs in the same network as their actual target. That way, it can't be blackholed without blackholing the entire hosting provider.
- nikitaga 7mo agoAll this reactionary outrage in the comments is funny. And lame. Yes, for the vast majority of the internet, serving traffic is near zero marginal cost. Not for LLMs though – those requests are orders of magnitude more expensive. This isn't controversial at all, it's a well understood fact, outside of this irrationally angry thread at least. I don't know, maybe you don't understand the economic term "marginal cost", thus not understanding the limited scope of my statement. If such DDOSes as you mention were common, such a scraping strategy would not have worked for the scraper at all. But no, they're rare edge cases, from a combination of shoddy scrapers and shoddy website implementations, including the lack of even basic throttling for expensive-to-serve resources. The vast majority of websites handle AI traffic fine though, either because they don't have expensive to serve resources, or because they properly protect such resources from abuse. If you're an edge case who is harmed by overly aggressive scrapers, take countermeasures. Everyone with that problem should, that's neither new nor controversial.
- expedition32 7mo agoOne euro is marginal for me for someone else it is their daily meal.
- grayhatter 7mo agoThe extent of the utilization is new. The number of bots that try to hide who they are, and don't bother to even check robots.txt is new.
- fireflash38 7mo agoIt's not a cost for me to scrape LLM. It is a cost for me for LLM to scrape me. Why should I care about costs that have when they don't care about the costs I have?
- ipaddr 7mo ago"such DDOSes as you mention were common, such a scraping strategy would not have worked for the scraper at all" They are common. The strategy works for the llm but not for the website owner or users who can't use a site during this attack. The majority of sites are not handling AI fine. Getting Ddosed only part of the time is not acceptable. Countermeasures like blocking huge ranges can help but also lock out legimate users.