6 ms·

Place alongside https://news.ycombinator.com/item?id=44962529 https://news.ycombinator.com/item?id=44962529 "Why are anime catgirls blocking my access to the Li

• rnhmjoj 1y ago
  I'm far from being an AI enthusiast as anyone can be, but this issue has nothing to do with AI specifically. It's just that some greedy companies are writing incredibly shitty crawlers that don't follow any of the enstablished conventions (respecting robots.txt, using a proper UA string, rate limiting, whatever). This situation could have easily happened earlier than the AI boom, for different reasons.
  • mostlysimilar 1y ago
    But it didn't, and it's happening now, because of AI.
    • kjkjadksj 1y ago
      People have been complaining about these crawlers for years well before AI
      • PaulDavisThe1st 1y ago
        The issue is 1 to 4 orders of magnitude worse than it was just a couple of years ago. This is not "crawlers suck". This is "crawlers are overwhelming us and almost impossible to fully block". It really isn't the same thing.
        • tadfisher 1y ago
          Tragedy of the commons. Before, it was cryptominers eating up all free sources of compute [0]. Now it's AI crawlers eating up all available bandwidth and server resources [1]. Reading SourceHut's struggles against the Once-lers of the world makes me want to introduce a new application layer protocol where consumers pay for abusing shared resources. Which sucks, because the Internet should remain free. [0]: https://drewdevault.com/2021/04/26/Cryptocurrency-is-a-disaster.html https://drewdevault.com/2021/04/26/Cryptocurrency-is-a-disas... [1]: https://drewdevault.com/2025/03/17/2025-03-17-Stop-externalizing-your-costs-on-me.html https://drewdevault.com/2025/03/17/2025-03-17-Stop-externali...
          • PaulDavisThe1st 1y ago
            > Tragedy of the commons. No, because there is no such thing, at least not as understood by Garrett Hardin, who put forward the phrase. Commons fail when selfish, greedy people subvert or destroy the governance structures that help control them. If those governance structures exist (and they do for all historical commons) and continue to exist, the commons suffers no tragedy. This recent slide deck talks about Ostrom's ideas on this, which even Hardin eventually conceded were correct, and that his diagnosis of a "tragedy of the commons" does not actually describe the historical processes by which commons are abused. https://dougwebb.site/slides/commons https://dougwebb.site/slides/commons That said ... arguably there is a problem here with a "commons" that does in fact lack any real governance structure.
            • erlend_sh 1y ago
              No idea why this is getting downvoted; this is a very important correction since the “tragedy of the commons” meme is based on a flawed premise that needs to be amended.
        • p3rls 1y ago
          i am getting almost 500,000 ai scraper requests a day according to cloudflare's ai audit. google requests the same pages 10+ times each an hour. it was never this bad before.
  • majkinetor 1y ago
    Obaying robots.txt can not be enforced. Even if one country makes laws about it, another one will have 0 fucks to give.
    • spinningslate 1y ago
      It was never intended to be "enforced": > The standard, developed in 1994, relies on voluntary compliance [0] It was conceived in a world with an expectation of collectively respectful behaviour: specifically that search crawlers could swamp "average Joe's" site but shouldn't. We're in a different world now but companies still have a choice. Some do still respect it... and then there's Meta, OpenAI and such. Communities only work when people are willing to respect community rules, not have compliance imposed on them. It then becomes an arms race: a reasonable response from average Joe is "well, OK, I'll allows anyone but [Meta|OpenAI|...] to access my site. Fine in theory, dificult in practice: 1. Block IP addresses for the offending bots --> bots run from obfuscated addresses 2. Block the bot user agent --> bots lie about UA. ...and so on. [0]: https://en.wikipedia.org/wiki/Robots.txt https://en.wikipedia.org/wiki/Robots.txt
      • majkinetor 1y ago
        Thanks for the info. However people seem to think that robots.txt will protect them while it was created for another world as you nicelly stated. I guess Nepenthes like tools will be more common in the future, now that tragedy of commons entered digital domain.
  • Fomite 1y ago
    I'd argue it's part of the baked in, fundamental disrespect AI firms have for literally everyone else.
  • 1vuio0pswjnm7 1y ago
    "It's just that some greedy companies are writing incredibly shitty crawlers that don't follow any of the enstablished [sic] conventions (respecting robots.txt, using proper UA string, rate limiting, whatever)." How does "proper UA string" solve this "blowing up websites" problem The only thing that matters with respect to the "blowing up websites" problem is rate-limiting, i.e., behaviour "Shitty crawlers" are a nuisance because of their behaviour, i.e., request rate, not because of whatever UA string they send; the behaviour is what is "shitty" not the UA string. The two are not necessarily correlated and any heuristic that naively assumes so is inviting failure "Spoofed" UA strings have been facilitated and expected since the earliest web browsers For example, https://raw.githubusercontent.com/alandipert/ncsa-mosaic/master/mosaic-spoof-agents https://raw.githubusercontent.com/alandipert/ncsa-mosaic/mas... To borrow the parent's phrasing, the "blowing up websites" problem has nothing to do with UA string specifically It may have something to do with website operator reluctance to set up rate-limiting though; this despite widespread impelementation of "web APIs" that use rate-limiting NB. I'm not suggesting rate-limiting is a silver bullet. I'm suggesting that without rate-limiting, UA string as a means of addressing the "blowing up websites" problem is inviting failure
    • AbortedLaunch 1y ago
      Some of these crawlers appear to be designed to avoid rate limiting based on IP. I regularly see millions of unique ips doing strange requests, each just one or at most a few per day. When a response contains a unique redirect I often see a geographically distinct address fetching the destination.
      • 1vuio0pswjnm7 1y ago
        "I regularly see millions of unique ips doing strange requests, each just one or at most a few per day." How would UA string help For example, a crawler making "strange" requests can send _any_ UA string, and a crawler doing "normal" requests can also send _any_ UA string. The "doing requests" is what I refer to as "behaviour" A website operator might think "Crawlers making strange requests send UA string X but not Y" Let's assume the "strange" requests cause a "website load" problem^1 Then a crawler, or any www user, makes a "normal" request and sends UA string X; the operator blocks or redirects the request, unnecessarily Then a crawler makes "strange" request and sends UA string Y; the operator allows the request and the website "blows up" What matters for the "blowing up websites" problem^1 is behaviour, not UA string 1. The article's title calls it the "blowing up websites" problem, but the article text calls it a problem with "website load". As always the details are missing. For example, what is the "load" at issue. Is it TCP connections or HTTP requests. What number of simultaneous connections and/or requests per second is acceptable, what number is not unacceptable. Again, behaviour is the issue, not UA string The acceptable numbers need to be published; for example, see documentation for "web APIs"
  • sznio 1y ago
    I strongly believe that AI companies are running a DDOS attack on the open web. Making websites go down aligns with their intetests: it removes training data that competitors could use, and it removes sources for humans to browse, making us even more reliant on chatbots to find anything. If it was crap coding, then the bots wouldn't have so many mechanisms to circumvent blocks. Once you block the OpenAI IP ranges, they start using residential proxies. Once you block their UA strings, they start impersonating other crawlers or browsers.
• superkuh 1y ago
  This isn't AI damaging anything. This is corporations damaging things. Same as it ever was. No need for scifi non-human persons when legal corporate persons exist. They latch on to whatever big new thing in tech that people don't understand which comes along and brand themselves with it and cause damage trying to make money; even if they mostly fail at it. And for most actual humans they only ever see or interact with the scammy corporation versions of $techthing and so come to believe $techthing = corporate behavior. And as for denying service and preventing human people from visiting websites: cloudflare does more of that damage in a single day than all these "AI" associated corporations and their crappy crawlers have in years.
  • autoexec 1y ago
    > This isn't AI damaging anything. This is corporations damaging things. This is corporations damaging things because of AI. Corporations will damage things for other reasons too but the only reason they are breaking the internet in this way, at this time, is because of AI. I think the "AI doesn't kill websites, corporations kill websites" argument is as flawed as the "Guns don't kill people, people kill people" argument.
    • superkuh 1y ago
      Correct. It's a good, legitimate argument in both contexts. I use both local AI and local firearms as a human person and I am not doing, and have not done, damage to anyone. The tools aren't the problem. The problem in this case is the near complete protection from legal liability that corporate structures give to the people behaving badly. Like how Coca Cola can get away with killing people (https://prospect.org/features/coca-cola-killings/ https://prospect.org/features/coca-cola-killings/) but a person can't, if you want to keep the firearms analogy going. But it's a bad analogy because the firearms as tool actually at least are involved in the bad (and good) actions. AI itself isn't even involved in the HTTP requests and probably isn't even running on the same premises.
  • ujkhsjkdhf234 1y ago
    Cloudflare exists because people can't be good stewards of the internet. > This isn't AI damaging anything. This is corporations damaging things This is the guns don't kill people, people kill people argument. The problem with crawlers is about 10x worse than it was previously because of AI and their hunger for data.
• renewiltord 1y ago
  If you don't want to receive data, don't. If you don't want to send data, don't. No one is asking you to receive traffic from my IPs or send to my IPs. You've just configured your server one way. Or to use a common HN aphorism “your business model is not my problem”. Disconnect from me if you don’t want my traffic.
  • PaulDavisThe1st 1y ago
    I don't know if I want your traffic until I see what your traffic is. You want to look at one of our git commits? Sure! That's what our web-fronted git repo is for. Go right head! Be our guest! Oh ... I see. You want to download every commit in our repository. One by one, when you have used git clone. Hmm, yeah, I don't want your traffic. But wait, "your traffic" seems to originate from ... consults fail2ban logs ... more than 900k different IP addresses, so "disconnecting" from you is non-trivial. I can't put it more politely than this: fuck off. Do not pass go. Do not collect stock options. Go to hell, and stay there.
    • renewiltord 1y ago
      There's a protocol for that. Just reject the connection. Don't implode, just write some code. Your business model isn't my problem.
      • PaulDavisThe1st 1y ago
        Reject the connection based on what? IP address (presumably after too many visits) ? So now the iptables mechanism has to scale to fit your business model (of hammering my git repository 1 commit at a time from nearly a million IP addresses) ? Why does the code I use have to fit your braindead model? We wouldn't care if you just used git clone, but you're too dumb to do that. The URL? Legitimate human (or other) users won't be happy about that. Our web-fronted git repo is not part of our business model. It's just a free service we like to offer people, unrelated to revenue flow or business operations. So your behavior is not screwing my business model, but it is screwing up people who for whatever reason want to use that service, who can no longer use the web-fronted git repo. ps. I've used "you" throughout the above because you used "my". No idea if you personally are involved in any such behavior.
• IT4MD 1y ago
  >AI is going to damage society not in fancy sci-fi ways but by centralizing profit made at the expense of everyone else on the internet 10/10. No notes.
• msgodel 1y ago
  This isn't really about AI. This is a couple corporations being bad netizens and abusing infrastructure. The same incentives to do this already existed for search engine operators.