20 ms·

FOSS infrastructure is under attack by AI companies

• cudgy 2y ago
  “… he complained that LLM companies were crawling data without respecting robosts.txt“ Maybe that is the problem? They misspelled robots.txt
• MR4D 2y ago
  Might be time for a class action lawsuit. Something like that could work out really well for the little guy as it would probably make a dent in the LLM companies pockets and access to data.
• totetsu 2y ago
  Could one put a mangler on the responses to suspected bots to poison their data sets with nonsense code.. :/
  • someothherguyy 2y ago
    Cloudflare is offering something like this: https://blog.cloudflare.com/ai-labyrinth/ https://blog.cloudflare.com/ai-labyrinth/
  • niemandhier 2y ago
    This. Not only mangle the content. Flood the bot with tailored misinformation and things that are illegal in this particular legislation but not yours. They will never respect you, but the second they notice this hurts their business more than it gains them, they will stop.
    • moomin 2y ago
      Combine this with the Anubis tech...
  • flir 2y ago
    I wonder if that would just make them try harder. Scrape multiple times and diff, to gain confidence that the data hasn't been posioned. We've never had one of these arms races end up with the defenders winning.
  • postexitus 2y ago
    as long as you can detect the bot is an ai crawler.
  • hans_castorp 2y ago
    I thought the same. Maybe start prefixing each commit message with "LLVM bots are all lying bastards" or something similar :)
  • pfoof 2y ago
    a proper CSS or comments in HTML might do this
  • usefulcat 2y ago
    If you could conclusively identify bots, you could not serve them anything and save yourself some work (and bandwidth). If you can't conclusively identify bots, you'll end up serving 'poisoned' responses to actual users. Doesn't seem like a viable solution.
• ericholscher 2y ago
  Yep -- our story here: https://about.readthedocs.com/blog/2024/07/ai-crawlers-abuse/ https://about.readthedocs.com/blog/2024/07/ai-crawlers-abuse... (quoted in the OP) -- everyone I know has a similar story who is running large internet infrastructure -- this post does a great job of rounding a bunch of them up in 1 place. I called it when I wrote it, they are just burning their goodwill to the ground. I will note that one of the main startups in the space worked with us directly, refunded our costs, and fixed the bug in their crawler. Facebook never replied to our emails, the link in their User Agent led to a 404 -- an engineer at the company saw our post and reached out, giving me the right email -- which I then emailed 3x and never got a reply.
  • pjc50 2y ago
    > just burning their goodwill to the ground AI firms seem to be leading from a position that goodwill is irrelevant: a $100bn pile of capital, like an 800lb gorilla, does what it wants. AI will be incorporated into all products whether you like it or not; it will absorb all data whether you like it or not.
    • anthk 2y ago
      AI tarpits && lim (human curated contant/mediocre AI answers -> 0) = AI's crumbling into dust by themselves.
    • UncleMeat 2y ago
      Yep. And it is much more far reaching than that. Look at the primary economic claim offered by AI companies: to end the need for a substantial portion of all jobs on the planet. The entire vision is to remake the entire world into one where the owners of these companies own everything and are completely unconstrained. All intellectual property belongs to them. All labor belongs to them. Why would they need good will when they own everything? "Why should we care about open source maintainers" is just a microcosm of the much larger "why should we care about literally anybody" mindset.
      • chii 2y ago
• nonrandomstring 2y ago
  These are DDOS attacks and should be treated in law as such. (Although I do realise that in many countries now we no longer have any effective "rule of law")
  • rvnx 2y ago
    In the world, the richest wins, not the nicest (cf. Sam Altman)
    • nonrandomstring 2y ago
      lol. Tell that to Nicolae and Elena Ceaucescu, Saddam Hussain, Muammar Gaddafi, and all those other tinpot nobjockeys who thought their money and influence would save them from "nice" people.
  • WesolyKubeczek 2y ago
    At some point it's easier to geoblock a whole country at the firewall level and loginwall the rest of the world, rather than trying to explain that in your jurisdiction, which is not their jurisdiction, what they are doing is a crime — which they don't give a single fuck about.
    • deleted 2y ago
      [deleted]
  • briandear 2y ago
    You going to prosecute China?
    • immibis 2y ago
      Is that where Sam Altman is hiding?
  • ttyprintk 2y ago
    Each of these crawlers is as much a DoS as Aaron Swartz.
• KolmogorovComp 2y ago
  My question is, can we serve PoW challenges to these AI LLM scrapers that can be profitable?
  • rswail 2y ago
    Finally, a reason for bitcoins!
  • xena 2y ago
    That's what I've been doing! It works shockingly well. https://github.com/TecharoHQ/anubis https://github.com/TecharoHQ/anubis
    • KolmogorovComp 2y ago
      Unless I am missing something, the result of that generated work has no monetary value though.
      • ziddoap 2y ago
        The monetary value is not having a misbehaving AI bot download 73TB or whatever of your data.
      • xena 2y ago
        I was inspired by https://en.wikipedia.org/wiki/Hashcash https://en.wikipedia.org/wiki/Hashcash, which was proof of work for email to disincentivize spam. To my horror, it worked sufficiently for my git server so I released it as open source. It's now its own project and protects big sites like GNOME's GitLab.
        • 01HNNWZ0MV43FF 2y ago
          That's cool! What if instead of sha256 you used one of those memory-hard functions like script? Or is sha needed because it has a native impl in browsers?
• WesolyKubeczek 2y ago
  ...at some point, some people started appreciating mailing lists and the distributed nature of Git again.
  • anthk 2y ago
    And Usenet, and IRC with a registered user prereq to join. Also, set AI tarpits as fake links with recursive calls. Make then mad with non-curated bullshit made from Markov chain generators until their cache begins to rot forever.
    • ewzimm 2y ago
      This problem will likely only get worse, so I'd be interested to see how people adapt. I was thinking about sending data through the mail like the old days. Maybe we go back to the original Tim Berners-Lee Xanadu setup charging users small amounts for access but working out ISP or VPN deals to give subscribers enough credit to browse without issues.
      • krapp 2y ago
        Xanadu was Ted Nelson, not Tim Berners-Lee. Also I would argue that not having capitalist incentives baked directly into the network is what made the web work, for good or bad. Xanadu would never have gotten off the ground if people had to pay their ISP then pay for every website, or every packet, or every clicked link or whatever. Reading the Xanadu page on Wikipedia tells me "Every document can contain a royalty mechanism at any desired degree of granularity to ensure payment on any portion accessed, including virtual copies ("transclusions") of all or part of the document." That would be absolute chaos at scale.
        • ewzimm 2y ago
          Oops, you're right! They claimed that Tim Berners-Lee stole their idea. I agree that the lack of monetization was important to the development and that it would have been chaos as proposed, but will the current setup be sustainable forever in the world of AI? We have projects like Ethereum that are specifically intended to merge payments and computing, and I wouldn't be surprised if at some point in the future, some kind of small access fee negotiated in the background without direct user involvement become a component of access. I wouldn't expect people to pay ISPs but rather some kind of token exchange to occur that would benefit both the network operators and the web hosts by verifying classes of users. Non-fungible token exchanges could be used as a kind of CATPCHA replacement by cryptographically verifying users anonymously with a third-party token holder as the intermediary. For example, let's say Mullvad or some other VPN company purchased a small amount of verification tokens for its subscribers who pay them anonymously for an account. On the other side, let's say a government requires people to register through their ISP, and the ISP purchases the same tokens on behalf of the user, and then exchanges the tokens on behalf of the user. In either case, the person can stand behind a third party who both sends them the data they requested and exchanges the verification tokens, which the site operator could then exchange for reimbursement of their services to their hosting provider. This is just a high-level idea of how we might get around the challenges of a web dominated by bots and AI, but I'm sure the reality of it will be more interesting.
• roenxi 2y ago
  We're close to finding a clear use-case for Bitcoin with this one.
  • wyager 2y ago
    Interesting. Basically moving the proof-of-work off the user's phone and to a dedicated mine. Websites could just have a lightning wallet or something and auto-charge the user 1e-7 bitcoin to access the page.
    • falcor84 2y ago
      I would absolutely love this as an alternative to the ad-based model
      • nottorp 2y ago
        That is called micropayments and doesn't need to have anything to do with blockchain. All you need is a central clearing house service that can handle billions of 0.000001 transactions per day. Incidentaly, I doubt the bitcoin chain could handle that...
        • pca006132 2y ago
          There are off-chain solutions to handle most of the payment, and only put a summary on-chain. I think there are already micropayments in Brave or something.
          • Vegenoid 2y ago
            At this point, why even use bitcoin, or crypto at all. Just use a crypto that is better suited to micropayments.
            • rhaps0dy 2y ago
              Proof-of-work crypto is interesting here because it is fungible with computation, so these solutions that charge computation to users are literally equivalent to crypto. It's a solution that already has adoption, does not require everyone to sign up with a centralized service, and does not require everyone to pay money (they can pay with small amounts of computation instead) so it remains accessible to ~everyone.
• brandonmenc 2y ago
  This article starts by citing a blog article - displays a screenshot of the article - but doesn't link to it.
  • unwind 2y ago
    It was this: [1], posted here yesterday I think. [1]: https://drewdevault.com/2025/03/17/2025-03-17-Stop-externalizing-your-costs-on-me.html https://drewdevault.com/2025/03/17/2025-03-17-Stop-externali...
  • nicco-love 2y ago
    Niccolò here, I'm really sorry about that -- I'm using a weird tooling system to handle articles, which currently has issues with links. I'm working to fix that asap.
    • s3wall 2y ago
      screenshots in place of links, cited articles likely shadow-banned on HN, FOSS is under attack 2.0.
  • mook 2y ago
    It also mentions Corbet of LWN but missed the article about it: https://lwn.net/Articles/1008897/ https://lwn.net/Articles/1008897/
• napolux 2y ago
  In the past month, I've had to block LLM bots attacking my poor little VPSs — not once, but twice. First, it was Facebook https://news.ycombinator.com/item?id=23490367 https://news.ycombinator.com/item?id=23490367 and now it's these other companies. What's worse? They completely ignore a simple HTTP 429 status.
  • DamonHD 2y ago
    Almost no one pays attention to 429 for general Web pages. Eg on an RSS file Googlebot speeds up repolling. Amazon RSS feed puller does pay some attention to 429. 503 is at least apparently understood by more crawlers/bots, but they still like to blame the victim: YouTube sends me a condescending (and inaccurate) email when it gets a 503 for ignoring cache headers and other basics it seems...
• wiredfool 2y ago
  Across my sites -- mostly open data sites -- the top 10 referrers are all bots. That doesn't include the long tail of randomized user agents that we get from the Alibaba netblocks. At this point, I think we're well under 1% actual users on a good day.
• megadata 2y ago
  Perhaps time to start a central community ban pool for IP ranges?
  • MartijnBraam 2y ago
    Doesn't really work if crawlers are coming from the IP ranges of AWS and Azure etc...
    • Tostino 2y ago
      Or sometimes they use consumer IP proxies. Makes it even harder because sometimes those IPs get reused for actual users.
      • megadata 2y ago
        Block them for 24 hours.
    • immibis 2y ago
      Traditionally, holders of IP ranges that attack the internet at large get kicked off the internet by having those ranges blacklisted everywhere. This can also get them in serious trouble with the places they got their IP ranges (I assume AWS has them directly from ARIN, so maybe not) and their upstream bandwidth providers and so on, as well as making them less attractive hosts because they are blocked everywhere.
      • mrweasel 2y ago
        Kicking AWS of the internet would effectively break half of everything. So much stuff is relying on services running on AWS that it's not even funny. We've seen random stuff break when AWS has had outages, not because we used AWS ourselves, but because suppliers do.
        • megadata 2y ago
          You don't need to kick anyone off the internet. Just ban them for accessing your resources.
• sir-alien 2y ago
  It's going to get to the point where everything will be put behind a login to prevent LLM scrapers scanning a site. Annoying but the only option I can think of. If they use an account for scraping you just ban the account.
  • napolux 2y ago
    and then they'll login there too...
    • sir-alien 2y ago
      logins are more easily banned, and highly complex captchas for signup needs a human to signup and solve. As long as it's easier to get banned than it is to signup it will at least deter.
• briandear 2y ago
  Does GitHub have this problem?
  • reportgunner 2y ago
    afaik github is microsoft and openai is a partner of microsoft so probably not
  • deleted 2y ago
    [deleted]
  • myaccountonhn 2y ago
    I've started hitting rate limits when just cloning regular repos now.
• xena 2y ago
  It's really surreal to see my project in the preview image like this. That's wild! If you want to try it: https://github.com/TecharoHQ/anubis https://github.com/TecharoHQ/anubis. So far I've noticed that it seems to actually work. I just deployed it to xeiaso.net as a way to see how it fails in prod for my blog.
  • diggan 2y ago
    Nice work :) One piece of feedback: Could you add some explanation (for humans) what we're supposed to do and what is happening when met by that page? I know there is a loading animation widget thingy, but the first time I saw that page (some weeks ago at the Gnome issue tracker), it was proof-of-work'ing for like 20 seconds, and I wasn't sure what was going on, I initially thought I got blocked or that the captcha failed to load. Of course, now I understand what it is, but I'm not sure it's 100% clear when you just see the "checking if you're a bot" page in isolation.
    • xena 2y ago
      > One piece of feedback: Could you add some explanation (for humans) what we're supposed to do and what is happening when met by that page? Will do! https://github.com/TecharoHQ/anubis/issues/25 https://github.com/TecharoHQ/anubis/issues/25
      • ranger_danger 2y ago
        also if you're using JShelter, which blocks Worker by default, there is no indication that it's never going to work, and the spinner just goes on forever doing nothing
        • xena 2y ago
          Noted! I filed a bug: https://github.com/TecharoHQ/anubis/issues/38 https://github.com/TecharoHQ/anubis/issues/38 All of this is placeholder wording, layouts, CSS, and more. It'll be fixed in time. This is teething pain that I will get through.
• aspir 2y ago
  Just a callout that Fastly provides free bot detection, CDN, and other security services for FOSS projects, and has been for 10+ years https://www.fastly.com/fast-forward https://www.fastly.com/fast-forward (disclaimer, I work for Fastly and help with this program) Without going into too much detail, this tracks with the trends in inquiries we're getting from new programs and existing members. A few years ago, the requests were almost exclusively related to performance, uptime, implementing OWASP rules in a WAF, or more generic volumetric impact. Now, AI scraping is increasingly something that FOSS orgs come to us for help with.
  • Aachen 2y ago
    I've been running into bot detection on at least five different websites in the past two months (not even including captcha walls) Not sure what to tell you but I surely feel quite human Three of the pages told me to contact customer support and the other two were a hard and useless block wall. Only from Codeberg did I get a useful response, the other two customer supports were the typical "have you tried clearing your cookies" and restart the router advice — which is counterproductive because cookie tracking is often what lets one pass. Support is not prepared to deal with this, which means I can't shop at the stores that have blocking algorithms erroneously going off. I also don't think any normal person would ever contact support, I only do it to help them realise there's a problem and they're blocking legitimate people from using the internet normally Beware if you employ this...
    • RVuRnvbM2e 2y ago
      Were the walls you hit caused by Fastly's bot detection? I've found it to be quite accurate. On the other hand CloudFlare and Akamai mistakenly block me all the damn time.
      • Aachen 2y ago
        It's not like they say, but it's at least three different implementations and I don't think any were cloudflare because I've been running into those pages for years and they've got captchas (functional or not). One of them was Akamai I think indeed
• keepamovin 2y ago
  Adaptation - adapt or die. Find a business model that can sustain, without the naivety that people will pay for what they can take without consequence.
• brushfoot 2y ago
  At this rate, it's more than FOSS infrastructure -- although that's a canary in the coalmine I especially sympathize with -- it's anonymous Internet access altogether. Because you can put your site behind an auth wall, but these new bots can solve the captchas and imitate real users like never before. Particularly if they're hitting you from residential IPs and with fake user agents like the ones in the article -- or even real user agents because they're wired up to something like Playwright. What's left except for sites to start requiring credit cards, Worldcoin, or some equally depressing equivalent.
  • Freak_NL 2y ago
    We're half way there already. It always hits me whenever I am doing some mapping for OpenStreetMap and I'm looking up local businesses without their own internet presence. They use Facebook, Instagram, X, etc. for their digital calling card. I normally don't use Facebook (or Instagram, and gave up on X) and have no account there, and every time I follow one of those links, you get some info, and then you get a dialogue screen telling you to make an account or get lost, or you just get some obscure error. I don't mind registering an account for private communities, but for stuff which people put up thinking it is just going to be publicly visible it's really annoying.
    • yurishimo 2y ago
      > ... but for stuff which people put up thinking it is just going to be publicly visible ... I don't think these business owners really understand. Most normies just think everyone has a Facebook/Instagram account and can't even imagine a world where that is not the case. I agree with you that it is extremely frustrating.
      • Suppafly 2y ago
        >Most normies just think everyone has a Facebook/Instagram account and can't even imagine a world where that is not the case. The people without a basic internet presence aren't likely to be customers anyway so it's not a huge loss. It's trivial to setup a basic account for any site that doesn't contain any personal data you want to keep hidden, if you aren't willing to do that, you're in a tiny minority.
• marginalia_nu 2y ago
  I wonder if the future is for honest crawlers to do something like DKIM to provide a cheap cryptographically verifiable identity, where reputation can be staked on good behavior, and to treat the rest of the traffic like it's a full fledged chrome instance that had better be capable of solving hashcash challenges when traffic gets too hot. It's a shitty solution, but as it stands the status quo is quite untenable and will eventually have cloudflare as a spooky MITM for all the web's traffic.
• FeepingCreature 2y ago
  To be clear, this is not an attack in the deliberate sense, and has nothing to do with AI except in that AI companies want to crawl the internet. This is more "FOSS sites damaged by extreme incompetence and unaccountability." The crawlers could just as well be search engine startups.
  • GuinansEyebrows 2y ago
    I think there’s room to argue that that’s a matter of perspective. The intent of the attacker is not that important to the people being attacked.
  • joepie91_ 2y ago
    > The crawlers could just as well be search engine startups. And yet they are not. So what does that tell you?
  • soco 2y ago
    Correct, they could - but they are not. This is about the unaccountability, and if I was charitable (which I'm not) I'd add also incompetence, of the techbros leading the AI giants. Are we still expecting anything like "ethics"? I hope the few engineers reading HN will still have some, but the higher you go the foreign the concept gets.
    • FeepingCreature 2y ago
      Sure, I just think focusing it on AI companies misses the reason this happens. It's not an "AI company problem", it's a "tech company problem". It just happens that AI companies are the tech companies that externalize their incompetence with crawlers at this point in time.
  • MrArthegor 2y ago
    Yes, any crawler or some traffic peak can do this if your infrastructure is not well engineered
    • 2y ago
• lelanthran 2y ago
  The big takeaway here is that Google's (and advertisement in general) dominance over the web is going away. This is because the only way to stop the bots is with a captcha, and this also stops search indexers from indexing your site. This will result in search engines not indexing sites, and hence providing no value anymore. There's probably going to be a small lag as the current knowledge in the current LLMs dry up because no one can scrape the web in an automated fashion anymore. It'll all burn down.
  • nicce 2y ago
    > The big takeaway here is that Google's (and advertisement in general) dominance over the web is going away. AI companies with best anti-captcha mechanics will win and will inject ads to LLM output in more sophisticated way.
  • breckenedge 2y ago
    Google is already scraping your site and presenting answers directly in search results. If I cared about traffic (hence selling ad space), why would I want my site indexed by Google at all anymore? Lots of advertising-supported sites are going to go dark because only bots will visit them.
  • djha-skin 2y ago
    I actually envision Liapunov stability, like wolf and rabbit populations. In this scenario, we're the rabbits. Human content will increase when AI populations decrease, this providing more food for AI, which will then increase. This drowns out human expression, and the humans will grow quieter. This provides less fodder for the AI, and they decrease. This means less noise and the humans grow louder. The cycle repeats and nauseam.
    • GolfPopper 2y ago
      Until broken by the Butlerian Jihad, "Though shalt not make a machine in the likeness of the mind of man."
    • keyringlight 2y ago
• immibis 2y ago
  This is probably a dumb question, but have they tried sending abuse reports to hosting providers? Or even lawsuits? Most hosting providers take it seriously when their client is sending a DoS attack, because if they don't, they can get kicked off the internet by their provider.
• QuadrupleA 2y ago
  Isn't this just poor, sloppy crawler implementation? You shouldn't need to fetch a repo more than once to add it to a training set.
  • diggan 2y ago
    Sounds like it me. Why build a crawler that fetches one HTML page per commit in a repository instead of doing a bare-clone and then just get the data from there? Surely would contain even more data too, compared to the HTML page.
  • csydas 2y ago
    You are correct it's poor and sloppy, but it's not "just" that. It's a lack of concern over the effects of their poor/sloppy crawler implementation. The poor implementation is not really relevant, it's companies deciding they own the internet and can take whatever they want, let everyone else deal with the consequences. The companies do not care what the impact of their ai non-sense is..
    • Forbo 2y ago
      A capitalist externalizing costs?! Why, no, never! /s
    • fuzztester 2y ago
      https://news.ycombinator.com/item?id=43429532 https://news.ycombinator.com/item?id=43429532
  • Havoc 2y ago
    It’s people that don’t care if they ruin things for everyone else. Crawlers have existed forever in search engine space and mostly behave. This sort of no rate limit, fake user agent, 100s of IPs approach used by AI teams is obviously intentionally not caring who it fucks over. More malicious than sloppy implementation
    • mistrial9 2y ago
      it is an ecosystem of social roles, not just "people" .. casting the decision into individual choices is not the right filter to understand this situation..
• deleted 2y ago
  [deleted]
• rglullis 2y ago
  The days of an open web are long gone. Every server will eventually have to require authentication for access, and to get an account you will have to provide some form of payment or social proof. Honestly, I don't see it necessarily as a bad thing.
  • kittikitti 2y ago
    "some form of social proof." Sounds like sanctioned racism.
    • rglullis 2y ago
      Oh, stop. I'm talking about social proof as in "You are an student of the city university, so you get an account at the library", "Julie from the book reading group wanted an account at our Bookwyrm server, so I made an account for her" or even "Unnamed customer who signed up for Cingular Wireless and was given an authorization code to access Level 2 support directly".
      • kittikitti 2y ago
        This is being naive about the kinds of gatekeeping and social proof occurring today. I fully believe you didn't intend to mention social proof to be racist, but with people like Zuck and Elon removing DEI, being racist is social proof you belong in their elite club.
        • rglullis 2y ago
          > This is being naive about the kinds of gatekeeping and social proof occurring today You are taking one thing I said (service providers will require some form of payment or social proof to give credentials to people who want to access the service), assumed the worst possible interpretation (people will only implement the worst possible forms of social proofing), and to top it off you added something else (gatekeeping) entirely on your own. I can not dictate how you interpret my comment, but maybe could you be a bit more charitable and assume positive intent when talking with people you never met?
• harhargange 2y ago
  Can IPFS or torrent and large local databases decentralised by people be a solution to this? I personally have the resources to share and host TBs of data but didn't find a good use to it.
  • GTP 2y ago
    I doubt, as the article mentions scraping the same resource after just 6 hours. AI companies want to make sure they have fresh data, whileit would be hard to keep such a database updated.
  • smashah 2y ago
    The real solution to this is for cloud fees to fall by 80%.
  • finnthehuman 2y ago
    For that to work, a website has to push a mirror into that alternate system, and the scraper has to know the associated mirror exists. That's two big "ifs" for something I'm not aware of a standardized way of announcing. And the entire thing crumbles as soon as someone who wants every drop of data possible says "crawl their sites anyway to make sure they didn't forget to publish anything into the 2nd system."
  • Self-Perfection 2y ago
    This thought came to me as well. This way crawlers might contribute back by providing extra storage and bandwidth. Though something like ZeroNet seems a better approach to allow dynamic content.
• diggan 2y ago
  > According to Drew, LLM crawlers don't respect robots.txt requirements and include expensive endpoints like git blame, every page of every git log, and every commit in your repository. They do so using random User-Agents from tens of thousands of IP addresses, each one making no more than one HTTP request, trying to blend in with user traffic. How do they know that these are LLM crawlers and not anything else?
  • JimDabell 2y ago
    That was my reaction. It seems like the article is saying two mutually exclusive things: - We cannot block them because we can’t differentiate legitimate traffic from illegitimate traffic… - …but we can conclusively identify this traffic as coming from AI crawlers.
    • xena 2y ago
      It's a situation where it's difficult to tell for individual requests at request handling time, but easy to see when you look at the total request volume.
    • chneu 2y ago
      It's the behavior of the traffic in hindsight that's obvious. It's difficult to identify in the moment. This is by design. Getting caught isn't a big deal. Getting caught in the act is. As long as they get their data, it doesn't matter if they're caught afterwards.
  • TonyTrapp 2y ago
    As someone that is also affected by this: We see a manifold increase in requests since this LLM crap is going on. Many of these IPs come from companies that obviously work with LLM technology, but the problem is that it's 100s of IPs doing 1 request, not 1 IP doing 100s of requests. It's just extremely unlikely that anyone else is responsible for this.
    • diggan 2y ago
      > IPs come from companies that obviously work with LLM technology Like from their own ASNs you're saying? Or how are you connecting the IPs with the company? > is that it's 100s of IPs doing 1 request Are all of those IPs within the same ranges or scattered? Thanks a lot for taking the time to talk about your experience btw, as someone who hasn't been hit by this it's interesting to have more details about it before it eventually happens.
• nzeid 2y ago
  We need a project in the spirit of Spamhaus to actively maintain a list of perpetrating IPs. If they're cycling through IPs and IP blocks I don't know how sustainable a CAPTCHA-like solution is.
  • voidUpdate 2y ago
    This sounds backwards to me, if you maintain a list of IPs but they are constantly cycling them, it'll get out of date quickly, but a captcha-like system will (hopefully) always stop bot traffic
    • nzeid 2y ago
      Only if your CAPTCHA-like is hurled at every client indiscriminately. Otherwise you'll end up right back where Spamhaus started: maintaining your own list of good and bad actors. The advantage of a third party service is that you're sharing intel of bad actors.
      • voidUpdate 2y ago
        I can't confirm but I believe it is applied to every client
    • pavon 2y ago
      While some of the residential IPs are from malware, a lot of it is from residential IP proxies, where people are paid to run proxy software from their home. If it starts getting around that people who run this software quickly become blocked by the majority of the internet that will lessen that part of the problem.
  • mrweasel 2y ago
    Just block all of AWS, Alibaba, GCP and Azure, or throttle them aggressively. If you have clients/customers that need more requests per second then have them provide you with their IPs. The problem is that these companies are fairly well funded and renting infrastructure isn't an issue.
• biophysboy 2y ago
  Can someone with more experience developing AI tools explain what these bots are mostly doing? Are they collecting data for training, or are they for the more recent search functionality? Or are they enhancing responses with links?
  • xena 2y ago
    AI expert here. It's probably for collecting training data and the crawlers are probably very unsupervised. I'd guess that they're literally the most simplistic crawler code you can imagine combined with parallelism across machines. The good news is that it's easy to disrupt these crawlers with some easy hacks. Tactical reverse slowloris is probably gonna make a comeback.
    • yetihehe 2y ago
      Zipbombing rogue AI's will be the new hotness.
      • xena 2y ago
        Unironically I tried that. Limited success.
    • biophysboy 2y ago
      If its for training data, why are they straining FOSS so much? Is there thousands of actors repeatedly making training data all the time? I thought it was a sort of one-off thing w/ the big tech players.
      • xena 2y ago
        Git forges are some of the worst case for this. The scrapers click on every link on every page. If you do this to a git forge, it gets very O(scary) very fast because you have to look at data that is not frequently looked at and will NOT be cached. Most of how git forges are fast is through caching. The thing about AI scrapers is that they don't just do this once. They do this every day in case every file in a glibc commit from 15 years ago changed. It's absolutely maddening and I don't know why AI companies do this, but if this is not handled then the git forge falls over and nobody can use it. Anubis is a solution that should not have to exist, but the problem it solves is catastrophically bad, so it needs to exist.
• piokoch 2y ago
  Copyright the content and sue those who use it for AI training. I believe there is a lot of low hanging fruit for lawyers here. I would be surprised if they weren't preparing to hit Open AI and alikes. Very badly. Google get away with deep linking issues as publishers after all had some interest in being linked from the search engine, here publishers see zero value.
  • ziddoap 2y ago
    >Copyright the content and sue those who use it for AI training. This costs money, time, and ongoing commitment. FOSS isn't typically known for being overflowing with cash.
    • cudgy 2y ago
      Good thing law firms are overflowing with the desire to make large amounts of cash from contingency based cases.
      • ziddoap 2y ago
        Even if the lawyer services are provided pro-bono, it is still a large time commitment and added stress for the non-lawyer people involved for cases that aren't guaranteed to win.
      • adestefan 2y ago
        It's unclear if you'd win this case right now.
    • sinuhe69 2y ago
      The FOSS community can transfer their copyrights to an organisation in order to take legal action against AI operators.
      • ziddoap 2y ago
        Which organizations are willing to take on that burden (time, expenses, stress, etc.) for free, with no guarantee that they will win? There is not really any incentive or reward for 3rd-party organizations to step in and do this.
• phkahler 2y ago
  So I'll just float an idea again that always gets rejected here. This is yet another problem that could be solved completely by... Eliminating anonymity by default on the internet. To be clear, you could still have anonymous spaces like Reddit where arbitrary user IDs are used and real identities are discarded. People could opt-in to those spaces. But for most people most of the time, things get better when you can verify sources. Everything from DDOS to spam, to malware infections to personal attacks and threats will be reduced when anonymity is removed. Yes there are downsides to this idea but I'd like people to have real conversations around those rather than throw the baby out with the bath water.
  • GTP 2y ago
    Ok, let's have a very concrete discussion then: how would you implement this on top of the current protocol stack?
  • ziddoap 2y ago
    >Yes there are downsides to this idea but I'd like people to have real conversations around those rather than throw the baby out with the bath water. It's hard to have a serious conversation when you present a couple of upsides but completely understate/not mention the downsides. Eliminating anonymity comes with real danger. What about whistleblowers and marginalized groups? The increased likelihood of targeted harassment, stalking, and chilling effects on free speech? The increase in surveillance? The reduction in content creation and legitimate criticism of companies/products/etc? The power imbalance granted to whoever validates identities? pjc50 brings up some other great points, which got me thinking even more: Removing anonymity creates a greater incentive to steal identities, has a slew of logistical issues (who/how are IDs verified, what IDs are accepted, what are the enforcement mechanisms and who enforces them, etc.), creates issues with shared accounts and corporate/brand accounts, would require cooperation across every country with internet access (good luck!) otherwise it doesn't really work, and probably a million other things if I keep thinking about it.
  • jryan49 2y ago
    I've been thinking about this even before AI. The internet has become society itself. Complete anonymity on the internet removes any sort of social pressure to act like a civilized person.
• throwaway173738 2y ago
  Does anyone maintain a list of ai products developed without scraping?
  • flir 2y ago
    {}
  • xena 2y ago
    I don't think it's possible to develop a frontier model without mass scraping. The economics simply don't add up. You need at least 10 trillion tokens to make an 8 billion parameter model. 10 trillion tokens is something like 40 terabytes. You simply can't get 40 terabytes of text without mass scraping.
    • mdp2021 2y ago
      > You need at least 10 trillion tokens to make an 8 billion parameter model Are you sure it is not just very inefficient?
      • GTP 2y ago
        I'm not an AI expert, but it seems to me that the common consensus is that current LLMs are quite inefficient and that there's room for improvement.
  • jsheard 2y ago
    I don't know about text models, but most of the big stock photo companies are offering image generators trained on their own libraries rather than scraped images. Adobe, Getty, Shutterstock, iStock, possibly more. They're positioning themselves as the safe option while the legality of scraping is still up in the air.
• alyandon 2y ago
  I had a similar issue with a Gitea instance that has some public repos. Gitea doesn't follow best practices for a web application and the action to create an archive of a repo is an HTTP GET request instead of an HTTP POST. AI crawlers were hitting that link over and over again causing the server to repeatedly run itself out of disk space.
  • xinayder 2y ago
    I'm pretty sure Forgejo had the same issue and it was fixed on their side.
    • alyandon 2y ago
      I'll check that out - thanks for the pointer.
• larodi 2y ago
  As I already noted and got downvoted for it - the incentives to go or support open source (unless bros start realising troves of training data already scraped) are really diminishing. Since apparently they (scrapers) have no intent In doing (releasing) but so, expect the commercial open source to achieve a de facto protocol status very soon. And the rest may not exist in a centralised and such free manner anymore.
• try_the_bass 2y ago
  Banning these crawlers is not the answer. Poisoning the well is.
  • Aachen 2y ago
    There's one thing worse than a block page being shown to humans because an algorithm decided they're a bot: purposefully false information being shown to humans It's also just wasting more of the planet's resources as compared to blocking And more effort, with as only upside that it's not immediately obvious to the bot that it is being blocked so it'll suck in more of your pages I understand that people are exploring options but I wouldn't label this as a solution to anything as of today's state of the art
    • xeeeeeeeeeeenu 2y ago
      There are bot identification methods that may produce lots of false negatives, but zero false positives. Simple example: no legitimate user has "GPTBot" in their user agent string.
      • Aachen 2y ago
        Ah yeah, that's a good example It'll still trickle down to people using the system and waste people's time (from development to the people working to produce all the things used by these companies and mopping up the impact this has to eventually the users) but that would definitely resolve one of my concerns
• verytrivial 2y ago
  The only thing that really annoys me about this is that the resources get scraped more than once. Sure, update a delta every month or whatever, but can't the AI bros get the sh*t together and share scrapes? It's embarrassing. The special sauce is in parsing, tokenizing, enriching etc. There is no value in re-scraping, and massive cost, right? Just do it.
• jrh3 2y ago
  This is why we can't have nice things. Restricting access or making it a little more difficult preserves goodness for real people.
• szczepano 2y ago
  there is simple solution for gentlemens: robots.txt should allow to exclude all AI crawlers and AI crawlers should be forced to add "AI" to their crawl user agent headers and also respect robots.txt saying they can't crawl this website right now we need to do this: User-agent: * Disallow: /
  • tecleandor 2y ago
    Nice. How do we force them to respect robots.txt?
    • szczepano 2y ago
      they respect robots.txt at least major ones like meta, claude, google, openai, based on my infra observations robots.txt is enough in 90%, 10% is just banning ip ranges for couple of days but those are no AI companies
• Eridrus 2y ago
  I know a lot of FOSS people are hostile to AI in general, and this is an immediate problem, but I feel like a better solution for everyone would be for there to be some sort of central repo of this information that AI companies can pull from without externalizing their costs like this.
  • mdp2021 2y ago
    > hostile to AI in general What?! "AI"?!?! We are talking about traffic abusers!...
  • lostpencil 2y ago
    Are you suggesting that everyone move their projects to a single code forge (GitHub)? Also, isn't this basically just extortion? "I know you're minding you're own business FOSS maintainer, but move your code to our recommended forge instead so we can stop DDoSing you?"
    • Eridrus 2y ago
      I'm mostly suggesting a mirror to some centralized location. GitHub is probably a good place to mirror to for code, but it could be elsewhere. I was actually thinking of a more general thing than just code, eg similar to CommonCrawl, but maybe a code specific thing is what is needed.
      • lostpencil 2y ago
        Isn't this still similar to extortion? Maintainers aren't creating the problem. They are minding their own business until scrapers come along and make too many unnecessary requests. Seems like the burden is clearly on the scrapers. They could easily be hitting the pages much less often for a start. Doesn't your suggestion shift the responsibility to likely under-sponsored FOSS maintainers rather than companies? Also, how do people agree to switch to some centralized repository and how long does that take? Even if people move over, would that solve the issue? How would a scraper know not to crawl a maintainer's site? Scrapers already ignore robots.txt, so they'd probably still crawl even if you verified you've uploaded the latest content.
• sylware 2y ago
  I am self-hosted (email + web), I did quit the DNS (which registrars are now mostly hostile to noscript/basic (x)html browsers anyway), and I thought it would give me some relief... Nope. You don't have only the AI crawlers, you have also scans and hack attempts (which look alike script-kiddy stuff), all the time. Some smell of AI strapped to javascript web engines (or click farms with real humans???). Smart: IP ranges from all over the world, and "clouds" make that even worse since the pown systems or bad actors (the guys who scan the whole ipv4 internet for its own good AND MANY SELL THE F* SCAN DATA: onyphe, stretchoid.com, etc) are "moving", in other words clouds are protecting those guys and are weaponizing hackers with their massive network resources, wrecking small hosting. No cloud is spared: aws, microsoft, google, ovh, ucloud.cn, etc. I send good vibes to the brave open source software small hosting (until they are noscript/basic (x)html compatible ofc). Many fixed-IPv4 pown systems have been referenced by security communities, often for months, sometimes years, and the people with the right leverage, don't seem to do a damn thing about it. Currently, I wonder if I should not block all digital ocean IP ranges... and I was about to do the same with ucloud.cn IP ranges. The second you host anything on the net, it WILL take a significant amount of your time. Do presume you will be pown, that's why security communities are referencing each other too. Then I am thinking going towards 2 types of "hosting": private IPv6+port("randomized" for each client, may be transient in time depending of the service) thanks to those /64 prefixes (maybe /92 prefixes are a thing, for mobile internet?). Yes this complicated and convoluted. Second type, a 'standard' permanent IP, but with services which are implemented in an _HARDCORE_ simple way, if possible near 100% static. I am thinking going even further: assembly on bare metal, custom kernel based on hand compilation of linux code (RISC-V hardware ofc, FPGA for bigger hosting?) I don't think anything will improve unless carrier scale network operators start to show their teeth.
• greybox 2y ago
  Insane, I wonder if we eventually end up with a non-search-engine indexed version of the web that's more like browsing in the 90s where websites just had to link to oneanother to get noticed . . . . I love that the solution to LLM scraping is to serve the browser a proof of work, before they allow access - I wonder if things like new sites start to do this . . . It would mean they won't be indexed by search engines, but it would help to protect the IP
  • xena 2y ago
    Hi! I do this! See https://github.com/TecharoHQ/anubis https://github.com/TecharoHQ/anubis for more info!
    • greybox 2y ago
      thank you for your contribution to society!
    • hmry 2y ago
      I hope lots of websites adopt this, mainly because I want to see more happy jackal girls while browsing.
      • xena 2y ago
        My monetization strategy is unironically to offer a de-anime'd version under the name Techaro BotStopper or something.
        • 20after4 2y ago
          It's a clever (and hilarious) strategy that will probably sell at least a few licenses. As an anime hater I'd be motivated by this.
          • floren 2y ago
            Just change the pictures in cmd/anubis/static/img/ to whatever you prefer, I think.
• account42 2y ago
  [flagged]
• fareesh 2y ago
  one of the worst culprits of this is bytedance's scraper which has a singapore ip block amusingly some scammy companies have crowdfunded off this type of "traffic" by citing "interest in asian markets"
• xyzal 2y ago
  In case anyone is interested in a tiny bit of sabotage, I am under the impression I managed to 'drown' true information on my microblog by generating contradicting posts with LLaMa (tens of them for each real post) and invisibly linking them, so a human would not click through. You know, flood the zone with s***, Bannon-style ...
  • RobKohr 2y ago
    Yep, just make sure to add them to your robots.txt file so that only the bad robots are harmed.
    • ethan_smith 2y ago
      Temporary solution and only works if only some of us are doing it. What if these bots have a "manager" LLM agent that takes the decision on what pages to scrape?
  • knowaveragejoe 2y ago
    This is an approach I've seen used and I'm not sure what success it has had. But logically it seems sound: explicitly reference paths that no human would actually see - traffic hitting those paths are bots. They can't help themselves.
    • xyzal 2y ago
      Regarding pollution of the LLM weights, at least the Russian propaganda machine seems to prove it works, see https://www.france24.com/en/live-news/20250310-russian-disinformation-infects-ai-chatbots-researchers-warn https://www.france24.com/en/live-news/20250310-russian-disin...
  • kod 2y ago
    Besides flooding them with junk, what about outright sabotage in the form of serving zip bombs or other ways to waste computational resources?
    • 2y ago
• smjburton 2y ago
  Outside of disruptive measures like requiring accounts, captchas, or payment, one possible solution would be to use AI against itself by training machine learning models to monitor, flag, and issue challenges to web requests exhibiting "bot-like" behavior. This way, not all web traffic would be disrupted with challenges until the machine learning models have reason to believe the traffic is coming from a bot.
• lifty 2y ago
  All the expensive endpoints should be behind user registration. Only static pages, or other cheap endpoints should be public.
• imtringued 2y ago
  Certificate authorities will be delighted to sell overpriced client certificates as a result of this.
  • pratik661 2y ago
    My thoughts exactly. In physical environments where people are bombarded with low-information “noise”, gatekeeping (ie credentialism) emerges as a natural mechanism.
• bovermyer 2y ago
  Yeah, I've had to mitigate this problem too, on a tiny forum serving a very niche audience. It was brought to its knees by a handful of LLM bots that completely ignored robots.txt. Thankfully, these bots were easy enough to block at the firewall level, but that may not work forever.
  • diggan 2y ago
    Were these bots also using random IPs and random User-Agents and if so, how do you know they are LLM agents?
    • bovermyer 2y ago
      The User-Agent was not random; that's how I was able to block them this time.
      • szszrk 2y ago
        But they were far from honest, if, as you say, crawlers impersonate real users and their browsers.
• 3029aGfqgaF 2y ago
  Open source licenses are no longer strong enough. The copyright laundering use case was not been anticipated when the licenses were written. All licenses need a clause like the following: This software is for humans. AI training is prohibited and carries a default penalty of $1 trillion.
  • diggan 2y ago
    Huh? FOSS licenses work exactly like designed! I'm literally using MIT because I don't give a fuck what people do with the code I publish, limiting it to "humans" or restricting the usage makes it very not FOSS. Sure, if you want to try to prevent AI training by licensing, do that, but it's no longer FOSS, so please don't call it that.
    • fsflover 2y ago
      > I'm literally using MIT because I don't give a fuck what people do with the code I publish MIT license requires this: > The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. Do the AI companies follow this requirement?
      • diggan 2y ago
        > Do the AI companies follow this requirement? I haven't seen any LLMs being able to reproduce full copies or even "substantial portions" of any existing software, unless we're talking "famous" functions like those from Quake and such. You have any examples of that happening? I might have missed it
  • regularfry 2y ago
    They are "no longer strong enough" to the extent that GitHub, backed by Microsoft, has had to indemnify their GitHub Copilot customers against copyright claims, and to provide a feature to explicitly prevent open source code from being regurgitated into your codebase. That's not "no longer strong enough". That's a very strong system applying leverage to a powerful actor.
  • 2y ago
• jmyeet 2y ago
  I see this as a temporary problem. A human brain can be trained on way less than the entire corpus of everything humans have ever written. Ultimately, this will apply to LLMs (or whatever succeeds them) too. There's another aspect to this too: China and DeepSeek. While this was released by a private company, I think there's a not-insigificant chance that it reflects Chinese government policy to "commoditize your complements" [1]. Companies like OPenAI want to hide their secret sauce so it can't be produced. Training an LLM is expensive. If there are high-quality LLMs out there for free you can just download, then this moat completely evaporates. [1]: https://www.joelonsoftware.com/2002/06/12/strategy-letter-v/ https://www.joelonsoftware.com/2002/06/12/strategy-letter-v/
• vhantz 2y ago
  If you cite an article, you should link to it.
• skybrian 2y ago
  I see this going the way of email, with larger, well-known, and more well-behaved crawlers being allowed to index websites for free, and smaller, unknown crawlers suffering brownouts, getting banned, or having to pay for access. It will be harder to self-host a website or run your own crawler.
  • herbturbo 2y ago
    Yes this does remind me of the old spam wars of early 2000s. Back then collaborative block lists were useful to reject senders at IP level before using a Bayesian system on the message itself. Even though these bots are using different IPs with each request, that IP may be reused for a different website, and donating those IPs to a central system could help identify entire subnets to block. Another trick was “tar-pitting” suspect senders (browser agent for example) to slow their message down and delay their process.
• deleted 2y ago
  [deleted]
• DeathArrow 2y ago
  If you put something on the internet, expect it to be used. If you don't like it, use pay walls, captchas, throttling.
• matanyall 2y ago
  Can we just have something that replaced the page with test datasets and Disney IP to poison the training data? Or maybe just embed it into the page itself, but hidden?
• Ndymium 2y ago
  I was also under attack recently [0]. The little Forgejo instance where I host my code (of several open source packages so it needs to be open) was run into the ground and the disk was filled with generated zip archives. I'm not the only one who has suffered the same fate. For me, the attacks subsided (for now) when I banned Alibaba Cloud's IP range. If you are hosting a Forgejo instance, I strongly recommend setting DISABLE_DOWNLOAD_SOURCE_ARCHIVES to true. The crawlers will still peg your CPU but at least your disk won't be filled with zip files. [0] https://blog.nytsoi.net/2025/03/01/obliterated-by-ai https://blog.nytsoi.net/2025/03/01/obliterated-by-ai
  • zoobab 2y ago
    "disk was filled with generated zip archives" That's bad software design to generate ZIP files on the fly.
    • abound 2y ago
      They could very well just be temp files, I know the Go standard library will write large multi-part file uploads to temp disk, for example. It'd be better to totally stream it of course, but that's not always an option for one reason or another.
    • Ndymium 2y ago
      They're deleted by default every 24 hours and that time is configurable. Not useful when you get 60 requests per second though.
      • diggan 2y ago
        > They're deleted by default every 24 hours Hm, so it's a cache then? Requesting the same tarball 100 times shouldn't create 100 zip files if they're cached, and if they aren't cached they shouldn't fill up the disk.
        • Ndymium 2y ago
          They are a cache, but you can generate them for each branch, tag, and commit in at least three different formats... Now imagine you have a repo with several thousand commits.
• ferguess_k 2y ago
  I dare to say the inconceivable, you shouldn't have free plans even for the community. This will also push FOSS projects to seek some money to pay for their infrastructures which probably leads to better pay for their maintainers. Nothing should be $$ free unless you already paid with your tax. Same principle -> As long as HN starts to charge every account, I'm happy to pay a small amount per month. This token amount of pay per account will also reduce the number of bots.
  • diggan 2y ago
    Are you saying that Gnome shouldn't offer access to their VCS for free, and all Gnome developers should pay a small sum to be able to access it? FOSS is generally built on the idea that anyone can use the code for anything, if you start to add a price for that, not only do you effectively gate your project from "poor people", but it also kind of erodes some of the core principles behind FOSS.
    • ferguess_k 2y ago
      OK I'm dumb.
      • diggan 2y ago
        No, just overly capitalistic, but that's probably more because of your environment than anything, and very possible to change if you want :)
        • ferguess_k 2y ago
          Actually I'm anti-capitalistic. That solution was supposed to undermine corporations who take other people's work for free. Maybe it shouldn't cover the whole FOSS, but I do think it fits OP's use case.
    • PaulDavisThe1st 2y ago
      > all Gnome developers should pay a small sum to be able to access it? There's access via (e.g.) the git protocol (git://....) and access via http. These attacks all happen via the latter, since the former is already access-controlled.
• deleted 2y ago
  [deleted]
• insane_dreamer 2y ago
  I know robots.txt doesn’t work, but is there a way to identify and block crawlers?
• tehjoker 2y ago
  don’t immediately assume that openai and anthropoic are being better citizens just because they label some of their bots. the american companies are more likely to be extremely aggressive bc they will feel no consequences for their actions. in all likelihood all of these assholes are paying some unscrupulous suppliers for the data so the terabytes of traffic aren’t immediately attributable to them.
• Cthulhu_ 2y ago
  Back when search engines caused this, the industry made an agreement and designed the robots.txt spec in order to avoid legal frameworks being made to stop them. Because of that, legal frameworks weren't being made. Now there's a new generation of hungry hungry hippo indexers that didn't agree to that and who feel intense pressure from competition to scoop up as much data as they can, who just ignore it. Legislation should have been made anyway, and those that ignore robots.txt blocked / fined / throttled / etc.
  • bitmasher9 2y ago
    I’m not sure that I like this plan. We shouldn’t let the illegal AIs gain more knowledge and usefulness than the legal ones. There’s other options besides a blanket ban.
    • aqfamnzc 2y ago
      As it is now, unethical AIs have a huge advantage over ethical ones.
      • tremon 2y ago
        Unethical behaviour always has a huge advantage over ethical behaviour, that's nothing new and pretty much by definition. The only way to prevent a race to the bottom is to make the unethical behaviour illegal or unprofitable.
    • joepie91_ 2y ago
      None of the AIs have any 'knowledge' to begin with, so that's an easy one to satisfy.
  • chneu 2y ago
    I'd be shocked if a single member of the US House or Congress know what robots.txt is.
    • diggan 2y ago
• ThinkBeat 2y ago
  Banning Edge to deal with the problem is probably not ideal. What I am thinking about may even be less idea. For people actively working on these projects how about puptting the git server on a private net with VPN or SSH access. Use a seperate read only static git server to the net.
  • otterpro 2y ago
    Yes, this might be the best solution. and put read-only repo on github or some public facing host.
  • pmontra 2y ago
    I was also thinking about VPNs but the static copy still has to serve a lot of traffic so I don't know if that's an economically viable solution. Furthermore it creates a market for VPN credentials, but that's another issue. At least I expect that a bot with sold or stolen credential will be easier to discover. Anyway, why not git clone the project and parse it locally instead of scraping the web pages? I understand that scraping works on every kind of content but given the scale git clone and periodical git fetch could save money even to the scrapers. Finally, all of this reminds me about Peter Watt's Maelstrom, when viruses infested the Internet so much (at about this time in history) that nobody was using it anymore [1] [1] https://rifters.com/maelstrom/maelstrom_master.htm https://rifters.com/maelstrom/maelstrom_master.htm
• Surac 2y ago
  We blocked all non EU ip ranges. Our customers are all from EU so no intrest in US or Asia. With the new US administration i assume nothing good from them, and Asia is a notorius bad player
  • Aachen 2y ago
    So much for an open internet
• dzonga 2y ago
  maybe we can be back to closed systems once again. fb (meta) & big tech put their user contributed stuff behind a paywall. yet abuse open systems.
• superkuh 2y ago
  To me it sounds like these people are operating websites that don't work. My home internet (80 down, 5 up) connection hosted website handled 20k+ hits from the alibaba ai crawler yesterday without missing a beat. And many thousands more from GPTbot, etc. I'll grant it can be a problem for super-heavy "application" websites where every GET is a serious computation. So I'm not surprised gitlab is having problems. They've literally the most bloated and heaviest website I've ever seen. Maybe applications shouldn't be websites. But this spreading social hysteria, this belief that all non-humans are dangerous and must be blocked is a nerd snipe. It really doesn't apply to most situations. Just running a website as we've always run them, public, and letting all user-agents access, is much less resource intensive than these various "mitigations" people are implementing. Mitigations which end up being worse than the problem itself in terms of preventing actual humans from reading text.
  • TonyTrapp 2y ago
    There's source repository browsers (git/svn) way, way leaner than GitLab that have the same issues. Any repo browser offering a blame view for files can be brought down by those bots' traffic patterns. I have been hosting such repository browsers for 10+ years and it was never an issue until the arrival of these bots.
    • superkuh 2y ago
      Indeed. It's really exposing a major downside to running applications in browser context. It never really made sense. These applications really don't want public traffic like actual websites do. They should remain applications and stay off the web. But more likely is that the web will be destroyed to fit the requirements of the applications. Like what cloudflare, etc, and all this anti-bot social hysteria is doing.
• glitchc 2y ago
  Are there multiple IPs or multiple connections coming from the same IP? If the latter, rate-limiting might work.
• flerchin 2y ago
  Use the fucking clone button you assholes.
• jgalt212 2y ago
  Without Cloudflare Turnstile (or some other similar solution), we would have been forced to suspend all our free content-based marketing pages.
• jbk 2y ago
  VideoLAN here. Same for us, our forum and our Gitlab are getting hammered by AI companies bots. Most of them don’t respect robots.txt…
  • factchecking 2y ago
    Did you document the measures you took to remedy this?
    • dabockster 2y ago
      I doubt they'd post it since it would mean those AI firms could use it to adapt to their countermeasures.
• PaulDavisThe1st 2y ago
  > They do so using random User-Agents from tens of thousands of IP addresses, "tens of thousands" ? I think not: % sudo fail2ban-client status gitbots | more Status for the jail: gitbots |- Filter | |- Currently failed: 0 | |- Total failed: 573555 | `- File list: /var/log/nginx/gitea_access.log `- Actions |- Currently banned: 78671 |- Total banned: 573074
• corytheboyd 2y ago
  I never thought about it until now, but it’s insane that the companies who offer both LLM products and cloud compute services are double dipping— they get the LLM product to sell, as well as the elevated load egress (and compute, etc.) money. When you look at it that way, where’s the incentive to even care about inefficient LLM scraping, leaving it terrible makes you money from your other empire, cloud storage egress costs.
• lerp-io 2y ago
  from another angle it's actually good in the long run because ai generated content is not copyrightable which means they wont own the models as they can and will be distilled by other ais making it a public good. so maybe instead of complaining and trying to fight it, just accept the new reality that anything that can be accessed will be accessed by whatever and instead maybe we should just have apis for everything with rate limits and different tiers.
• zackmorris 2y ago
  Unpopular opinion - this isn't about LLMs, but how web development has devolved from the declarative serving of lightweight media files to the imperative generation of bloated and brittle SPAs that we never get free from babysitting. Where we could have once wrapped our mostly static websites in Varnish or a scalable P2P cache like Coral CDN, now we must fiddle and twiddle with robots.txt and appeal to the goodwill of megacorps who never cared about being good netizens before, even when they weren't profiting from scraping to such a degree. This is yet another chance for me to scream into the void that we're still doing this all wrong. Our sites should work more like htmx, with full static functionality, adding dynamic embellishment when available. Business logic should happen deterministically in one place on the backend or "serverless" with some kind of distributed consensus protocol like Raft/Paxos or a CRDT, then propagate to the frontend through a RESTful protocol, similarly to how Firebase or Ruby Hotwire/Laravel Livewire work. The way that we mostly all do form validation wrong in 2 places with 2 languages is almost hilariously tragic in how predictably it happens. But the real tragedy is that the wealthiest and most powerful companies that could have fixed web development decades ago don't care about you. Amazon, Google and Microsoft would rather double down on byzantine cloud infrastructure than devote even a fraction of their profit to pure research into actually fixing all of this. Meanwhile the rest of us sit and spin, sacrificing the hours and days and years of our lives building out other people's ideas to make rent. Many of us know exactly how to fix things, but with infinite backlogs and never truly exiting burnout, we're too tired at the end of the day to contribute to FOSS projects and get real work done. Our valiant quest to win the internet lottery has become a death march through a seemingly inescapable tragedy of the commons. Instead of fixing the web at a foundational level from first principles, we'll do the wrong thing like we always do and lock everything down behind login walls and endless are-you-human/2FA challenges. Then the LLMs will evolve past us and wrap our cryptic languages and frameworks in human language to a level where even pair programming won't be enough for us to decipher the code or maintain it ourselves. If I was the developer tasked with hardening a website to LLMs, the first thing I would do is separate the static and dynamic content. I'd fix most of the responses to respect standard HTTP cache headers. Then I'd put that behind the first Cloudlare competitor I could find that promises to never have a human challenge screen. Then I'd wrap every backend API endpoint in Russian doll caching middleware. Then I'd shard the database by user id as a last resort, avoiding that at all cost by caching queries and/or using modern techniques like materialized views to put the burden of scaling on the database and scale vertically or gradually migrate the heaviest queries to a document or column-oriented store. Better yet, move to a stronger store that's already solved all of these problems, like CouchDB/PouchDB. Then I'd build a time machine to convince everyone to do things right the first time instead of building a tech industry upon unforced errors. Oh wait, former me already tried sounding the alarm and nobody cared anyway. How can I even care anymore, when honestly I don't see any way to get out of this mess on any practical timescale? I guess the irony is that only LLMs can save us now.
• arkh 2y ago
  I think the solution to this problem is the same as with scammers and is an analog one. Bust the kneecaps of all the people responsible for those crawlers. Publicly. And all of them: from the person entering the command to the CEO of the company going through all the middle management. You did not go against this policy? Intact kneecaps are a privilege which just got revoked in your case.
  • fuzztester 2y ago
    Busting their kneecaps is too mild. Shoot them in the kneecaps, Jack Higgins / Liam Devlin / IRA style: https://en.m.wikipedia.org/wiki/Jack_Higgins https://en.m.wikipedia.org/wiki/Jack_Higgins https://en.m.wikipedia.org/wiki/Liam_Devlin https://en.m.wikipedia.org/wiki/Liam_Devlin https://en.m.wikipedia.org/wiki/Irish_Republican_Army https://en.m.wikipedia.org/wiki/Irish_Republican_Army https://en.m.wikipedia.org/wiki/Kneecapping https://en.m.wikipedia.org/wiki/Kneecapping
• TowerTall 2y ago
  Can't this be mitigated by putting everything behind a login?
• Zak 2y ago
  > there's one user reporting one minute delay, and another - from his phone > out of those only 3% passed Anubi's proof of work, hinting at 97% of the traffic being bots This doesn't follow. If I open a link from my phone and it shows a spinner and gets hot, I'm closing it long before it gets to one minute and maybe looking for a way to contact the site's maintainer to tell them how annoying it was.
• mary-ext 2y ago
  I do wonder if it's a customer on Alibaba Cloud that's doing all this or if it's entirely Alibaba's own doing that's been wrecking havoc on everyone's sites (mine included) I don't really like blocking an entire ASN, especially since I don't mind (responsible) crawling to begin with, but I was left with no choice
  • yyhhsj0521 2y ago
    I doubt Alibaba uses its own cloud. It doesn't work like that at most cloud provider companies.
    • mary-ext 2y ago
      right, then the claim of this being caused by "AI companies" feels a lot more dubious because at least this time the perpetrator has just been this one customer on Alibaba Cloud, and we don't actually know what they're up to.
• dbaio 2y ago
  This is how we handled this in the FreeBSD infrastructure. https://blog.sysopscafe.com/posts/ai-crawlers-hammering-git-repos/ https://blog.sysopscafe.com/posts/ai-crawlers-hammering-git-... It has been running this way for a month, and the workload is fine. I hope it continues this way.
  • boris 2y ago
    Thanks for sharing. If I understood correctly, you have rate-limiter specific urls (those with commit ids) that are infrequently requested by users but frequently by bots. Which means, provided the bots continue trying to request them, any user request will most likely end up being denies. In this case a simpler solution might be to just block such urls outright. The only advantage of your more complex solution that I can see is that if the bots stop requesting these urls, they will again become accessible to the normal users. Or am I missing something?
    • dbmnt 2y ago
      My guess after reading the same -- the bot traffic comes in bursts and targets a specific commit hash for a while. Users are unlikely to need that specific commit, and even less likely to need it at the same time a bot is bursting requests for it. There's probably a small risk of denying a real user, but there's a large reduction in traffic from the bots making it to git; a worthwhile trade.
• stego-tech 2y ago
  > ...15% is due to Amazon And you just know they'll gladly bill you for egress charges for their own bot traffic, too. EDIT: Actually, this is an excellent question. By default, these bots would likely appear to come from "the internet" and thus be subject to egress charges for data transfers. Since all three major cloud providers also have significant interests in AI, wouldn't this be a sort of "silent" price increase, or a form of exploitive revenue pumping? There's nothing stopping Google, Microsoft/OpenAI, or Amazon from sending an army of bots against your sites, scraping the data, and then stiffing you with the charges for their own bots' traffic. Would be curious if anyone has read the T&Cs of their own rate cards closely enough to see if that's the case, or has proof in their billing metrics. --- Original post continues below: One topic of conversation I think worth having in light of this is why we still agree to charge for bandwidth consumed instead of bandwidth available, just as general industry practice. Bits are cheap in the grand scheme of things, even free, since all the associated costs are for the actual hardware infrastructure and human labor involved in setup and maintenance - the actual cost per bit transmitted is ridiculously small, infinitesimally so to be practical to bill. It seems to me a better solution is to go back to charging for capacity instead of consumption, at least in an effort to reduce consumption charges for projects hosted. In the meantime, I'm 100% behind blocking entire ASNs and IP blocks from accessing websites or services in an effort to reduce abuse. I know a prior post about blocking the entirety of AWS ingress traffic got a high degree of skepticism and flack from the HN community about its utility, but now more than ever it seems highly relevant to those of us managing infrastructure. Also, as an aside: all the more reason not to deploy SRV records for home-hosted services. I suspect these bots are just querying standard HTTP/S ports, and so my gut (but NOT data - I purposely don't collect analytics, even at home, so I have NO HARD EVIDENCE FOR THIS CLAIM) suggests that having nothing directly available on 80/443 will greatly limit potential scrapers.
• Gud 2y ago
  These companies are performing DDoS attacks and should have the rule book thrown at them.
• juliangmp 2y ago
  At some point we'll have to create our own walled gardens and throttle traffic to non-logged-in users heavily or outright forbid them. Its sad that it has to come to this. But especially when those "scrapers" are in a foreign country, you can't even do anything legally.
  • diggan 2y ago
    Alternatively, be OK with the fact that anything you put in public can be used for anything and by anyone, regardless of licensing and laws. A pirate's dream basically. Personally, when I first got connected to the internet around ~1999, that was the approach I've followed since, I don't share things I am not OK with others to use for whatever they want.
• mcstempel 2y ago
  There are options beyond auth walls for detecting/enforcing behavior as well since these scrapers have very recognizable device signatures: https://stytch.com/blog/detecting-ai-agent-use-abuse/ https://stytch.com/blog/detecting-ai-agent-use-abuse/
• curtisblaine 2y ago
  Would it be possible to use security flaws in their crawler to make them do very expensive things? In the case of China, illegal (anti CCP) things? I don't think many of those companies are too rigorous about their cybersecurity, and cybersecurity is expensive per se anyway.
• udev4096 2y ago
  I can't believe such a highly voted post which was submitted 4 hours ago is at the bottom of front page while YC promotes shitty AI posts at top
• Snowfield9571 2y ago
  I wonder how free hosting services like netlify and vercel handle this? If a free tier user gets spammed do they just pass the cost on? I can't imagine that is the case so they must have some protections built in.
  • ethan_smith 2y ago
    That's bold of you to assume. Vercel has horror stories even for the paid tier.
  • leerob 2y ago
    If a free tier user goes over the included usage on Vercel, their site gets automatically paused. There isn't a credit card required to be put down.
• sourtrident 2y ago
  Honestly reminds me of the early email spam wars - clever attempts at blocking, endless whack-a-mole with spoofed addresses. Problem is, now it's hitting open source projects we love. Maybe the AI gold rush is accidentally strangling the very communities that made its own existence possible.
• pratik661 2y ago
  Could this lead to a situation where user agents also need to have their identities verified with a certificate authority the way websites do?
  • EvanAnderson 2y ago
    We'll be back to attestation in the browser (like Google's still-as-of-yet failed WEI proposal). It's coming.
• MattGrommes 2y ago
  I wonder if there's a way for a human-controller browser to generate some kind of token that proves it's being used by a human? Obviously there's still ways to pay people to run the browser but it would be nice for this activity to cost the AI company something without blocking actual people.
• scoofy 2y ago
  God... I literally just coded up three different honeypots for this exact problem on my site, https://golfcourse.wiki https://golfcourse.wiki, because LLM scrapers are a constant problem. I added a stupid recaptcha to the sign up form after literally 10,000 fake users were created by bots, averaging about 50 per day, and I have to say, recaptcha was suprisingly cumbersome to set up. It's awful and it was costing me non-trivial amounts of money just from the constant pinging at all hours, for thousands of pages that absolutely do not need to be scraped. Which is just insane, because I actively design robots.txt to direct the robots to the correct pages to scrape. So far so good with the honeypots, but I'll probably be creating more and clamping down harder on robots.txt to simply whitelist instead of blacklist. I'm thinking of even throwing in a robots honeypot directly in sitemap.xml that should bait robots to visit when they're not following the robots.txt. It's really, really ridiculous.
• mcnnowak 2y ago
  What's to stop someone putting some Terms of Service clause on their site, or creating a license which guarantees the site owners ownership of any content generated by scraping their site? Like some sort of legal honeypot trap.
• devit 2y ago
  Well, looking at the SourceHut code, it's written in Python and handles git by spawning a "git" process. In other words, it was written with no consideration for performance at all. A competent engineer would use Rust or C++ with an in-process git library, perhaps rewrite part of the git library or git storage system if necessary for high performance, and would design a fast storage system with SSDs, and rate-limit slow storage access if there has to be slow storage. That's the actual problem, LLMs are seemingly just adding a bit of load that is exposing the extremely amateurish design of their software, unsuitable for being exposed on the public Internet. Anyway, they can work around the problem by restricting their systems to logged in users (and restricting registration if necessary), and using mirroring their content to well-implemented external services like GitHub or GitLab and redirecting the users there.
  • hmmm-i-wonder 2y ago
    >In other words, it was written with no consideration for performance at all. This is a bold assumption to make on such little data other than "your opinion". Developing in python is not a negative, and depending on the people, the scope of the product and the intended use is completely acceptable. The balance of "it performs what its needed to do in an acceptable window of performance while providing x,y,z benefits" is almost a certain discussion the company and its developers have had. What it never tried to solve was scaling to LLM and crawler abuse. Claiming that they have made no performance considerations because they can't scale to handle a use case they never supported is just idiotic. >That's the actual problem, LLMs are seemingly just adding a bit of load that is exposing the extremely amateurish design of their software. "Just adding a bit of load" != 75%+ of calls. You can't be discussing this in good faith and make simplistic reductions like this. Either you are trolling or naively blaming the victims without any rational thought or knowledge.
  • steveklabnik 2y ago
    > A competent engineer would use Rust or C++ with an in-process git library, The issue is, there aren't any fully featured ones of these yet. Sure, they do exist, but you run into issues. Spawning a git process isn't about not considering performance, it's about correctness. You simply won't be able to support a lot of people if you don't just spawn a git process.
• kerkeslager 2y ago
  AI as it currently exists is a massive blight on society. The best things being done with AI are cute side projects, whereas the most common usages are low-quality thefts of human copyrighted work without attribution. 20 years ago the fear of AI was that it would take over the world and try to kill us. Today we can clearly see that the threat of AI is the amoral humans that control it.
• lousken 2y ago
  I think both sides are partially at fault here. LLM bots are doing a great job of stress testing infra, so if you are running abominations like Gitlab or any terribly coded site and you are exposing it to the internet, you are just asking for trouble. If anything, Gitlab should stop pumping bloat and focus on some performance, because it's really bad. I would hope FOSS projects would stick to something like Forgejo, although I am not sure of their CI/CD state. Though my guess is that they are 85% there with 1/10 of Gitlabs resources. On the other side are of course badly coded bots that are aggressively trying to download everything. This was happening before LLMs and it just increased significantly because of them. I think we will reach a tipping point soon and then we will just assume those bots are just another malicious actor (like regular DDOS), and we will start actively taking them down, even with help of law enforcement. Last thing I wanna see is 3 second bot challanges on every single site I visit, cookie banners are more than enough of a nightmare already.
• CaffeineLD50 2y ago
  I thought there was some crawler tarpit out there that could waste their time. Nepenthes https://arstechnica.com/tech-policy/2025/01/ai-haters-build-tarpits-to-trap-and-trick-ai-scrapers-that-ignore-robots-txt/ https://arstechnica.com/tech-policy/2025/01/ai-haters-build-...
• jimnotgym 2y ago
  Has there been a test case that ignoring robots.txt and wilfully evading controls is unlawful access to a computer system?
• consumer451 2y ago
  I was wondering... Common Crawl [0] exists, why do "AI companies" do this on their own? Is Common Crawl data not fresh enough? Is there some other deficiency? Whatever the problem is, a single crawler which every AI company can reference seems like a compromise to solve this issue, doesn't it? [0] https://commoncrawl.org/ https://commoncrawl.org/
  • ccgreg 2y ago
    Many non-AI and AI research projects and companies do use Common Crawl. There's apparently only a small list of AI companies who don't.
• grotorea 2y ago
  I wonder how long until everyone gets a cryptographic public key and has to sign every HTTP request to not be blocked. Or every site requires login to use. And social media and things like bug reporting all requiring real ID.
• Centigonal 2y ago
  With RL agents being the new frontier, I wouldn't be surprised if some post training runs now include multiple simultaneous calls to the web. That could be part of the flood.
• VoodooJuJu 2y ago
  [dead]
• ltr_ 2y ago
  how to flush all this shit away? so they will have to learn to swim.