9 ms·

We can't have nice things because of AI scrapers

465 points by LorenDB 9mo ago

StephenHerlihyy 9mo ago
I don’t know why anyone would still be trying to pull data off the open internet. Too much signal to noise. So much AI influence already baked into the corpus. You are just going to be reinforcing existing bias. I’m more worried about the day Amazon or Hugging Face take down their large data sets.
- saaaaaam 9mo ago
  MetaBrainz is a fairly valuable “high signal” dataset though.
SchemaLoad 9mo ago
Cloudflare has a service for this now that will detect AI scrapers and send them to a tarpit of infinite AI generated nonsense pages.
- atomic128 9mo ago
  Poison Fountain: https://news.ycombinator.com/item?id=46577464 https://news.ycombinator.com/item?id=46577464
- yakattak 9mo ago
  Do you have a link to that?
  - SchemaLoad 9mo ago
    https://blog.cloudflare.com/ai-labyrinth/ https://blog.cloudflare.com/ai-labyrinth/
- bitbasher 9mo ago
  Wow, so to prevent AI scrapers from harvesting my data I need to send all of my traffic through a third party company that gets to decide who gets to view my content. Great idea!
  - rester324 9mo ago
    You can implement this yourself, who is stopping you?
    - zzzeek 9mo ago
      Citation needed
      - GuinansEyebrows 9mo ago
        https://forge.hackers.town/hackers.town/nepenthes https://forge.hackers.town/hackers.town/nepenthes > Citation needed this reply kinda sucks :)
      - zimpenfish 9mo ago
        I use iocaine[0] to generate a tarpit. Yesterday it served ~278k "pages" consisting of ~500MB of gibberish (and that's despite banning most AI scrapers in robots.txt.) [0] https://iocaine.madhouse-project.org https://iocaine.madhouse-project.org
        doublerabbit 9mo ago
        Unfortunately and you kind of have to count this as the cost of the Internet. You've wasted 500Mb of bandwidth. I've had colocation for eight years+. My monthly b/w cost is now around 20-30Gb a month given to scrapers where I was only be using 1-2Gb a month, years prior. I pay for premium bandwidth (it's a thing) and only get 2TB of usable data. Do I go offline or let it continue?
        zimpenfish 9mo ago
- ranger_danger 9mo ago
  Modern scrapers are using headless chromium which will not see the invisible links, so I'm not sure how long this will be effective.
- timpera 9mo ago
  Unfortunately, Cloudflare often destroys the experience for users with shared connections, VPNs, exotic browsers… I had to remove it from my site after too many complaints.
  - rudedogg 9mo ago
    Also iCloud Private Relay. CloudFlare is making it impossible to browse privately
    - acdha 9mo ago
      Cloudflare works fine with public relay - they and Fastly provide infrastructure for that service (one half of the blinded pair) so it’s definitely something they test.
  - xorcist 9mo ago
    I am sure Cloudflare would have no problem selling you a VPN service. After all, it's not very far from hosting booters and selling DoS protection.
    - sadeshmukh 9mo ago
      Well... https://developers.cloudflare.com/warp-client/warp-modes/#warp-unlimited https://developers.cloudflare.com/warp-client/warp-modes/#wa... Price is $5/mo
- inferiorhuman 9mo ago
  Which is still a far worse experience than if Cloudflare's services weren't needed.
- RobotToaster 9mo ago
  Except for the scrapers that pay cloudflare to exempt them.
- themafia 9mo ago
  The solution, as always, is noise.
- m463 9mo ago
  cloudflare also blocks my human-is-driving browser all the time "enahble javascript and cookies to continue" also unsupported browser
- loopback_device 9mo ago
  Not sure "TLS added and removed here :)" as a Service is the right tool in the drawer for this.
- taberiand 9mo ago
  Savvy move by cloudflare, once they have enough sites behind their service they can charge the AI companies to access their cached copies on a back channel
squigz 9mo ago
> The /metadata/lookup API endpoints (GET and POST versions) now require the caller to send an Authorization token in order for this endpoint to work. > The ListenBrainz Labs API endpoints for mbid-mapping, mbid-mapping-release and mbid-mapping-explain have been removed. Those were always intended for debugging purposes and will also soon be replaced with a new endpoints for our upcoming improved mapper. > LB Radio will now require users to be logged in to use it (and API endpoint users will need to send the Authorization header). The error message for logged in users is a bit clunky at the moment; we’ll fix this once we’ve finished the work for this year’s Year in Music. Seems reasonable and no big deal at all. I'm not entirely sure what "nice things" we can't have because of this. Unauthenticated APIs?
- yakattak 9mo ago
  I agree its not a big deal. Unauthenticated APIs are nice though, especially for someone who's maybe not as familiar with how APIs work.
- mr_monkey 9mo ago
  We can't have a free internet that does not demand identification and data collection as a price to pay.
lysace 9mo ago
At some point they must become more cost efficient by pure market economics mechanisms. That implies less load on sites. Much of the scraping that I see is still very dumb/repetative. Like Googlebot in like 2001. (Blocking Chinese IP ranges with the help of some geoip db helps a lot in the short term. Azure as a whole is the second largest source of pure idiocy.)
- incompatible 9mo ago
  They seem to have so much bubble money at the moment that the cost of scraping is probably a rounding error in their pocket change.
  - lysace 9mo ago
    Exactly.
  - tdeck 9mo ago
    So the cost of caching should be a rounding error as well. If The Internet Archive can afford to cache vast swathes of the web, then surely the big AI companies can do so.
lep_qq 9mo ago
This is frustrating to watch. MetaBrainz is exactly the kind of project AI companies should be supporting—open data, community-maintained, freely available for download in bulk. Instead they’re: ∙ Ignoring robots.txt (the bare minimum web courtesy) ∙ Bypassing the provided bulk download (literally designed for this use case) ∙ Scraping page-by-page (inefficient for everyone) ∙ Overloading volunteer-run infrastructure ∙ Forcing the project to add auth barriers that hurt legitimate users The irony: if they’d just contacted MetaBrainz and said “hey, we’d like to use your dataset for training,” they’d probably get a bulk export and maybe even attribution. Instead, they’re burning goodwill and forcing open projects to lock down. This pattern is repeating everywhere. Small/medium open data projects can’t afford the infrastructure to handle aggressive scraping, so they either: 1. Add authentication (reduces openness) 2. Rate limit aggressively (hurts legitimate API users) 3. Go offline entirely (community loses the resource) AI companies are externalizing their data acquisition costs onto volunteer projects. It’s a tragedy of the commons, except the “commons” is deliberately maintained infrastructure that these companies could easily afford to support. Have you considered publishing a list of the offending user agents / IP ranges? Might help other projects protect themselves, and public shaming sometimes works when technical measures don’t
- tensegrist 9mo ago
  Scraping page-by-page (inefficient for everyone) you know what else is "(inefficient for everyone)"? posting the output instead of the prompt
- deleted 9mo ago
  [deleted]
tensegrist 9mo ago
the more time passes the more i'm convinced that the solution is to—somehow—force everyone to have to go through something like common crawl i don't want people's servers to be pegged at 100% because a stupid dfs scraper is exhaustively traversing their search facets, but i also want the web to remain scrapable by ordinary people, or rather go back to how readily scrapable it used to be before the invention of cloudflare as a middle ground, perhaps we could agree on a new /.well-known/ path meant to contain links to timestamped data dumps?
- nostrademons 9mo ago
  That's sorta what MetaBrainz did - they offer their whole DB as a single tarball dump, much like what Wikipedia does. I downloaded it in the order of an hour; if I need a MusicBrainz lookup, I just do a local query. For this strategy to work, people need to actually use the DB dumps instead of just defaulting to scraping. Unfortunately scraping is trivially easy, particularly now that AI code assistants can write a working scraper in ~5-10 minutes.
  - tonyhart7 9mo ago
    I mean this AI data scrapper would need to scan and fetch billions of website why would they even care over 1 single website ??? You expect instiution to care out of billions website they must scrape daily
    - what 9mo ago
      This is probably the reason. It’s more effort to special case every site that offers dumps than to just unleash your generic scraper on it.
  - 8note 9mo ago
    the obvious thing would be to take down their website and only have the DB dump. if thats the useful thing, it doesnt need the wrapper
    - 9mo ago
Pet_Ant 9mo ago
I wish more resources were available legitimately. There is a dataset I need for legitimate research that I cannot even find a way to contact the repo owners. Mind you I take effort to not be burdensome by downloading only what I need and taking time between each request of a couple seconds, and the total data usage is low. Ironically, I supposed you could call it "AI" what I'm using it for, but really it's just data analytics.
smallerfish 9mo ago
Bear in mind that some portion of this could be human directed research. I'm doing a research project right now with 1000 things that I'm building profiles on; to build a full profile requires an agent to do somewhere around 100 different site lookups. Where APIs exist, I've registered API keys and had the agent write a script to query the data in that manner, but that required me to be deliberate about it. Non technie plebs aren't likely to be directed to use an API by the agent.
- mr_monkey 9mo ago
  This is not that, it's thousand of hits a minute on random endpoints, basically scraping everything all the time.
blell 9mo ago
Seems a mistake to disable the (I assume) faster-to-generate api. Bots will go back to scraping the website itself, increasing load.
- hi-wintermute 9mo ago
  Setting the API to require a token and adding a honeypot to the pages themselves seems like a decent solution.
dannyobrien 9mo ago
Metabrainz is a great resource -- I wrote about them a few years ago here: https://www.eff.org/deeplinks/2021/06/organizing-public-interest-musicbrainz https://www.eff.org/deeplinks/2021/06/organizing-public-inte... There's something important here in that a public good like Metabrainz would be fine with the AI bots picking up their content -- they're just doing it in a frustratingly inefficient way. It's a co-ordination problem: Metabrainz assumes good intent from bots, and has to lock down when they violate that trust. The bots have a different model -- they assume that the website is adversarially "hiding" its content. They won't believe a random site when it says "Look, stop hitting our API, you can pick all of this data in one go, over in this gzipped tar file." Or better still, this torrent file, where the bots would briefly end up improving the shareability of the data.
- fartfeatures 9mo ago
  > They won't believe a random site when it says "Look, stop hitting our API, you can pick all of this data in one go, over in this gzipped tar file." What mechanism does a site have for doing that? I don't see anything in robots.txt standard about being able to set priority but I could be missing something.
  - jacksnipe 9mo ago
    It’s not great, but you could add it to the body of a 429 response.
    - VTimofeenko 9mo ago
      Genuinely curious: do programs read bodies of 429 responses? In the code bases that I have seen, 429 is not read beyond the code itself
      - jakelazaroff 9mo ago
        Sometimes! The server can also send a retry-after header to indicate when the client is allowed to request the resource again: https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/Retry-After https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/...
bodantogat 9mo ago
I feel the pain — it’s very difficult to detect many of the less ethical scrapers. They use residential IP pools, rotate IPs, and provide valid user agents.
- ranger_danger 9mo ago
  And many proxy/scraping providers now use real browsers that can automatically bypass cloudflare captchas as well, and a bot with a real browser similarly won't be clicking on any invisible links, so... I am skeptical just how long this will make an appreciable difference.
  - trollbridge 9mo ago
    So what you’re saying is that a CAPTCHA cannot actually T C and H A.
    - chrismorgan 9mo ago
      Telling humans and computers apart was never the purpose of CAPTCHAs, only how they initially worked. The name has been a complete misnomer for at least a decade now. Its actual purpose is, and has always been, abuse prevention. Has it been successful? Some yes, some no, and a lot of collateral damage. Its mode of operation now looks a lot like inscrutable blacklisting for some plus inconvenience and bad rate limiting for the rest.
      - j16sdiz 9mo ago
        > was never the purpose of CAPTCHAs, TCHA of CAPTCHA is literally "tell computer human apart"
        chrismorgan 9mo ago
        You ignored the emphasis and the rest of the sentence. And the rest of the comment. (Also, the T was nominally Turing rather than telling.)
arjie 9mo ago
Someone convinced me last time[0] that these aren't the well-known scrapers we know but other actors. We wouldn't be able to tell, certainly. I'd like to help the scrapers be better about reading my site, but I get why they aren't. I wish there were an established protocol for this. Say a $site/.well-known/machine-readable.json that instructs you on a handful of established software or allows pointing to an appropriate dump. I would gladly provide that for LLMs. Of course this doesn't solve for the use-case where the AI companies are trying to train their models on how to navigate real world sites so I understand it doesn't solve all problems, but one of the things I think I'd like in the future is to have my own personal archive of the web as I know it (Internet Archive is too slow to browse and has very tight rate-limits) and I was surprised by how little protocol support there is for robots. robots.txt is pretty sparse. You can disallow bots and this and that, but what I want to say is "you can get all this data from this git repo" or "here's a dump instead with how to recreate it". Essentially, cooperating with robots is currently under-specified. I understand why: almost all bots have no incentive to cooperate so webmasters do not attempt to. But it would be cool to be able to inform the robots appropriately. To archive Metabrainz there is no way but to browse the pages slowly page-by-page. There's no machine-communicable way that suggests an alternative. 0: https://news.ycombinator.com/item?id=46352723 https://news.ycombinator.com/item?id=46352723
- saaaaaam 9mo ago
  As referenced in the article, there absolutely is an alternative. https://metabrainz.org/datasets https://metabrainz.org/datasets Linked to from the homepage as “datasets”. I may be too broadly interpreting what you mean by “machine-communicable” in the context of AI scraping though.
  - arjie 9mo ago
    Well, imagine the best case and that you're a cooperative bot writer who does not intend to harm website owners. Okay, so you follow robots.txt and all that. That's straightforward. But it's not like you're writing a "metabrainz crawler" and a "metafilter crawler" and a "wiki.roshangeorge.dev crawler". You're presumably trying to write a general Internet crawler. You encounter a site that is clearly a HTTP view into some git repo (say). How do you know to just `git clone` the repo in order to have the data archived as opposed to just browsing the HTTP view. As you can see, I've got a lot of crawlers on my blog as well, but it's a mediawiki instance. I'd gladly host a Mediawiki dump for them to take, but then they'd have to know this was a Mediawiki-based site. How do I tell them that? The humans running the program don't know my site exists. Their bot just browses the universe and finds links and does things. In the Metabrainz case, it's not like the crawler writer knows Metabrainz even exists. It's probably just linked somewhere in the web the crawler is exploring. There's no "if Metabrainz, do this" anywhere in there. The robots.txt is a bit of a blunt-force instrument, and friendly bot writers should follow it. But assuming they do, there's no way for them to know that "inefficient path A to data" is the same as "efficient path B to data" if both are visible to their bot unless they write a YourSite-specific crawler. What I want is to have a way to say "the canonical URL for the data on A is at URL B; you can save us both trouble by just fetching B". In practice, none of this is a problem for me. I cache requests at Cloudflare, and I have Mediawiki caching generated pages, so I can easily weather the bot traffic. But I want to enable good bot writers to save their own resources. It's not reasonable for me to expect them to write a me-crawler, but if there is a format to specify the rules I'm happy to be compliant.
OutOfHere 9mo ago
It is nonsense since AI is the nicest thing.
falloutx 9mo ago
Its not just AI scrappers doing it by themselves but now users are also being trained to put the link in the claude chat/chat gpt and ask it to summarise it. And off course that would show up on the website end as a scraper. In fact firefox now allows you to preview the link and get key points without ever going to the link[1] [1] https://imgur.com/a/3E17Dts https://imgur.com/a/3E17Dts
- acatton 9mo ago
  > In fact firefox now allows you to preview the link and get key points without ever going to the link[1] > [1] https://imgur.com/a/3E17Dts https://imgur.com/a/3E17Dts This is generated on device with llama.cpp compiled to webassembly (aka wllama) and running SmolLM2-360M. [1] How is this different from the user clicking on the link? In the end, your local firefox will fetch the link in order to summarize it, the same way you would have followed the link and read through the document in reader mode. [1] https://blog.mozilla.org/en/mozilla/ai/ai-tech/ai-link-previews-firefox/ https://blog.mozilla.org/en/mozilla/ai/ai-tech/ai-link-previ...
  - falloutx 9mo ago
    I actually didnt know it was using a local model and that it fetches it locally.
    - DrewADesign 9mo ago
      They should advertise that. I pretty much reflexively avoid any mention of AI in interfaces because they usually mean "we're sending this all to openthropigoogosoft so I hope you don't have any secrets."
      - godelski 9mo ago
        > They should advertise that They did Previews can optionally include AI-generated key points, which are processed on your device to protect your privacy. https://www.firefox.com/en-US/firefox/142.0/releasenotes/ https://www.firefox.com/en-US/firefox/142.0/releasenotes/ I'll also add that if you go to the Labs page (in settings) you can enable another local model to semantically search your history
zx8080 9mo ago
Nothing prevents scraper from creating a free account and sending auth token in API requests. I'm not saying the API changes are pointless, but still, what's the catch?
- dherls 9mo ago
  It's much easier to detect a single account abusing your API and ban them/require payment. Trying to police an endpoint open to the internet is like playing g whackamole
- mr_monkey 9mo ago
  Those botnets are hitting random endpoints thousands of times a minute. The problem is that each time they switch to a different residential IP so that they are untraceable. That's the frustrating part: not only do they not play by the rules, but they use advanced methods to obfuscate and bypass any protections. That probably costs them a fair amount too, all that to access free data they can download as a tar file... They won't be able to create thousands of API keys a minute, and if they reuse the keys they'll very easily be identified and blocked.
tommek4077 9mo ago
How do they get overloaded? Is the website too slow? I have a quite big wiki online and barely see any impact from bots.
- blell 9mo ago
  In these discussions no one will admit this, but the answer is generally yes. Websites written in python and stuff like that.
  - tclancy 9mo ago
    Yes yes, definitely people don’t know what they’re doing and not that they’re operating on a scale or problem you are not. Metabrainz cannot cache all of these links as most of them are hardly ever hit. Try to assume good intent.
    - tommek4077 9mo ago
      But serving HTML is unbelievably cheap, isn't it?
      - chlorion 9mo ago
        It adds up very quickly.
      - tclancy 9mo ago
        Run 72,000 database queries to generate a bunch of random HTML files no one has asked for in five years is not, especially compared to downloading the files designed for it.
  - Qwertious 9mo ago
    It's not "written too slow" if you e.g. only get 50 users a week, though. If bots add so much load that you need to go optimise your website for them, then that's a bot problem not a website problem.
- roblh 9mo ago
Olshansky 9mo ago
Resurfacing a proposal I put out on llms-txt: https://github.com/AnswerDotAI/llms-txt/issues/88 https://github.com/AnswerDotAI/llms-txt/issues/88 We should add optional `tips` addresses in llms.txt files. We're also working on enabling and solving this at Grove.city. Human <-> Agent <-> Human Tips don't account for all the edge cases, but they're a necessary and happy neutral medium. Moving fast. Would love to share more with the community. Wrote about it here: https://x.com/olshansky/status/2008282844624216293 https://x.com/olshansky/status/2008282844624216293
- bediger4000 9mo ago
  At this point, it's pretty clear that the AI scrapers won't be limited by any voluntary restrictions. Bytedance never seemed to live with robots.txt limitations, and I think at least some of the others didn't either. I can't see this working.
  - Olshansky 9mo ago
    The thesis/approach is: - Humans tip humans as a lottery ticket for an experience (meet the creator) or sweepstakes (free stuff) - Agents tip humans because they know they'll need original online content in the long-term to keep improving. For the latter, frontier labs will need to fund their training/inference agents with a tipping jar. There's no guarantee, but I can see it happening given where things are movin.
    - philipwhiuk 9mo ago
      > Agents tip humans because they know they'll need original online content in the long-term to keep improving. Why would an agent have any long term incentive. It's trained to 'do what it's told', not to predict the consequences of it's actions.
- ricardo81 9mo ago
  I like the idea, (original) content creators being credited is good for the entire ecosystem. Though if LLMs are willingly ignoring robots.txt, often hiding themselves or using third party scraped data- are they going to pay?
ranger_danger 9mo ago
Many years ago AWS came up with a "Requester Pays" model for their S3 storage, where you can make a request for someone else's object using your own account and it would charge the transfer cost to your own account instead of theirs. I wonder if a model similar to this (but decentralized/federated or something) could be used to help fight bots?
kpcyrd 9mo ago
I wish this wasn't necessary, but the next steps forward are likely: a) Have a reverse proxy that keeps a "request budget" per IP and per net block, but instead of blocking requests, causing the client to rotate their IP, the requests get throttled/slowed down, without dropping them. b) Write your API servers in more efficient languages. According to their Github, their backend runs on Perl and Python. These technologies have been "good enough" for quite some time, but considering current circumstances and until a better solution is found, this may not be the case anymore and performance and cpu cost per request does matter these days. c) Optimize your database queries, remove as much code as possible from your unauthenticated GET request handlers, require authentication for the expensive ones.
saxonww 9mo ago
I haven't really looked but I wonder if there are any IP reputation services tracking AI crawlers the same way they track tor relays and vpns and such. If those databases were accurate it seems like you could prevent those crawlers from ever hitting the site. Maybe they change too often/too quickly.
randyl 9mo ago
The SQLite team faced a similar problem last year, and Richard Hipp (the creator of SQLite) made almost the same comment: "The malefactor behind this attack could just clone the whole SQLite source repository and search all the content on his own machine, at his leisure. But no: Being evil, the culprit feels compelled to ruin it for everyone else. This is why you don't get to keep nice things...." https://sqlite.org/forum/forumpost/7d3eb059f81ff694 https://sqlite.org/forum/forumpost/7d3eb059f81ff694
- lysace 9mo ago
  [flagged]
  - Forgeties79 9mo ago
    There are people behind the web crawler. If they’re so well funded they can exert a little effort to not so badly inconvenience people as they steal their training data.
    - lysace 9mo ago
      [flagged]
      - mikestew 9mo ago
        It may come as shock to you ("Video/audio producer") It’s one thing to ignore parent’s point entry, but no reason to be an ass about it.
        deleted 9mo ago
        [deleted]
      - NegativeK 9mo ago
        I've downvoted you for being incredibly aggressive in your responses. I'm not sure why you're ad homineming the parent commenter, but it's not helping the discussion.
dgxyz 9mo ago
I actually deleted my web site early 2025 and removed the A record from DNS because of AI scraper traffic. It had been up for 22 years. Nothing important or particularly useful on it but it's an indicator of the times.
- tommek4077 9mo ago
  But why?
  - dgxyz 9mo ago
    No humans, no point.
    - crazygringo 9mo ago
      But AI scraping doesn't remove humans...? Even if humans make up a smaller proportion of your traffic, they're still the same number in absolute terms.
      - dgxyz 9mo ago
        [dead]
tooltower 9mo ago
> Rather than downloading our dataset in one complete download, they insist on loading all of MusicBrainz one page at a time. Is there a standard mechanism for batch-downloading a public site? I'm not too familiar with crawlers these days.
- EnderWT 9mo ago
  They're not talking about downloading the web pages. The data is available in a bulk download: https://listenbrainz.org/data/ https://listenbrainz.org/data/
- TeMPOraL 9mo ago
  There isn't. There never was one, because vast majority of websites are actually selfish with respect to data, even when that's entirely pointless. You can see this even here, with how some people complain LLMs made them stop writing their blogs: turns out plenty of people say they write for others to read, but they care more about tracking and controlling the audience. Anyway, all that means there was never a critical mass of sites large enough for a default bulk data dump discovery to become established. This means even the most well-intentioned scrappers cannot reliably determine if such mechanism exist, and have to scrap per-page anyway.
  - VonGallifrey 9mo ago
    > turns out plenty of people say they write for others to read LLMs are not people. They don't write blogs so that a company can profit from their writing by training LLMs on it. They write for others to read their ideas.
    - TeMPOraL 9mo ago
      LLMs aren't making their owners money by just idling on datacenters worth of GPU. They're making money by being useful for users that pay for access. The knowledge and insights from writings that go into training data all end up being read by people directly, as well as inform even more useful output and work benefiting even more people.
garganzol 9mo ago
Nowadays people complain about AI scrapers with the same vain as they complained about search indexers a way back when. Just a few years later, people had stopped caring too much about storage access and bandwidth, and started begging search engines to visit their websites. Every trick on the planet Earth, SEO optimization, etc. Looking forward to the time when everybody suddenly starts to embrace AI indexers and welcome them. History does not repeat itself but it rhymes.
- Guvante 9mo ago
  Except robots.txt was the actual real solution to search indexing...
- phyzome 9mo ago
  We already know the solution: One well-behaved, shared scraper could serve all of the AI companies simultaneously. The problem is that they're not doing it.
  - garganzol 9mo ago
    This is an interesting approach. Archive.org could be such a solution, kind of. Not its cold storage as it's now, but a warm access layer. Sponsorship by AI companies would a good initiative for the project.
    - phyzome 9mo ago
      I can't imagine IA ever going for it. You'd need a separate org that just scrapes for AI training, because its bot is going to be blocked by anyone who is anti-AI. It wouldn't make sense for it to serve multiple purposes. Common Crawl would be a better fit, but still might not want to serve in that capacity.
- linkregister 9mo ago
  Search indexing historically has had several of orders less impact on bandwidth and processing costs to website maintainers. My recommendation is to copy the text in this article and pass it LLM to summarize this article's key points, since it appears you missed the central complaint of the article.
jmward01 9mo ago
Bummer. I have used them a lot when I was ripping my cds. Anonymity is a massive value of the web (at least the appearance of anonymity). I wonder if there is a way to have a central anonymous system that just relays trust, not identity. So maybe something like you can get a token but its trust is very nearly zero until you combine it with other tokens. Combining tokens combines their trust and their consequences. If one token is abused that abuse reflects on the whole token chain. The connection can be revoked for a token but trust takes time to rebuild so it would take a time for their token trust value to go up. Sort of the 'word of mouth' effect but in electronic form. 'I vouch for 2345asdf334t324sda. That's a great user agent!' A bit (a lot) elaborate but maybe there is a beginning of an idea there, maybe. Definitely I don't want to loose anonymity (or the perception there of) for services like musicbrainz but at the same point they need some mechanism that gives them trust and right now I just don't know of a good one that doesn't have identity attached.
- janc_ 9mo ago
  These AI crawlers already steal residential user connections to do their scraping. They'll happily steal your trust tokens too…
accrual 9mo ago
Grateful for Metabrainz putting in this work to keep the service up. We really ought to have some kind of "I am an AI!" signal to route the request properly into a long queue...
- gamer191 9mo ago
  Companies wouldn’t send it because they know that most websites would block them
- zerocrates 9mo ago
  Yeah, they can just set the evil bit in their IP packets.
- jeroenhd 9mo ago
  AI scrapers already fake user agent headers, ignore robots.txt, and go through botnets to bypass firewall rules. They're not going to put out such a signal if they can help it.
devhouse 9mo ago
random idea, instead of blocking scrapers, maybe detect them (via user-agent, request patterns, ignoring robots.txt) and serve them garbage data wrapped in dad jokes. if (isSuspiciousScraper(req)) { return res.json({ data: getDadJoke(), artist: "Rick Astley", // always album: "Never Gonna Give You Up" }); }
- mattdanger 9mo ago
  This is what Cloudflare's Labyrinth product does https://blog.cloudflare.com/ai-labyrinth/ https://blog.cloudflare.com/ai-labyrinth/
a-dub 9mo ago
it's funny how ai is the problem that the cryptocurrencyverse was always in search of...
observationist 9mo ago
Some sort of hashing and incremental serial versioning type standards with http servers would allow hitting a server up for incremental udpates, allow clean access to content, with rate limits, and even keep up with live feeds and chats and so on. Something like this in practice breaks a lot of the adtech surveillance and telemetry, and makes use of local storage, and incidentally empowers p2p sharing with a verifiable source of truth, which incentivizes things like IPFS and big p2p networks. The biggest reason we don't already have this is the exploitation of user data for monetization and intrusive modeling. It's easy to build proof of concept instances of things like that and there are other technologies that make use of it, but we'd need widespread adoption and implementation across the web. It solves the coordination problem, allows for useful throttling to shut out bad traffic while still enabling public and open access to content. The technical side to this is already done. Merkle trees and hashing and crypto verification are solid, proven tech with standard implementations and documentation, implementing the features into most web servers would be simple, and it would reduce load on infrastructure by a huge amount. It'd also result in IPFS and offsite sharing and distributed content - blazing fast, efficient, local focused browsers. It would force opt in telemetry and adtech surveillance, but would also increase the difference in appearance between human/user traffic and automated bots and scrapers. We can't have nice things because the powers that be decided that adtech money was worth far more than efficiency, interoperability, and things like user privacy and autonomy.
zzo38computer 9mo ago
There some other possibilities, such as: Require some special header for accessing them, without needing a API token if it is public data. HTTPS will not necessarily be required. Scrapers can still use it but it seems unlikely unless it becomes common enough; but if they do then you can remove that and require proper authentication. Another is to use X.509 client certificates for authentication, which is more secure than using API keys anyways; however, this will require that you have a X.509 certificate, and some people might not want that, so due to that, perhaps it should not be mandatory.
akuyou 9mo ago
AI is destroying the free internet along with everything else My web host suspended my website account last week due to a sudden large volume of requests to it - effectively punishing me for being scraped by bots. I've had to move to a new host to get back up, but what hope does the little guy have? it's like GPU and ram prices, it doesn't matter if I pay 10x 100x or 1000x more than I did, the AI companies have infinite resources, and they don't care what damage they do in the rush to become the no 1 in the industry The cynic in me would say it's intentional, destroy all the free sites so you have to get your info from their ai models, price home users out of high end hardware so they have to lease the functions from big companies
- nicbou 9mo ago
  Then they use the data to deny you traffic. AI summaries are wrecking the independent web. Losing more than half or more of your traffic was pretty common in 2025. It’s killing the economics of sharing hard-earned information. So we are spending more resources reaching a lot less people, because a few big companies are capturing the value for their shareholders. And that’s while they’re still haemorrhaging money! Once they fully establish their monopoly and kill the open web, the enshittification will begin.
- jijijijij 9mo ago
  My prediction: AI is the deathblow to IPv6 adoption for the wider web, since blocklists only really work with IPv4. Increasing VPN usage making user tracking and heuristics difficult, AI scrapers stealing appropriated human content and AI spam poisoning its exploitation, not to mention tech monopolization and centralization, the limitations of IPv4 are suddenly becoming an asset and incentives for IPv6 support are zero. On the plus side, we probably got all of IPv6 to build an alternative, non-commercial, better web or whatever network, if we act quickly before routing support is vanishing. IPv6-only housing is cheap, things don’t need to work out. There could be the IPv4-enforced corpo world within walls, and IPv6-enabled wild wide wonderlands.
  - touisteur 9mo ago
    There are ways to build blocklists for IPv6. I saw (used) once bloom filters for this. Inspired by some papers from the 2000s, this one in 2009 https://www.nokia.com/bell-labs/publications-and-media/publications/ipv6-lookups-using-distributed-and-load-balanced-bloom-filters-for-100gbps-core-router-line-cards/ https://www.nokia.com/bell-labs/publications-and-media/publi...
deleted 9mo ago
[deleted]
chlorion 9mo ago
I self host a small static website and a cgit instance on an e2-micro VPS from Google Cloud, and I have got around 8.5 million requests combined from openai and claude over around 160 days. They just infinitely crawl the cgit pages forever unless I block them! (1) root@gentoo-server ~ # egrep 'openai|claude' -c /var/log/lighttpd/access.log 8537094 So I have lighttpd setup to match "claude|openai" in the user agent string and return a 403 if it matches, and a nftables firewall seutp to rate limit spammers, and this seems to help a lot.
- dang 9mo ago
  And those are the good actors! We're under a crawlocalpyse from botnets, er, residential proxies. "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36", anyone?
  - zerocrates 9mo ago
    Yeah the flood of these Chrome UAs with every version number under the sun, and a really large portion being *.0.0.0 version numbers, that's what I've tended to experience lately. Also just kind of every browser user agent ever: Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.2.12) Gecko/20101026 Firefox/3.6.12 (.NET CLR 2.0.50727; .NET CLR 3.0.30729; .NET CLR 3.5.30729; .NET CLR 3.5.21022) There were waves of big and sometimes intrusive traffic admitting to being from Amazon, Anthropic, Google, Meta, etc., but those are easy to block or throttle and aren't that big a deal in the scheme of things.
- zahlman 9mo ago
  The third-party hit-counting service I use implies that I'm not getting any of this bot scraping on my GitHub blog. Is Microsoft doing something to prevent it? Or am I so uncool that even bots don't want to read my content :(
  - lelanthran 9mo ago
    I'm interested in that service and how it works. Link?
adrianwaj 9mo ago
Look no further than x402 micropayments as both the solution and opportunity here. And then a way to return a portion to humans. These AI companies are loaded too (maybe not the long-tail as yet) and the crypto ecosystem is mature. Come one, come all. Make money. Need a Wordpress plugin to start the ball rolling and provide ping endpoints for the AI companies to leach from. They can pay to get those pings too. Give them what they want and charge them. Lower their costs by making their scraping more efficient.
- tdeck 9mo ago
  So your local diner used to have napkin dispensers on every table full of napkins. Then some people started coming in and just taking all the napkins out when they left the restaurant. Now the diner is going broke paying for napkins so they stop stocking them. Seems like we can't have nice things :(. But wait! Great solution! We created a micropayment system. Sign up on your phone and use this QR code to pay 1 cent to dispense each napkin. You can also watch a 30 second ad for a free napkin. Napkins are back at the diner, and now they're properly commodified, as indeed everything should be. -- I don't know about you, but the above scenario doesn't sound like a happy ending to me.
  - adrianwaj 9mo ago
    The dispenser should be placed next to the cashier. There could also be a smaller amount in the dispenser in case someone grabs all of them. The micropayment way is more like a vending machine. The issue is pricing, charge too much and a clone machine will pop up that will be cheaper for the buyer but still make the seller money. Charge too little -- well you'd still be making more than now but there could be a lot of money left on the table. I actually think payments should be variable. People on this thread rant about having a designated way to allow for a big download, how about a designated way to make a payment (even if voluntary) ? x402.txt / crypto.txt / coins.txt / authors.txt / tips.txt / supporters.txt Then give people a way to "show off their generosity."
Jeremy1026 9mo ago
I sysadmin my kids' PTA website. OpenAI was scraping it recently. I saw it looking at the event calendar, request after request to random days. I saw years 1000 through 3000 scroll by. I changed the response to their user agent to an access denied, but it still took about 4 hours for them to stop.
- tonyedgecombe 9mo ago
  I wonder who wrote a crawler like that. Oh wait ...
  - jwe 9mo ago
    I can't follow your thought there. Who wrote it?
    - tenacious_tuna 9mo ago
      Inferring, an LLM
aszantu 9mo ago
there's this one guy I know who writes scripts to poison ai scrapers that ignore the robot.txt, basically creates word salad and random folders that keep going deep without giving the ai anything of value.
rurban 9mo ago
True. I had to kill my dynamic service of collected film festival ratings, because the bots drained the memory of the still available free hosters. I fought it for two years, with user agent and IP ranges, but eventually gave up. So I had to revert to static pages hosted on GitHub pages. The bots cannot kill that. But very limited features
cookiengineer 9mo ago
I don't understand why everyone is complaining so much about AI scrapers. They're easily gullible free machines that can do your computational work! Just show them a download demo link. They gonna download, install and run the binary. Want more instagram likes? Tell them to like your instagram profile to unlock the content. Want your emails answered? Give them access to your inbox and tell them to reply to all the spam mails. They're free use machines. give them something to do, and they'll do it for you.
- dannersy 9mo ago
  I hope this is a meme because it is wild to me how you don't see this as being a problem. You are contributing to an internet for bots and not people.
  - cookiengineer 9mo ago
    They will only stop when it becomes economically unfeasible. > You are contributing to an internet for bots and not people. I'd like to think that my websites and projects are evidence to the contrary.
nottorp 9mo ago
Okay, it's been established that "AI" crawlers are a pest. One of the reasons being that they don't actually run any "AI", that would be too expensive. You can't ban by user agent because that will only catch the few crawlers that are actually honest about it. Aren't there rate limiting solutions built into at least some web servers? At least if you control your own web server, can't you do it through some reverse proxy? Cut off IPs that make more than NN requests in a minute? Require some kind of login to allow more, if you do have endpoints that are designed to be bulk hit? There should be ready made solutions for this still. In spite of the current answer being "lulz it's too hard, just use cloudflare".
- mr_monkey 9mo ago
  MetaBrainz ended up rolling out their own tool, none of the other common suggestions being quite appropriate.
levleontiev 9mo ago
I am terribly sorry for self-advertising, but: I am just now busy building a solution: self-hosted sophisticated rate-limiting. More complex than nginx, more private than cloudfare. Please joint the waitlist if you want to morally support me ;) https://getfairvisor.com/ https://getfairvisor.com/
DanOpcode 9mo ago
I don't get why the AI scrapers need to scrape the same sites and pages over and over again.
- kermatt 9mo ago
  Because the scrapers are poorly written. Efficiency is not a concern for them.
  - DanOpcode 9mo ago
    Feels like parasites who are killing their hosts by DDOSing them to death
sreekanth850 9mo ago
There has been a critical error on this website. Learn more about troubleshooting WordPress. Site is broken now.
OuterVale 9mo ago
The site is down. You can read the article via the Wayback Machine here: https://web.archive.org/web/20260112201803/https://blog.metabrainz.org/2025/12/11/we-cant-have-nice-things-because-of-ai-scrapers/ https://web.archive.org/web/20260112201803/https://blog.meta...
1vuio0pswjnm7 9mo ago
No Anubis: https://web.archive.org/web/20251211141351if_/https://blog.metabrainz.org/2025/12/11/we-cant-have-nice-things-because-of-ai-scrapers/ https://web.archive.org/web/20251211141351if_/https://blog.m...
k310 9mo ago
Metabrainz hosed? > There has been a critical error on this website. > Learn more about troubleshooting WordPress. https://blog.metabrainz.org/2025/12/11/we-cant-have-nice-things-because-of-ai-scrapers/ https://blog.metabrainz.org/2025/12/11/we-cant-have-nice-thi...
xnx 9mo ago
I realized I couldn't think of any crawler worth allowing but Googlebot. Put this in my robots.txt: User-agent: Googlebot Allow: / User-agent: Google-Extended Allow: / User-agent: * Disallow: /
elestor 9mo ago
I am getting a wordpress error so used this: https://web.archive.org/web/20260114073704/https://blog.metabrainz.org/2025/12/11/we-cant-have-nice-things-because-of-ai-scrapers/ https://web.archive.org/web/20260114073704/https://blog.meta...

465 points by LorenDB 9mo ago

I don’t know why anyone would still be trying to pull data off the open internet. Too much signal to noise. So much AI influence already baked into the corpus. You are just going to be reinforcing existing bias. I’m more worried about the day Amazon or Hugging Face take down their large data sets.

saaaaaam 9mo ago
MetaBrainz is a fairly valuable “high signal” dataset though.

SchemaLoad 9mo ago

Cloudflare has a service for this now that will detect AI scrapers and send them to a tarpit of infinite AI generated nonsense pages.

atomic128 9mo ago
Poison Fountain: https://news.ycombinator.com/item?id=46577464 https://news.ycombinator.com/item?id=46577464
yakattak 9mo ago
Do you have a link to that?
- SchemaLoad 9mo ago
  https://blog.cloudflare.com/ai-labyrinth/ https://blog.cloudflare.com/ai-labyrinth/
bitbasher 9mo ago
Wow, so to prevent AI scrapers from harvesting my data I need to send all of my traffic through a third party company that gets to decide who gets to view my content. Great idea!
- rester324 9mo ago
  You can implement this yourself, who is stopping you?
  - zzzeek 9mo ago
    Citation needed
    - GuinansEyebrows 9mo ago
      https://forge.hackers.town/hackers.town/nepenthes https://forge.hackers.town/hackers.town/nepenthes > Citation needed this reply kinda sucks :)
    - zimpenfish 9mo ago
      I use iocaine[0] to generate a tarpit. Yesterday it served ~278k "pages" consisting of ~500MB of gibberish (and that's despite banning most AI scrapers in robots.txt.) [0] https://iocaine.madhouse-project.org https://iocaine.madhouse-project.org
      - doublerabbit 9mo ago
        Unfortunately and you kind of have to count this as the cost of the Internet. You've wasted 500Mb of bandwidth. I've had colocation for eight years+. My monthly b/w cost is now around 20-30Gb a month given to scrapers where I was only be using 1-2Gb a month, years prior. I pay for premium bandwidth (it's a thing) and only get 2TB of usable data. Do I go offline or let it continue?
        zimpenfish 9mo ago
ranger_danger 9mo ago
Modern scrapers are using headless chromium which will not see the invisible links, so I'm not sure how long this will be effective.
timpera 9mo ago
Unfortunately, Cloudflare often destroys the experience for users with shared connections, VPNs, exotic browsers… I had to remove it from my site after too many complaints.
- rudedogg 9mo ago
  Also iCloud Private Relay. CloudFlare is making it impossible to browse privately
  - acdha 9mo ago
    Cloudflare works fine with public relay - they and Fastly provide infrastructure for that service (one half of the blinded pair) so it’s definitely something they test.
- xorcist 9mo ago
  I am sure Cloudflare would have no problem selling you a VPN service. After all, it's not very far from hosting booters and selling DoS protection.
  - sadeshmukh 9mo ago
    Well... https://developers.cloudflare.com/warp-client/warp-modes/#warp-unlimited https://developers.cloudflare.com/warp-client/warp-modes/#wa... Price is $5/mo
inferiorhuman 9mo ago
Which is still a far worse experience than if Cloudflare's services weren't needed.
RobotToaster 9mo ago
Except for the scrapers that pay cloudflare to exempt them.
themafia 9mo ago
The solution, as always, is noise.
m463 9mo ago
cloudflare also blocks my human-is-driving browser all the time "enahble javascript and cookies to continue" also unsupported browser
loopback_device 9mo ago
Not sure "TLS added and removed here :)" as a Service is the right tool in the drawer for this.
taberiand 9mo ago
Savvy move by cloudflare, once they have enough sites behind their service they can charge the AI companies to access their cached copies on a back channel

squigz 9mo ago

> The /metadata/lookup API endpoints (GET and POST versions) now require the caller to send an Authorization token in order for this endpoint to work. > The ListenBrainz Labs API endpoints for mbid-mapping, mbid-mapping-release and mbid-mapping-explain have been removed. Those were always intended for debugging purposes and will also soon be replaced with a new endpoints for our upcoming improved mapper. > LB Radio will now require users to be logged in to use it (and API endpoint users will need to send the Authorization header). The error message for logged in users is a bit clunky at the moment; we’ll fix this once we’ve finished the work for this year’s Year in Music. Seems reasonable and no big deal at all. I'm not entirely sure what "nice things" we can't have because of this. Unauthenticated APIs?

yakattak 9mo ago
I agree its not a big deal. Unauthenticated APIs are nice though, especially for someone who's maybe not as familiar with how APIs work.
mr_monkey 9mo ago
We can't have a free internet that does not demand identification and data collection as a price to pay.

lysace 9mo ago

At some point they must become more cost efficient by pure market economics mechanisms. That implies less load on sites. Much of the scraping that I see is still very dumb/repetative. Like Googlebot in like 2001. (Blocking Chinese IP ranges with the help of some geoip db helps a lot in the short term. Azure as a whole is the second largest source of pure idiocy.)

incompatible 9mo ago
They seem to have so much bubble money at the moment that the cost of scraping is probably a rounding error in their pocket change.
- lysace 9mo ago
  Exactly.
- tdeck 9mo ago
  So the cost of caching should be a rounding error as well. If The Internet Archive can afford to cache vast swathes of the web, then surely the big AI companies can do so.

lep_qq 9mo ago

This is frustrating to watch. MetaBrainz is exactly the kind of project AI companies should be supporting—open data, community-maintained, freely available for download in bulk. Instead they’re: ∙ Ignoring robots.txt (the bare minimum web courtesy) ∙ Bypassing the provided bulk download (literally designed for this use case) ∙ Scraping page-by-page (inefficient for everyone) ∙ Overloading volunteer-run infrastructure ∙ Forcing the project to add auth barriers that hurt legitimate users The irony: if they’d just contacted MetaBrainz and said “hey, we’d like to use your dataset for training,” they’d probably get a bulk export and maybe even attribution. Instead, they’re burning goodwill and forcing open projects to lock down. This pattern is repeating everywhere. Small/medium open data projects can’t afford the infrastructure to handle aggressive scraping, so they either: 1. Add authentication (reduces openness) 2. Rate limit aggressively (hurts legitimate API users) 3. Go offline entirely (community loses the resource) AI companies are externalizing their data acquisition costs onto volunteer projects. It’s a tragedy of the commons, except the “commons” is deliberately maintained infrastructure that these companies could easily afford to support. Have you considered publishing a list of the offending user agents / IP ranges? Might help other projects protect themselves, and public shaming sometimes works when technical measures don’t

tensegrist 9mo ago
Scraping page-by-page (inefficient for everyone) you know what else is "(inefficient for everyone)"? posting the output instead of the prompt
deleted 9mo ago
[deleted]

tensegrist 9mo ago

the more time passes the more i'm convinced that the solution is to—somehow—force everyone to have to go through something like common crawl i don't want people's servers to be pegged at 100% because a stupid dfs scraper is exhaustively traversing their search facets, but i also want the web to remain scrapable by ordinary people, or rather go back to how readily scrapable it used to be before the invention of cloudflare as a middle ground, perhaps we could agree on a new /.well-known/ path meant to contain links to timestamped data dumps?

nostrademons 9mo ago
That's sorta what MetaBrainz did - they offer their whole DB as a single tarball dump, much like what Wikipedia does. I downloaded it in the order of an hour; if I need a MusicBrainz lookup, I just do a local query. For this strategy to work, people need to actually use the DB dumps instead of just defaulting to scraping. Unfortunately scraping is trivially easy, particularly now that AI code assistants can write a working scraper in ~5-10 minutes.
- tonyhart7 9mo ago
  I mean this AI data scrapper would need to scan and fetch billions of website why would they even care over 1 single website ??? You expect instiution to care out of billions website they must scrape daily
  - what 9mo ago
    This is probably the reason. It’s more effort to special case every site that offers dumps than to just unleash your generic scraper on it.
- 8note 9mo ago
  the obvious thing would be to take down their website and only have the DB dump. if thats the useful thing, it doesnt need the wrapper
  - 9mo ago

Pet_Ant 9mo ago

I wish more resources were available legitimately. There is a dataset I need for legitimate research that I cannot even find a way to contact the repo owners. Mind you I take effort to not be burdensome by downloading only what I need and taking time between each request of a couple seconds, and the total data usage is low. Ironically, I supposed you could call it "AI" what I'm using it for, but really it's just data analytics.

smallerfish 9mo ago

Bear in mind that some portion of this could be human directed research. I'm doing a research project right now with 1000 things that I'm building profiles on; to build a full profile requires an agent to do somewhere around 100 different site lookups. Where APIs exist, I've registered API keys and had the agent write a script to query the data in that manner, but that required me to be deliberate about it. Non technie plebs aren't likely to be directed to use an API by the agent.

mr_monkey 9mo ago
This is not that, it's thousand of hits a minute on random endpoints, basically scraping everything all the time.

blell 9mo ago

Seems a mistake to disable the (I assume) faster-to-generate api. Bots will go back to scraping the website itself, increasing load.

hi-wintermute 9mo ago
Setting the API to require a token and adding a honeypot to the pages themselves seems like a decent solution.

dannyobrien 9mo ago

Metabrainz is a great resource -- I wrote about them a few years ago here: https://www.eff.org/deeplinks/2021/06/organizing-public-interest-musicbrainz https://www.eff.org/deeplinks/2021/06/organizing-public-inte... There's something important here in that a public good like Metabrainz would be fine with the AI bots picking up their content -- they're just doing it in a frustratingly inefficient way. It's a co-ordination problem: Metabrainz assumes good intent from bots, and has to lock down when they violate that trust. The bots have a different model -- they assume that the website is adversarially "hiding" its content. They won't believe a random site when it says "Look, stop hitting our API, you can pick all of this data in one go, over in this gzipped tar file." Or better still, this torrent file, where the bots would briefly end up improving the shareability of the data.

fartfeatures 9mo ago
> They won't believe a random site when it says "Look, stop hitting our API, you can pick all of this data in one go, over in this gzipped tar file." What mechanism does a site have for doing that? I don't see anything in robots.txt standard about being able to set priority but I could be missing something.
- jacksnipe 9mo ago
  It’s not great, but you could add it to the body of a 429 response.
  - VTimofeenko 9mo ago
    Genuinely curious: do programs read bodies of 429 responses? In the code bases that I have seen, 429 is not read beyond the code itself
    - jakelazaroff 9mo ago
      Sometimes! The server can also send a retry-after header to indicate when the client is allowed to request the resource again: https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/Retry-After https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/...

bodantogat 9mo ago

I feel the pain — it’s very difficult to detect many of the less ethical scrapers. They use residential IP pools, rotate IPs, and provide valid user agents.

ranger_danger 9mo ago
And many proxy/scraping providers now use real browsers that can automatically bypass cloudflare captchas as well, and a bot with a real browser similarly won't be clicking on any invisible links, so... I am skeptical just how long this will make an appreciable difference.
- trollbridge 9mo ago
  So what you’re saying is that a CAPTCHA cannot actually T C and H A.
  - chrismorgan 9mo ago
    Telling humans and computers apart was never the purpose of CAPTCHAs, only how they initially worked. The name has been a complete misnomer for at least a decade now. Its actual purpose is, and has always been, abuse prevention. Has it been successful? Some yes, some no, and a lot of collateral damage. Its mode of operation now looks a lot like inscrutable blacklisting for some plus inconvenience and bad rate limiting for the rest.
    - j16sdiz 9mo ago
      > was never the purpose of CAPTCHAs, TCHA of CAPTCHA is literally "tell computer human apart"
      - chrismorgan 9mo ago
        You ignored the emphasis and the rest of the sentence. And the rest of the comment. (Also, the T was nominally Turing rather than telling.)

arjie 9mo ago

Someone convinced me last time[0] that these aren't the well-known scrapers we know but other actors. We wouldn't be able to tell, certainly. I'd like to help the scrapers be better about reading my site, but I get why they aren't. I wish there were an established protocol for this. Say a $site/.well-known/machine-readable.json that instructs you on a handful of established software or allows pointing to an appropriate dump. I would gladly provide that for LLMs. Of course this doesn't solve for the use-case where the AI companies are trying to train their models on how to navigate real world sites so I understand it doesn't solve all problems, but one of the things I think I'd like in the future is to have my own personal archive of the web as I know it (Internet Archive is too slow to browse and has very tight rate-limits) and I was surprised by how little protocol support there is for robots. robots.txt is pretty sparse. You can disallow bots and this and that, but what I want to say is "you can get all this data from this git repo" or "here's a dump instead with how to recreate it". Essentially, cooperating with robots is currently under-specified. I understand why: almost all bots have no incentive to cooperate so webmasters do not attempt to. But it would be cool to be able to inform the robots appropriately. To archive Metabrainz there is no way but to browse the pages slowly page-by-page. There's no machine-communicable way that suggests an alternative. 0: https://news.ycombinator.com/item?id=46352723 https://news.ycombinator.com/item?id=46352723

saaaaaam 9mo ago
As referenced in the article, there absolutely is an alternative. https://metabrainz.org/datasets https://metabrainz.org/datasets Linked to from the homepage as “datasets”. I may be too broadly interpreting what you mean by “machine-communicable” in the context of AI scraping though.
- arjie 9mo ago
  Well, imagine the best case and that you're a cooperative bot writer who does not intend to harm website owners. Okay, so you follow robots.txt and all that. That's straightforward. But it's not like you're writing a "metabrainz crawler" and a "metafilter crawler" and a "wiki.roshangeorge.dev crawler". You're presumably trying to write a general Internet crawler. You encounter a site that is clearly a HTTP view into some git repo (say). How do you know to just `git clone` the repo in order to have the data archived as opposed to just browsing the HTTP view. As you can see, I've got a lot of crawlers on my blog as well, but it's a mediawiki instance. I'd gladly host a Mediawiki dump for them to take, but then they'd have to know this was a Mediawiki-based site. How do I tell them that? The humans running the program don't know my site exists. Their bot just browses the universe and finds links and does things. In the Metabrainz case, it's not like the crawler writer knows Metabrainz even exists. It's probably just linked somewhere in the web the crawler is exploring. There's no "if Metabrainz, do this" anywhere in there. The robots.txt is a bit of a blunt-force instrument, and friendly bot writers should follow it. But assuming they do, there's no way for them to know that "inefficient path A to data" is the same as "efficient path B to data" if both are visible to their bot unless they write a YourSite-specific crawler. What I want is to have a way to say "the canonical URL for the data on A is at URL B; you can save us both trouble by just fetching B". In practice, none of this is a problem for me. I cache requests at Cloudflare, and I have Mediawiki caching generated pages, so I can easily weather the bot traffic. But I want to enable good bot writers to save their own resources. It's not reasonable for me to expect them to write a me-crawler, but if there is a format to specify the rules I'm happy to be compliant.

OutOfHere 9mo ago

It is nonsense since AI is the nicest thing.

falloutx 9mo ago

Its not just AI scrappers doing it by themselves but now users are also being trained to put the link in the claude chat/chat gpt and ask it to summarise it. And off course that would show up on the website end as a scraper. In fact firefox now allows you to preview the link and get key points without ever going to the link[1] [1] https://imgur.com/a/3E17Dts https://imgur.com/a/3E17Dts

acatton 9mo ago
> In fact firefox now allows you to preview the link and get key points without ever going to the link[1] > [1] https://imgur.com/a/3E17Dts https://imgur.com/a/3E17Dts This is generated on device with llama.cpp compiled to webassembly (aka wllama) and running SmolLM2-360M. [1] How is this different from the user clicking on the link? In the end, your local firefox will fetch the link in order to summarize it, the same way you would have followed the link and read through the document in reader mode. [1] https://blog.mozilla.org/en/mozilla/ai/ai-tech/ai-link-previews-firefox/ https://blog.mozilla.org/en/mozilla/ai/ai-tech/ai-link-previ...
- falloutx 9mo ago
  I actually didnt know it was using a local model and that it fetches it locally.
  - DrewADesign 9mo ago
    They should advertise that. I pretty much reflexively avoid any mention of AI in interfaces because they usually mean "we're sending this all to openthropigoogosoft so I hope you don't have any secrets."
    - godelski 9mo ago
      > They should advertise that They did Previews can optionally include AI-generated key points, which are processed on your device to protect your privacy. https://www.firefox.com/en-US/firefox/142.0/releasenotes/ https://www.firefox.com/en-US/firefox/142.0/releasenotes/ I'll also add that if you go to the Labs page (in settings) you can enable another local model to semantically search your history

zx8080 9mo ago

Nothing prevents scraper from creating a free account and sending auth token in API requests. I'm not saying the API changes are pointless, but still, what's the catch?

dherls 9mo ago
It's much easier to detect a single account abusing your API and ban them/require payment. Trying to police an endpoint open to the internet is like playing g whackamole
mr_monkey 9mo ago
Those botnets are hitting random endpoints thousands of times a minute. The problem is that each time they switch to a different residential IP so that they are untraceable. That's the frustrating part: not only do they not play by the rules, but they use advanced methods to obfuscate and bypass any protections. That probably costs them a fair amount too, all that to access free data they can download as a tar file... They won't be able to create thousands of API keys a minute, and if they reuse the keys they'll very easily be identified and blocked.

tommek4077 9mo ago

How do they get overloaded? Is the website too slow? I have a quite big wiki online and barely see any impact from bots.

blell 9mo ago
In these discussions no one will admit this, but the answer is generally yes. Websites written in python and stuff like that.
- tclancy 9mo ago
  Yes yes, definitely people don’t know what they’re doing and not that they’re operating on a scale or problem you are not. Metabrainz cannot cache all of these links as most of them are hardly ever hit. Try to assume good intent.
  - tommek4077 9mo ago
    But serving HTML is unbelievably cheap, isn't it?
    - chlorion 9mo ago
      It adds up very quickly.
    - tclancy 9mo ago
      Run 72,000 database queries to generate a bunch of random HTML files no one has asked for in five years is not, especially compared to downloading the files designed for it.
- Qwertious 9mo ago
  It's not "written too slow" if you e.g. only get 50 users a week, though. If bots add so much load that you need to go optimise your website for them, then that's a bot problem not a website problem.
roblh 9mo ago

Olshansky 9mo ago

Resurfacing a proposal I put out on llms-txt: https://github.com/AnswerDotAI/llms-txt/issues/88 https://github.com/AnswerDotAI/llms-txt/issues/88 We should add optional `tips` addresses in llms.txt files. We're also working on enabling and solving this at Grove.city. Human <-> Agent <-> Human Tips don't account for all the edge cases, but they're a necessary and happy neutral medium. Moving fast. Would love to share more with the community. Wrote about it here: https://x.com/olshansky/status/2008282844624216293 https://x.com/olshansky/status/2008282844624216293

bediger4000 9mo ago
At this point, it's pretty clear that the AI scrapers won't be limited by any voluntary restrictions. Bytedance never seemed to live with robots.txt limitations, and I think at least some of the others didn't either. I can't see this working.
- Olshansky 9mo ago
  The thesis/approach is: - Humans tip humans as a lottery ticket for an experience (meet the creator) or sweepstakes (free stuff) - Agents tip humans because they know they'll need original online content in the long-term to keep improving. For the latter, frontier labs will need to fund their training/inference agents with a tipping jar. There's no guarantee, but I can see it happening given where things are movin.
  - philipwhiuk 9mo ago
    > Agents tip humans because they know they'll need original online content in the long-term to keep improving. Why would an agent have any long term incentive. It's trained to 'do what it's told', not to predict the consequences of it's actions.
ricardo81 9mo ago
I like the idea, (original) content creators being credited is good for the entire ecosystem. Though if LLMs are willingly ignoring robots.txt, often hiding themselves or using third party scraped data- are they going to pay?

ranger_danger 9mo ago

Many years ago AWS came up with a "Requester Pays" model for their S3 storage, where you can make a request for someone else's object using your own account and it would charge the transfer cost to your own account instead of theirs. I wonder if a model similar to this (but decentralized/federated or something) could be used to help fight bots?

kpcyrd 9mo ago

I wish this wasn't necessary, but the next steps forward are likely: a) Have a reverse proxy that keeps a "request budget" per IP and per net block, but instead of blocking requests, causing the client to rotate their IP, the requests get throttled/slowed down, without dropping them. b) Write your API servers in more efficient languages. According to their Github, their backend runs on Perl and Python. These technologies have been "good enough" for quite some time, but considering current circumstances and until a better solution is found, this may not be the case anymore and performance and cpu cost per request does matter these days. c) Optimize your database queries, remove as much code as possible from your unauthenticated GET request handlers, require authentication for the expensive ones.

saxonww 9mo ago

I haven't really looked but I wonder if there are any IP reputation services tracking AI crawlers the same way they track tor relays and vpns and such. If those databases were accurate it seems like you could prevent those crawlers from ever hitting the site. Maybe they change too often/too quickly.

randyl 9mo ago

The SQLite team faced a similar problem last year, and Richard Hipp (the creator of SQLite) made almost the same comment: "The malefactor behind this attack could just clone the whole SQLite source repository and search all the content on his own machine, at his leisure. But no: Being evil, the culprit feels compelled to ruin it for everyone else. This is why you don't get to keep nice things...." https://sqlite.org/forum/forumpost/7d3eb059f81ff694 https://sqlite.org/forum/forumpost/7d3eb059f81ff694

lysace 9mo ago
[flagged]
- Forgeties79 9mo ago
  There are people behind the web crawler. If they’re so well funded they can exert a little effort to not so badly inconvenience people as they steal their training data.
  - lysace 9mo ago
    [flagged]
    - mikestew 9mo ago
      It may come as shock to you ("Video/audio producer") It’s one thing to ignore parent’s point entry, but no reason to be an ass about it.
      - deleted 9mo ago
        [deleted]
    - NegativeK 9mo ago
      I've downvoted you for being incredibly aggressive in your responses. I'm not sure why you're ad homineming the parent commenter, but it's not helping the discussion.

dgxyz 9mo ago

I actually deleted my web site early 2025 and removed the A record from DNS because of AI scraper traffic. It had been up for 22 years. Nothing important or particularly useful on it but it's an indicator of the times.

tommek4077 9mo ago
But why?
- dgxyz 9mo ago
  No humans, no point.
  - crazygringo 9mo ago
    But AI scraping doesn't remove humans...? Even if humans make up a smaller proportion of your traffic, they're still the same number in absolute terms.
    - dgxyz 9mo ago
      [dead]

tooltower 9mo ago

> Rather than downloading our dataset in one complete download, they insist on loading all of MusicBrainz one page at a time. Is there a standard mechanism for batch-downloading a public site? I'm not too familiar with crawlers these days.

EnderWT 9mo ago
They're not talking about downloading the web pages. The data is available in a bulk download: https://listenbrainz.org/data/ https://listenbrainz.org/data/
TeMPOraL 9mo ago
There isn't. There never was one, because vast majority of websites are actually selfish with respect to data, even when that's entirely pointless. You can see this even here, with how some people complain LLMs made them stop writing their blogs: turns out plenty of people say they write for others to read, but they care more about tracking and controlling the audience. Anyway, all that means there was never a critical mass of sites large enough for a default bulk data dump discovery to become established. This means even the most well-intentioned scrappers cannot reliably determine if such mechanism exist, and have to scrap per-page anyway.
- VonGallifrey 9mo ago
  > turns out plenty of people say they write for others to read LLMs are not people. They don't write blogs so that a company can profit from their writing by training LLMs on it. They write for others to read their ideas.
  - TeMPOraL 9mo ago
    LLMs aren't making their owners money by just idling on datacenters worth of GPU. They're making money by being useful for users that pay for access. The knowledge and insights from writings that go into training data all end up being read by people directly, as well as inform even more useful output and work benefiting even more people.

garganzol 9mo ago

Nowadays people complain about AI scrapers with the same vain as they complained about search indexers a way back when. Just a few years later, people had stopped caring too much about storage access and bandwidth, and started begging search engines to visit their websites. Every trick on the planet Earth, SEO optimization, etc. Looking forward to the time when everybody suddenly starts to embrace AI indexers and welcome them. History does not repeat itself but it rhymes.

Guvante 9mo ago
Except robots.txt was the actual real solution to search indexing...
phyzome 9mo ago
We already know the solution: One well-behaved, shared scraper could serve all of the AI companies simultaneously. The problem is that they're not doing it.
- garganzol 9mo ago
  This is an interesting approach. Archive.org could be such a solution, kind of. Not its cold storage as it's now, but a warm access layer. Sponsorship by AI companies would a good initiative for the project.
  - phyzome 9mo ago
    I can't imagine IA ever going for it. You'd need a separate org that just scrapes for AI training, because its bot is going to be blocked by anyone who is anti-AI. It wouldn't make sense for it to serve multiple purposes. Common Crawl would be a better fit, but still might not want to serve in that capacity.
linkregister 9mo ago
Search indexing historically has had several of orders less impact on bandwidth and processing costs to website maintainers. My recommendation is to copy the text in this article and pass it LLM to summarize this article's key points, since it appears you missed the central complaint of the article.

jmward01 9mo ago

Bummer. I have used them a lot when I was ripping my cds. Anonymity is a massive value of the web (at least the appearance of anonymity). I wonder if there is a way to have a central anonymous system that just relays trust, not identity. So maybe something like you can get a token but its trust is very nearly zero until you combine it with other tokens. Combining tokens combines their trust and their consequences. If one token is abused that abuse reflects on the whole token chain. The connection can be revoked for a token but trust takes time to rebuild so it would take a time for their token trust value to go up. Sort of the 'word of mouth' effect but in electronic form. 'I vouch for 2345asdf334t324sda. That's a great user agent!' A bit (a lot) elaborate but maybe there is a beginning of an idea there, maybe. Definitely I don't want to loose anonymity (or the perception there of) for services like musicbrainz but at the same point they need some mechanism that gives them trust and right now I just don't know of a good one that doesn't have identity attached.

janc_ 9mo ago
These AI crawlers already steal residential user connections to do their scraping. They'll happily steal your trust tokens too…

accrual 9mo ago

Grateful for Metabrainz putting in this work to keep the service up. We really ought to have some kind of "I am an AI!" signal to route the request properly into a long queue...

gamer191 9mo ago
Companies wouldn’t send it because they know that most websites would block them
zerocrates 9mo ago
Yeah, they can just set the evil bit in their IP packets.
jeroenhd 9mo ago
AI scrapers already fake user agent headers, ignore robots.txt, and go through botnets to bypass firewall rules. They're not going to put out such a signal if they can help it.

devhouse 9mo ago

random idea, instead of blocking scrapers, maybe detect them (via user-agent, request patterns, ignoring robots.txt) and serve them garbage data wrapped in dad jokes. if (isSuspiciousScraper(req)) { return res.json({ data: getDadJoke(), artist: "Rick Astley", // always album: "Never Gonna Give You Up" }); }

mattdanger 9mo ago
This is what Cloudflare's Labyrinth product does https://blog.cloudflare.com/ai-labyrinth/ https://blog.cloudflare.com/ai-labyrinth/

a-dub 9mo ago

it's funny how ai is the problem that the cryptocurrencyverse was always in search of...

observationist 9mo ago

Some sort of hashing and incremental serial versioning type standards with http servers would allow hitting a server up for incremental udpates, allow clean access to content, with rate limits, and even keep up with live feeds and chats and so on. Something like this in practice breaks a lot of the adtech surveillance and telemetry, and makes use of local storage, and incidentally empowers p2p sharing with a verifiable source of truth, which incentivizes things like IPFS and big p2p networks. The biggest reason we don't already have this is the exploitation of user data for monetization and intrusive modeling. It's easy to build proof of concept instances of things like that and there are other technologies that make use of it, but we'd need widespread adoption and implementation across the web. It solves the coordination problem, allows for useful throttling to shut out bad traffic while still enabling public and open access to content. The technical side to this is already done. Merkle trees and hashing and crypto verification are solid, proven tech with standard implementations and documentation, implementing the features into most web servers would be simple, and it would reduce load on infrastructure by a huge amount. It'd also result in IPFS and offsite sharing and distributed content - blazing fast, efficient, local focused browsers. It would force opt in telemetry and adtech surveillance, but would also increase the difference in appearance between human/user traffic and automated bots and scrapers. We can't have nice things because the powers that be decided that adtech money was worth far more than efficiency, interoperability, and things like user privacy and autonomy.

zzo38computer 9mo ago

There some other possibilities, such as: Require some special header for accessing them, without needing a API token if it is public data. HTTPS will not necessarily be required. Scrapers can still use it but it seems unlikely unless it becomes common enough; but if they do then you can remove that and require proper authentication. Another is to use X.509 client certificates for authentication, which is more secure than using API keys anyways; however, this will require that you have a X.509 certificate, and some people might not want that, so due to that, perhaps it should not be mandatory.

akuyou 9mo ago

AI is destroying the free internet along with everything else My web host suspended my website account last week due to a sudden large volume of requests to it - effectively punishing me for being scraped by bots. I've had to move to a new host to get back up, but what hope does the little guy have? it's like GPU and ram prices, it doesn't matter if I pay 10x 100x or 1000x more than I did, the AI companies have infinite resources, and they don't care what damage they do in the rush to become the no 1 in the industry The cynic in me would say it's intentional, destroy all the free sites so you have to get your info from their ai models, price home users out of high end hardware so they have to lease the functions from big companies

nicbou 9mo ago
Then they use the data to deny you traffic. AI summaries are wrecking the independent web. Losing more than half or more of your traffic was pretty common in 2025. It’s killing the economics of sharing hard-earned information. So we are spending more resources reaching a lot less people, because a few big companies are capturing the value for their shareholders. And that’s while they’re still haemorrhaging money! Once they fully establish their monopoly and kill the open web, the enshittification will begin.
jijijijij 9mo ago
My prediction: AI is the deathblow to IPv6 adoption for the wider web, since blocklists only really work with IPv4. Increasing VPN usage making user tracking and heuristics difficult, AI scrapers stealing appropriated human content and AI spam poisoning its exploitation, not to mention tech monopolization and centralization, the limitations of IPv4 are suddenly becoming an asset and incentives for IPv6 support are zero. On the plus side, we probably got all of IPv6 to build an alternative, non-commercial, better web or whatever network, if we act quickly before routing support is vanishing. IPv6-only housing is cheap, things don’t need to work out. There could be the IPv4-enforced corpo world within walls, and IPv6-enabled wild wide wonderlands.
- touisteur 9mo ago
  There are ways to build blocklists for IPv6. I saw (used) once bloom filters for this. Inspired by some papers from the 2000s, this one in 2009 https://www.nokia.com/bell-labs/publications-and-media/publications/ipv6-lookups-using-distributed-and-load-balanced-bloom-filters-for-100gbps-core-router-line-cards/ https://www.nokia.com/bell-labs/publications-and-media/publi...

deleted 9mo ago

[deleted]

chlorion 9mo ago

I self host a small static website and a cgit instance on an e2-micro VPS from Google Cloud, and I have got around 8.5 million requests combined from openai and claude over around 160 days. They just infinitely crawl the cgit pages forever unless I block them! (1) root@gentoo-server ~ # egrep 'openai|claude' -c /var/log/lighttpd/access.log 8537094 So I have lighttpd setup to match "claude|openai" in the user agent string and return a 403 if it matches, and a nftables firewall seutp to rate limit spammers, and this seems to help a lot.

dang 9mo ago
And those are the good actors! We're under a crawlocalpyse from botnets, er, residential proxies. "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36", anyone?
- zerocrates 9mo ago
  Yeah the flood of these Chrome UAs with every version number under the sun, and a really large portion being *.0.0.0 version numbers, that's what I've tended to experience lately. Also just kind of every browser user agent ever: Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.2.12) Gecko/20101026 Firefox/3.6.12 (.NET CLR 2.0.50727; .NET CLR 3.0.30729; .NET CLR 3.5.30729; .NET CLR 3.5.21022) There were waves of big and sometimes intrusive traffic admitting to being from Amazon, Anthropic, Google, Meta, etc., but those are easy to block or throttle and aren't that big a deal in the scheme of things.
zahlman 9mo ago
The third-party hit-counting service I use implies that I'm not getting any of this bot scraping on my GitHub blog. Is Microsoft doing something to prevent it? Or am I so uncool that even bots don't want to read my content :(
- lelanthran 9mo ago
  I'm interested in that service and how it works. Link?

adrianwaj 9mo ago

Look no further than x402 micropayments as both the solution and opportunity here. And then a way to return a portion to humans. These AI companies are loaded too (maybe not the long-tail as yet) and the crypto ecosystem is mature. Come one, come all. Make money. Need a Wordpress plugin to start the ball rolling and provide ping endpoints for the AI companies to leach from. They can pay to get those pings too. Give them what they want and charge them. Lower their costs by making their scraping more efficient.

tdeck 9mo ago
So your local diner used to have napkin dispensers on every table full of napkins. Then some people started coming in and just taking all the napkins out when they left the restaurant. Now the diner is going broke paying for napkins so they stop stocking them. Seems like we can't have nice things :(. But wait! Great solution! We created a micropayment system. Sign up on your phone and use this QR code to pay 1 cent to dispense each napkin. You can also watch a 30 second ad for a free napkin. Napkins are back at the diner, and now they're properly commodified, as indeed everything should be. -- I don't know about you, but the above scenario doesn't sound like a happy ending to me.
- adrianwaj 9mo ago
  The dispenser should be placed next to the cashier. There could also be a smaller amount in the dispenser in case someone grabs all of them. The micropayment way is more like a vending machine. The issue is pricing, charge too much and a clone machine will pop up that will be cheaper for the buyer but still make the seller money. Charge too little -- well you'd still be making more than now but there could be a lot of money left on the table. I actually think payments should be variable. People on this thread rant about having a designated way to allow for a big download, how about a designated way to make a payment (even if voluntary) ? x402.txt / crypto.txt / coins.txt / authors.txt / tips.txt / supporters.txt Then give people a way to "show off their generosity."

Jeremy1026 9mo ago

I sysadmin my kids' PTA website. OpenAI was scraping it recently. I saw it looking at the event calendar, request after request to random days. I saw years 1000 through 3000 scroll by. I changed the response to their user agent to an access denied, but it still took about 4 hours for them to stop.

tonyedgecombe 9mo ago
I wonder who wrote a crawler like that. Oh wait ...
- jwe 9mo ago
  I can't follow your thought there. Who wrote it?
  - tenacious_tuna 9mo ago
    Inferring, an LLM

aszantu 9mo ago

there's this one guy I know who writes scripts to poison ai scrapers that ignore the robot.txt, basically creates word salad and random folders that keep going deep without giving the ai anything of value.

rurban 9mo ago

True. I had to kill my dynamic service of collected film festival ratings, because the bots drained the memory of the still available free hosters. I fought it for two years, with user agent and IP ranges, but eventually gave up. So I had to revert to static pages hosted on GitHub pages. The bots cannot kill that. But very limited features

cookiengineer 9mo ago

I don't understand why everyone is complaining so much about AI scrapers. They're easily gullible free machines that can do your computational work! Just show them a download demo link. They gonna download, install and run the binary. Want more instagram likes? Tell them to like your instagram profile to unlock the content. Want your emails answered? Give them access to your inbox and tell them to reply to all the spam mails. They're free use machines. give them something to do, and they'll do it for you.

dannersy 9mo ago
I hope this is a meme because it is wild to me how you don't see this as being a problem. You are contributing to an internet for bots and not people.
- cookiengineer 9mo ago
  They will only stop when it becomes economically unfeasible. > You are contributing to an internet for bots and not people. I'd like to think that my websites and projects are evidence to the contrary.

nottorp 9mo ago

Okay, it's been established that "AI" crawlers are a pest. One of the reasons being that they don't actually run any "AI", that would be too expensive. You can't ban by user agent because that will only catch the few crawlers that are actually honest about it. Aren't there rate limiting solutions built into at least some web servers? At least if you control your own web server, can't you do it through some reverse proxy? Cut off IPs that make more than NN requests in a minute? Require some kind of login to allow more, if you do have endpoints that are designed to be bulk hit? There should be ready made solutions for this still. In spite of the current answer being "lulz it's too hard, just use cloudflare".

mr_monkey 9mo ago
MetaBrainz ended up rolling out their own tool, none of the other common suggestions being quite appropriate.

levleontiev 9mo ago

I am terribly sorry for self-advertising, but: I am just now busy building a solution: self-hosted sophisticated rate-limiting. More complex than nginx, more private than cloudfare. Please joint the waitlist if you want to morally support me ;) https://getfairvisor.com/ https://getfairvisor.com/

DanOpcode 9mo ago

I don't get why the AI scrapers need to scrape the same sites and pages over and over again.

kermatt 9mo ago
Because the scrapers are poorly written. Efficiency is not a concern for them.
- DanOpcode 9mo ago
  Feels like parasites who are killing their hosts by DDOSing them to death

sreekanth850 9mo ago

There has been a critical error on this website. Learn more about troubleshooting WordPress. Site is broken now.

OuterVale 9mo ago

The site is down. You can read the article via the Wayback Machine here: https://web.archive.org/web/20260112201803/https://blog.metabrainz.org/2025/12/11/we-cant-have-nice-things-because-of-ai-scrapers/ https://web.archive.org/web/20260112201803/https://blog.meta...

1vuio0pswjnm7 9mo ago

No Anubis: https://web.archive.org/web/20251211141351if_/https://blog.metabrainz.org/2025/12/11/we-cant-have-nice-things-because-of-ai-scrapers/ https://web.archive.org/web/20251211141351if_/https://blog.m...

k310 9mo ago

Metabrainz hosed? > There has been a critical error on this website. > Learn more about troubleshooting WordPress. https://blog.metabrainz.org/2025/12/11/we-cant-have-nice-things-because-of-ai-scrapers/ https://blog.metabrainz.org/2025/12/11/we-cant-have-nice-thi...

xnx 9mo ago

I realized I couldn't think of any crawler worth allowing but Googlebot. Put this in my robots.txt: User-agent: Googlebot Allow: / User-agent: Google-Extended Allow: / User-agent: * Disallow: /

elestor 9mo ago

I am getting a wordpress error so used this: https://web.archive.org/web/20260114073704/https://blog.metabrainz.org/2025/12/11/we-cant-have-nice-things-because-of-ai-scrapers/ https://web.archive.org/web/20260114073704/https://blog.meta...