5 ms·
AI haters build tarpits to trap and trick AI scrapers that ignore robots.txt
- dinobones 2y agook... if depth > 5 and if sem_hash(content) in hist: return
- anotherhue 2y agoI can generate more random content than you can store.
- rossdavidh 2y agoIt's a radar gun/radar detector kind of situation. You can always change your strategies; countless top-level links that are shown on the menu in a way that no human sees them (due to font color or size or etc). Small number of pages with a few that go on forever, in a way that any human would stop reading but a bot may have trouble detecting. Real (for humans) text in images, with endless invisible text for bots. Etc. I think it will probably make it harder for screen readers, unfortunately.
- dylan604 2y ago> countless top-level links that are shown on the menu in a way that no human sees them (due to font color or size or etc). wouldn't this lower their page rankings? that's the kind of shenanigans of the old days with meta key word stuff and what not
- smaudet 2y agoI'm not sure the old search engines are any good anymore, though, so we don't really care... That's kind of the point. AI has just been more terrible news after terrible news. It is NOT a good thing. Unless you know, you like being covered in oil and lit on fire...
- rossdavidh 2y agoIf you've got your robots.txt set to tell search engines to go away, or at least don't look at those pages, then you probably aren't worried about that, I assume?
- dylan604 2y agothere's a difference between an ai-bot and a search engine bot though
- andrewfromx 2y ago"trap crawlers in infinite mazes of gibberish data, potentially increasing AI training costs and poisoning datasets. While their effectiveness is debated, creators see them as a form of resistance against unchecked AI development."
- dangle1 2y agoRelated to previous: https://news.ycombinator.com/item?id=42725147 https://news.ycombinator.com/item?id=42725147
- dang 2y agoThanks! Macroexpanded: Nepenthes is a tarpit to catch AI web crawlers - https://news.ycombinator.com/item?id=42725147 https://news.ycombinator.com/item?id=42725147 - Jan 2025 (263 comments)
- BizarreByte 2y agoAI haters? I don't hate AI I just don't want things I've created being used to enrich multi-billion dollar companies for free. These companies are behaving poorly and they should expect this kind of push back.
- someone_eu 2y agoIt is not even just the copyright issue. The article completely misses the point that AI scrapers are not a "future threat of AI domination". They already do damage by DDOSing site's networking infrastructure and inflicting very real costs to a site hoster. Even when the data is completely free, like in case of Wikipedia or OpenstreetMaps, scraping it is unethical and should be illegal. Most of the open data resources have procedures, which allow downloading of the data in the archived form, without need for scraping. They are built with sharing in mind. So the arguments the article tries to use (what if it is for public good?) has no sense. 1) it is not 2) there are many ways to fetch the open data properly and respectfully.
- senko 2y agoIs this because they are multi-billion dollar companies, or because they behave poorly, or because you haven't been properly compensated for your contribution to the content on the internet? It is very likely that voting, or voting with your wallet, or probably any kind of activism, would have more impact than withdrawing from the (online) public life.
- BitterCritter 2y agoI’m very confused as to what my $23.45 wallet has to do with what billion dollar Ai companies do.
- JTyQZSnP3cQGa8B 2y ago> because they are multi-billion dollar companies, or because they behave poorly Both for me. They should be spanked for their behavior and lack of respect. I don’t want compensation though because I write open-source applications, but I want them to respect the license (which they don’t obviously). Also I don’t understand why you feel anybody is withdrawing from the internet. It’s only a tarpit and I’m sure most of those who react don’t have ChatGPT subscriptions.
- anotherhue 2y agoI wonder if we could run the cheapest smallest 1b model (or smaller) llm to generate this data in a way that's just plausible enough to be ingested. A little note in robots.txt offering commercial terms would also be available.
- Dwedit 2y agoUsually it's just a Markov Chain rather than an LLM to generate the garbage data.
- Dwedit 2y agoThere's also the other kind of AI haters who do not give any anti-bot indicators about their tarpits (no robots.txt entry, no "nofollow", etc), and want to intentionally feed them poisoned data.
- dylan604 2y agowhat are the odds that one of these people are running a server that is popular enough to garner that kind of attention?
- grajaganDev 2y agoThe odds a pretty good if the people running the server work for an intelligence agency.
- selfhoster 2y agoThat's a great idea! Since there are probably not any laws against this yet, it sounds like a great way to fight back!! Poison the AI well, I like it.
- mvdtnz 2y agoI'm not obligated to design my websites with AI bots (or search engine bots) in mind. If I want to publish mountains of Markhov generated data on my own servers that's my right. There's no rule that I need to use robots.txt, I didn't ask your shitty bot to crawl my site.
- Der_Einzige 2y ago[flagged]
- crazydoggers 2y agoI hope you’re being sarcastic. Today’s “AI” is as close to being a general intelligence with sentience as a toaster is to a laptop. You might as well be concerned about revenge for all the electronics you sent to the dump or the software lines you deleted. Why anyone would think some future sentient AI would concern itself with what you did in the past with your software bots or chat bots?? People really need to get a grip.
- rsynnott 2y ago> Why anyone would think some future sentient AI would concern itself with what you did in the past with your software bots or chat bots?? This is some version of one of the more ridiculous things that has been thrown up by the nerd new religious movement on LessWrong and similar sites: https://en.wikipedia.org/wiki/Roko%27s_basilisk https://en.wikipedia.org/wiki/Roko%27s_basilisk (It's somewhat unclear to me whether anyone actually believes in this, but some of the more fanatical AI-will-change-everything people _seem_ to more or less accept the premise.)
- amelius 2y agoWhat if the scrapers use breadth-first search?
- uberman 2y agoArticle seems super biased to me. Why are tarpits repeatedly characterized as attacks rather than as defense?
- grajaganDev 2y agoAgreed. Some commenters where pretty upset about the tarpits the last time this was posted on HN.
- simondanerd 2y agoIn my opinion, if you access my content or web service, you accept the risk of not asking me first or knowing what's going to happen to the content when you do access it. It's mine, I've given you access, I'm not going to guarantee it's quality or how many times you'll need to try to get it before your scraper gets blocked for too many requests.
- nickphx 2y agoI block them because they're not paying me to use my resources. They would block me if I made a similar volume of requests.
- marginalia_nu 2y agoTo be fair there's tens of thousands of content farms filling the web with ai slop. That's far more likely to harm AI scrapers than these hijinks. Most crawlers use some form of timeout mechanism, usually informed by some priority scheduling. This deals reasonably well with crawler traps. Since Nephentes-like traps are getting so common now (and in particular, not always behind robots.txt), I added a clause to Marginalia's crawler that prevents it from extracting links from pages that are less than 2 Kb and take more than 9 seconds to load. It's 4 lines of code and means the crawler doesn't get stuck at all. I totally get the frustration though. My sites get an insane amount of bot traffic as well. I think roughly 1% of the search traffic to the html endpoint is human, and that's while providing a free API they could use instead. ... I just don't think this is going to fix anything.
- rglover 2y agoI've yet to do it, but my pending defense strategy was to download plain text copies of erotica novels and when I detect a chat bot crawler, just redirect it to that folder and let it go...hard. Calling people who don't want certain things hoovered up by an LLM "AI haters" is a level of manipulation I'd think was only reserved for someone with a vested interest in the tech. Just encourages devious behavior instead of a more diplomatic approach of respecting people's wishes (read: robots.txt).
- rsynnott 2y ago> but my pending defense strategy was to download plain text copies of erotica novels and when I detect a chat bot crawler, just redirect it to that folder and let it go Might I suggest https://tvtropes.org/pmwiki/pmwiki.php/Literature/BelindaBlinked https://tvtropes.org/pmwiki/pmwiki.php/Literature/BelindaBli... As a bonus, beyond introducing the magic robot to the wonderful world of really poorly written erotica, it's also very anachronistic (widespread use of fax machines, pagers, smartphones, LinkedIn, and East Germany appear to coexist at a single moment in time), so will cause further confusion.
- bsnnkv 2y agoI run a service[1] that was getting hit pretty hard by these crawlers. Ultimately instead of going down this path, I decided to just start charging for access to the service (it was long overdue)[2]. Users who are logged out can still see old cached content (which is a single DB read op), but to aggregate new content requires an account. I feel like this is a good (enough) middleground solution for now. [1]: https://kulli.sh https://kulli.sh [2]: https://lgug2z.com/articles/in-the-age-of-ai-crawlers-i-have-chosen-to-paywall/ https://lgug2z.com/articles/in-the-age-of-ai-crawlers-i-have...
- andrewmutz 2y agoRegardless of your views on AI, LLMs are going to be influential in the future. If you work to keep your content away from models, it's hard to see how you benefit. 25 years ago, if you had blocked the googlebot scraper because you resented google search, it would only have worked to marginalize the information you were offering up on the internet. Avoiding LLM training datasets will lead to similar outcomes.
- grajaganDev 2y agoI think this is a weak analogy. What benefit is gained by allowing AI companies to train on your content? LLMs work on a token by token basis.
- andrewmutz 2y agoDepends on who you are and what your content is, but shutting yourself off is unlikely to benefit you.
- freitasm 2y ago> AI haters build tarpits to trap and trick AI scrapers that ignore robots.txt > That's likely an appealing bonus feature for any site owners who, like Aaron, are fed up with paying for AI scraping and just want to watch AI burn. Website owners aren't "haters" if bots ignore robots.txt, consuming resources that translate to expenses and a bad experience for legitimate website visitors. Why would the website owner have to commission much larger server(s), pay more for this traffic and get nothing in return? At least search engines send human visitors your way. It's not "AI haters". It's exploitation hating.
- deleted 2y ago[deleted]