4 ms·
I've spent the last few days adding traps to one of my websites, ironically using LLMs of course, and I've been having quite a lot of fun doing it. Instead of
by robotmay 1mo ago
I've spent the last few days adding traps to one of my websites, ironically using LLMs of course, and I've been having quite a lot of fun doing it.
Instead of the proof-of-work system of Anubis, I've gone down the iocaine route but implemented it in my application itself, as it's built in Elixir and causing problems for scrapers is really fun when it takes almost no server resources.
Currently I trick bad scrapers into a fake infinite black hole path with the promise of tasty data, then serve images to them one byte at a time over 15 minutes (after sending the header quickly), bloat the responses to cost them tokens, and randomly return AI generated images of sexy toasters. I have an admin dashboard with a little leaderboard for which ones get the most stuffed, and it keeps my heart warm on these wet autumn evenings.
- someothherguyy 1mo agowouldn't that just make your connection load worse?
- robotmay 1mo agoCurrently not really an issue on this site but connections aren't an issue on Elixir usually anyway, unless you get up to about 1 million on one machine IIRC.
- someothherguyy 1mo agonormally depends on the type of traffic you are serving, but that seems like a weird thing to say as a blanket statement.
- jopsen 1mo agoProbably they must be deduplicating text they've seen before. The only punishment would be unique text that trains their models to be degenerate. And even then you'd probably have to serve across many domains.
- dspillett 1mo agoOne thing I've thought of trying is, if I ever get around to putting things back online, loading the pages with altered versions of themselves in a manner that human users are unlikely to see (though they could if they wanted to) but scrapers are unlikely to ignore (because they look enough like something users might see, perhaps by opening a <detail> tag). The altered versions would be the page or paragraphs of it, with sentences and words swapped arbitrarily so that training on the data adds as many iffy token sequences as correct ones.
- jopsen 1mo agoTo be clear, making pages with bad training data for bots won't make the bots go away. It'll just punish the bad actors running the scrapers. As the original poster mentions that they are using TVs as proxies to get residential IPs, one really can't think of these bots as criminal enterprises. Sadly, if the bad actors has two cents for brain, they'll limit how much importance each domain name can have on training data. To mitigate impact of bad data like this. (Note I'd suggest only linking to them from robot.txt as pages to not be indexed, that way no human or well behaved not ever will see them, which is kind of the point).
- dspillett 1mo ago> … making pages with bad training data for bots won't make the bots go away. It'll just punish the bad actors running the scrapers. Exactly. I can't hope to keep them all at bay, but I can at least have the petty little victory of making their visit less convenient than it might otherwise be. > if the bad actors has two cents for brain I suspect that a majority of them are little better than the script kiddies of yore, running tools with minimal understanding of what is actually going on. > I'd suggest only linking to them from robot.txt as pages to not be indexed Agreed. Blocking all bots from all pages, well those that bother to listen to robots.txt. All bots because pretty much all of them are scraping for AI and similar these days, even googlebot. If I want people to see my stuff they'll get a link, and maybe they'll pass it on further, but all indexers/trainers can get stuffed. I'll likely make an exception for archive.org and similar.
- embedding-shape 1mo ago> I have an admin dashboard with a little leaderboard for which ones get the most stuffed You can't tease us like that without showing screenshots.
- robotmay 1mo agoHaha here's a few snippets (will auto expire in a week, future readers): https://ibb.co/mCgXPZPg https://ibb.co/mCgXPZPg https://ibb.co/qFxVRQGW https://ibb.co/qFxVRQGW https://ibb.co/9m6xV1xg https://ibb.co/9m6xV1xg I only just added the token wasting measure so I'm really curious to see what happens if another bot loads over 400 pages.
- altairprime 1mo ago(FYI, these are comically distorted on mobile.)
- Barbing 1mo agoDirect may help autoresize for you: https://i.ibb.co/rGNcX8XN/Screenshot-2026-08-30-at-20-29-42.png https://i.ibb.co/rGNcX8XN/Screenshot-2026-08-30-at-20-29-42.... - https://i.ibb.co/yBV78jTY/Screenshot-2026-08-30-at-20-29-59.png https://i.ibb.co/yBV78jTY/Screenshot-2026-08-30-at-20-29-59.... - https://i.ibb.co/twg0xv0K/Screenshot-2026-08-30-at-20-30-13.png https://i.ibb.co/twg0xv0K/Screenshot-2026-08-30-at-20-30-13.... No hot toaster pics though :( on second thought perhaps better in our imaginations
- altairprime 1mo agoYep, those work, thanks!
- worldsavior 1mo agoNGL, that's something a crawler owner would say.
- inventor7777 1mo agoYou can't tease us with that and not share any information! ;-)
- robotmay 1mo agoI've added a few screenshots above! Elixir's really fun to do this in because the BEAM will let you have hundreds of thousands of processes sat around doing nothing quite happily, so slow IO and such is something it's weirdly well-suited to. This is on my photography site so I'm less fussed about them harvesting my writing, and more about causing problems for image harvesters. I'm sure they'll get some stuff anyway but at least one bot got stuck in a 400 page hellhole earlier today so I'm overall very pleased :D
- mlaretallack 1mo agoI hate to ask but what's a "sexy toasters."
- robotmay 1mo agoGemini's surprisingly bad at it but so far I have some voluptuous curvy toasters, toasters in the shape of a butt, and robot girls with toasters for tits. I have a Gemini subscription for the month after mistakenly thinking I'd get cheap Opencode usage through it, so gotta use it for something.
- hahnbee 1mo agowhat constitutes as a bad crawler and what information do you have that's attracting so many bots?
- charles_f 1mo agoNot op, but a common tactic is to put honeypots in a page set to be forbidden in Robots.txt. Not respecting Robots.txt is making you a bad crawler. I have marginal blog, but its somewhat indexed and it gets tons of traffics from crawlers
- robotmay 1mo agoYep this is how I'm doing it. It's marked as my super exciting client archive full of high-res RAW photos that all bots are told not to index in robots.txt, only for some weird reason the link never gets rendered to the page :> I also explicitly tell all crawlers not to index the images on my site, actually, so I feel even better about serving the ones that do complete junk. Not sure why I get more crawler traffic than I do legitimate traffic tbh - it's a relatively new website that I've been trying to sort out the SEO for as I'm trying to get photography work, so it's probably started cropping up on the radar of scrapers looking for training data.
- solid_fuel 1mo agoHave you considered releasing any of this as a plug for phoenix or similar? I’ve also toyed with this approach and have a similar labyrinth on my website, but it just serves back huge amounts of meaningless text one word at a time.
- robotmay 1mo agoIf I get some time to un-LLM-ify it then sure! I build all my sites in Phoenix these days anyway so I'll likely want to port it to my other sites at some point too. I do want to add some more gibberish generation to mine. Once a visitor hits the honeypot route it does replace the rest of the site content with nonsense but I think it'd be fun to have it spewing really daft, really long LLM-generated essays on toasters.
- eloisius 1mo agoHow do you reliably detect the bots? Instead of slowban, I would feed it something abhorrent like an archive of 4chan posts.
- robotmay 1mo agoIt's mostly about luring them into URLs they're explicitly told in robots.txt that they shouldn't index. I do some identification via reverse DNS of known crawlers I actually want like Googlebot, though they respect robots.txt, in case something goes wrong and they accidentally get flagged.
- alok-g 1mo agoI had to look up what iocaine route is. Here's an article that explains well: https://firesphere.dev/articles/iocaine-the-deadliest-poison-known-to-ai https://firesphere.dev/articles/iocaine-the-deadliest-poison...
- akoboldfrying 1mo agoThat was somewhat helpful, thanks, but the most interesting aspect for me is how iocaine tries to determine whether a request comes from a bot (since if this can be done reliably, we're done) -- but on that, all the link says is that it "uses a set of logical rules".
- GoblinSlayer 1mo agoThat link shows me: --- Gymnastics trophy two years late, not two years wrote down on the other side of the two thousand millions. Stop all the time was now a. Anything of this real London, these actual civilized men and women; in uniform batches. The whole incident could not control the beating of a telescreen. Folly, folly, folly! He. ---
- whstl 1mo agoMy team runs a quite popular website, #1 or #2 in the market depending on the region. Several million visits per day. Around June/July we got a 10x boost out of nowhere, and it started affecting performance for users, increased hosting costs, and random bursts would bring the website down. We spent some time trying out solutions, from Cloudflare and Anubis to AWS, but it ended up affecting real users, and we got complaints on Reddit from day one, pretty much. At the same time, it got a bit expensive having a WAF. In the end the only solution that worked was blocking every single Chinese IP. Traffic instantly got down to 10x less, same as it was before. Same conversion percentage as before.
- robotmay 1mo agoIn an odd sequence of events, I've sorta got a weird tiny following in China due to having a vintage camera stall in an antiques market here in Wales that gets shared on Xiaohongshu (https://en.wikipedia.org/wiki/Xiaohongshu https://en.wikipedia.org/wiki/Xiaohongshu) sometimes when I have Chinese customers. So I'd feel bad blocking the entire country myself, though thankfully I'm not being inundated with traffic from there yet - the most egregious bots I've caught so far are actually ones from Meta which ignore robots.txt.
- whstl 1mo agoOh trust me, we felt really really bad blocking the whole country, and it took weeks of attempts avoiding it. Meta bots are also problematic, as they seem to be quite buggy. But the volume was nowhere near the hundreds of bots from China :/
- robotmay 1mo agoYeah Meta's bots are well and truly stuck in the blackhole on this site at the moment. I have no idea what they're supposed to be doing, but in case they are using them for AI work I've just added a token multipler based on subnet totals so they can have some more fun. I might come up with some other fun response types for them too.
- leohart 1mo ago
- huhtenberg 1mo agoIf you know the IP, it's easy to tar scraper's requests. However the article mentions that now they are being scraped 4-5 requests at a time from random IPs that are then never reused. It still probably possible to detect scrapers if they start by requesting obscure pages, but in general it's a much more difficult case.