3 ms·

I have been speculating on adding a tar pit on my personal web site. A script that produces a page of random nonsense and random looking links to the same scrip

I have been speculating on adding a tar pit on my personal web site. A script that produces a page of random nonsense and random looking links to the same script. The thing not linked to anywhere, but explicitly forbidden on robots.txt. If the crawlers start on it let them get lost. Bit of rate limiting should keep my server safe, and slow down the crawlers. Maybe I should add some confusing prompts on the page as well... Probably I never get around to it, but the idea sounds tempting.

reactordev 1y ago
I did something similar. On a normal browser it just displays the matrix rain effect. For a bot, it's a page of links on links to pages that link to each other using a clever php script and .htaccess fun. The fun part is watching the logs to see how long they get stuck for. As each link is unique and can build a tree structure several GB deep on my server. I did this once before with an ssh honey pot on my Mesos cluster in 2017.
shakna 1y ago
I have a single <a> element in my website's head, to a route banned by robots and the page is also marked by noindex meta tags and http headers. When something grabs it, which AI crawlers regularly do, it feeds them the text of 1984, about a sentence per minute. Most crawlers stay on the line for about four hours.
- dbalatero 1y ago
  That's hilarious, can I steal the source for my own site?
  - _moof 1y ago
    Only if you aren't a crawler.
    - E39M5S62 1y ago
      This is a long shot, but are you the same moof that ran the bot 'regurg' on EFnet in the late 90's / early 2000's for the BeOS community?
  - anileated 1y ago
    That’s what LLM would say…
J_McQuade 1y ago
I loved reading about something similar that popped up on HN a wee while back: https://zadzmo.org/code/nepenthes/ https://zadzmo.org/code/nepenthes/
- fbunnies 1y ago
  I loved reading about something dissimilar that did not pop up on HN yet: https://apnews.com/article/rabbits-with-horns-virus-colorado-tentacles-papillomavirus-98b1ad95ba3a0f308bf884d79d1eea7c https://apnews.com/article/rabbits-with-horns-virus-colorado...
phyzome 1y ago
Should be possible to do this with a static site, even. Here's what I've been doing so far: https://www.brainonfire.net/blog/2024/09/19/poisoning-ai-scrapers/ https://www.brainonfire.net/blog/2024/09/19/poisoning-ai-scr... (serving scrambled versions of my posts to LLM scrapers)
gleenn 1y ago
Check out doing a compression bomb too, you can host a very small file for you that uncompresses into a massive file for crawlers and hopefully runs them out of ram and they die. Someone posted about it recently on HN even but I can't immediately find the link
- extraduder_ire 1y ago
  It's either this one https://news.ycombinator.com/item?id=44670319 https://news.ycombinator.com/item?id=44670319 or the comments from this one https://news.ycombinator.com/item?id=44651536 https://news.ycombinator.com/item?id=44651536 I also recall reading it. I think wasting their time is more effective than making them crash and give up in this instance though.
xyzal 1y ago
Or, serve "Emergent Misalignment" dataset. https://github.com/emergent-misalignment/emergent-misalignment https://github.com/emergent-misalignment/emergent-misalignme...

reactordev 1y ago

I did something similar. On a normal browser it just displays the matrix rain effect. For a bot, it's a page of links on links to pages that link to each other using a clever php script and .htaccess fun. The fun part is watching the logs to see how long they get stuck for. As each link is unique and can build a tree structure several GB deep on my server. I did this once before with an ssh honey pot on my Mesos cluster in 2017.

shakna 1y ago

I have a single <a> element in my website's head, to a route banned by robots and the page is also marked by noindex meta tags and http headers. When something grabs it, which AI crawlers regularly do, it feeds them the text of 1984, about a sentence per minute. Most crawlers stay on the line for about four hours.

dbalatero 1y ago
That's hilarious, can I steal the source for my own site?
- _moof 1y ago
  Only if you aren't a crawler.
  - E39M5S62 1y ago
    This is a long shot, but are you the same moof that ran the bot 'regurg' on EFnet in the late 90's / early 2000's for the BeOS community?
- anileated 1y ago
  That’s what LLM would say…

J_McQuade 1y ago

I loved reading about something similar that popped up on HN a wee while back: https://zadzmo.org/code/nepenthes/ https://zadzmo.org/code/nepenthes/

fbunnies 1y ago
I loved reading about something dissimilar that did not pop up on HN yet: https://apnews.com/article/rabbits-with-horns-virus-colorado-tentacles-papillomavirus-98b1ad95ba3a0f308bf884d79d1eea7c https://apnews.com/article/rabbits-with-horns-virus-colorado...

phyzome 1y ago

Should be possible to do this with a static site, even. Here's what I've been doing so far: https://www.brainonfire.net/blog/2024/09/19/poisoning-ai-scrapers/ https://www.brainonfire.net/blog/2024/09/19/poisoning-ai-scr... (serving scrambled versions of my posts to LLM scrapers)

gleenn 1y ago

Check out doing a compression bomb too, you can host a very small file for you that uncompresses into a massive file for crawlers and hopefully runs them out of ram and they die. Someone posted about it recently on HN even but I can't immediately find the link

extraduder_ire 1y ago
It's either this one https://news.ycombinator.com/item?id=44670319 https://news.ycombinator.com/item?id=44670319 or the comments from this one https://news.ycombinator.com/item?id=44651536 https://news.ycombinator.com/item?id=44651536 I also recall reading it. I think wasting their time is more effective than making them crash and give up in this instance though.

xyzal 1y ago

Or, serve "Emergent Misalignment" dataset. https://github.com/emergent-misalignment/emergent-misalignment https://github.com/emergent-misalignment/emergent-misalignme...