12 ms·

I love the insanity of this idea. Not saying it's a good idea, but it's a very highly entertaining one, and I like that! I've also had enormous luck with Anubi

I love the insanity of this idea. Not saying it's a good idea, but it's a very highly entertaining one, and I like that! I've also had enormous luck with Anubis. AI scrapers found my personal Forgejo server and were hitting it on the order of 600K requests per day. After setting up Anubis, that dropped to about 100. Yes, some people are going to see an anime catgirl from time to time. Bummer. Reducing my fake traffic by a factor of 6,000 is worth it.

n1xis10t 11mo ago
*anime jackalgirl Also you mentioned Anubis, so it’s creator will probably read this. Hi Xena!
- kstrauser 11mo ago
  Correct; my bad! And hey, Xena! (And thank you very much!)
- ziml77 11mo ago
  I checked Xe's profile when I hadn't seen them post here for a while. According to that, they're not really using HN anymore.
  - n1xis10t 11mo ago
    See this thread from yesterday or so: https://news.ycombinator.com/item?id=46302496#46306025 https://news.ycombinator.com/item?id=46302496#46306025
- xena 11mo ago
  Ohai! I'm working on dataset poisoning. The early prototype generates vapid LinkedIn posts but future versions will be fully pluggable with WebAssembly.
  - n1xis10t 11mo ago
    That sounds fun, I look forward to reading a writeup about that
    - xena 11mo ago
      So I can plan it, how much detail do you want? Here's what I have about the prototype: https://anubis.techaro.lol/docs/admin/honeypot/overview https://anubis.techaro.lol/docs/admin/honeypot/overview
      - n1xis10t 11mo ago
        Probably any detail that you think is cool, I would be interested in reading about. When in doubt err on the side of too much detail. That was a good read. I hadn’t heard of spintax before, but I’ve thought of doing things like that. Also “pseudoprofound anti-content”, what a great term, that’s hilarious!
      - kstrauser 11mo ago
        As the owner of honeypot.net, I always appreciate seeing the name used as intended out in the wild.
      - 63stack 11mo ago
        This is amazing, I was just wondering about if it's possible to tie anubis together with iocaine, but it seems you already thought of that.
        xena 11mo ago
        It's slightly different in subtle ways. If I recall iocaine makes you configure a subprocess that it executes to generate garbage. One rule I have for Anubis in the code is that fork()/exec() are banned. So the pluggable garbage generator is gonna be powered by CGI handlers compiled to WebAssembly. It should be fun!
  - tommica 11mo ago
    Hi Xena! Your blog is amazing! Didn't realize you're working on Anubis - it's a really nice tool for the internet! Reminds me a bit of the ye' olde internet for some reason.
- ramonga 11mo ago
  what do people use to get keyword alerts in HN?
  - n1xis10t 11mo ago
    I think that most people don't do this, and the ones that do have custom solutions. Xena's uses cron, but that's all I know. It's probably a custom shell script.
- GaryBluto 11mo ago
  [dead]
n1xis10t 11mo ago
That’s so many scrapers. There must be a ton of companies with very large document collections at this point, and it really sucks that they don’t at least do us the courtesy of indexing them and making them available for keyword search, but instead only do AI. It’s kind of crazy how much scraping goes on and how little search engine development goes on. I guess search engines aren’t fashionable. Reminds me of this article about search engines disappearing mysteriously: https://archive.org/details/search-timeline https://archive.org/details/search-timeline I try to share that article as much as possible, it’s interesting.
- kstrauser 11mo ago
  So! Much! Scraping! They were downloading every commit multiple times, and fetching every file as seen at each of those commits, and trying to download archives of all the code, and hitting `/me/my-repo/blame` endpoints as their IP's first-ever request to my server, and other unlikely stuff. My scraper dudes, it's a git repo. You can fetch the whole freaking thing if you wanna look at it. Of course, that would require work and context-aware processing on their end, and it's easier for them to shift the expense onto my little server and make me pay for their misbehavior.
  - n1xis10t 11mo ago
    Crazy
- miki123211 11mo ago
  But there is a lot of search engine development going on, it's just that the results of the new search engines are fed straight into AI instead of displayed in the legacy 10-links-per-page view.
- PeterStuer 11mo ago
  Or some anti-ddos/bot companies using ultra cheap scraping services to annoy you enough to get you into their "free" anti bot protection, so they can charge the few real ai scrapers for access to your site.
anonymous908213 11mo ago
As someone on the browsing end, I love Anubis. I've only seen it a couple of times, but it sparks joy. It's rather refreshing compared to Cloudfare, which will usually make me immediately close the page and not bother with whatever content was behind it.
- kstrauser 11mo ago
  Same here, really. That's why I started using it. I'd seen it pop up for a moment on a few sites I'd visited, and it was so quirky and completely not disruptive that I didn't mind routing my legit users through it.
  - n1xis10t 11mo ago
    So maybe there are more people who like the “anime catgirl” than there are who think it’s weird
    - kstrauser 11mo ago
      *anime jackalgirl ;-) Quite possibly. Or, in my case, I think it's more quirky and fun than weird. It's non-zero amounts of weird, sure, but far below my threshold of troublesome. I probably wouldn't put my business behind it. I'm A-OK with using it on personal and hobby projects. Frankly, anyone so delicate that they freak out at the utterly anodyne imagery is someone I don't want to deal with in my personal time. I can only abide so much pearl clutching when I'm not getting paid for it.
      - Imustaskforhelp 11mo ago
        For what its worth, I think that a UN/(Unicef?) website (not sure which one) did use anubis so maybe you can put it behind businesses too :)
      - D-Machine 11mo ago
        The Digital Research Alliance of Canada (the main organization unifying and handling all the main HPC compute clusters in Canada) now uses Anubis for their wiki. Granted this is not a business, but still! https://docs.alliancecan.ca/wiki/Technical_documentation https://docs.alliancecan.ca/wiki/Technical_documentation
buu700 11mo ago
It's actually a well established concept: https://youtu.be/p9KeopXHcf8 https://youtu.be/p9KeopXHcf8
tonymet 11mo ago
[flagged]
- kstrauser 11mo ago
  Which cartoon are you referring to? The version of Anubis I installed only has the G-rated default images.
  - tonymet 11mo ago
    [flagged]
    - n1xis10t 11mo ago
      If you keep referring to non-explicit material as pornography, you will continue to confuse people. If you have an objection to the image other than it’s pornographic status, please word it clearly.
      - tonymet 11mo ago
        I was clear on the issue
    - kstrauser 11mo ago
      I'm being sincere here: I genuinely don't know what you're talking about. I'm referring to these default images: https://github.com/TecharoHQ/anubis/tree/main/docs/static/img/anubis https://github.com/TecharoHQ/anubis/tree/main/docs/static/im.... Do you mean something different?
      - tonymet 11mo ago
        Similar but yeah. Whatever prompts during the challenge . It’s creepy , out of context and inappropriate .
amypetrik8 11mo ago
>I love the insanity of this idea. Not saying it's a good idea, but it's a very highly entertaining one, and I like that! An even more insane idea -- minding the idea here is porn is radioactive to AI data training scrapers -- is there is something the powers that be view as far more disruptive and against community guidelineish than porn. And that would be wrongthink. The narratives. The historic narratives. The woke ideology. Anything related to an academic department whose field is <population subgroup> studies. Alls you need to do is plop in a little diatribe staunchly opposing any such enforced views and that AI bot will shoot away from your website and lightspeed
- lelanthran 11mo ago
  I like this better than of NSFW links; just include a (possible LLM generated) paragraph about not supporting transitions in minor children. Or perhaps that libraries that remove instructional booklets for how to have same-sex intercourse aren't actually banning the books. That sort of thing; nothing that 80% of people object to (so there's no problem if someone actually sees it), but something that definitely triggers the filters.
- GoblinSlayer 11mo ago
  I'm afraid AI bot and scraper are different things. Looks like poison is filtered after scraping no matter where it comes from, so there's no need to disable scraping you, because that's extra work.

n1xis10t 11mo ago

*anime jackalgirl Also you mentioned Anubis, so it’s creator will probably read this. Hi Xena!

kstrauser 11mo ago
Correct; my bad! And hey, Xena! (And thank you very much!)
ziml77 11mo ago
I checked Xe's profile when I hadn't seen them post here for a while. According to that, they're not really using HN anymore.
- n1xis10t 11mo ago
  See this thread from yesterday or so: https://news.ycombinator.com/item?id=46302496#46306025 https://news.ycombinator.com/item?id=46302496#46306025
xena 11mo ago
Ohai! I'm working on dataset poisoning. The early prototype generates vapid LinkedIn posts but future versions will be fully pluggable with WebAssembly.
- n1xis10t 11mo ago
  That sounds fun, I look forward to reading a writeup about that
  - xena 11mo ago
    So I can plan it, how much detail do you want? Here's what I have about the prototype: https://anubis.techaro.lol/docs/admin/honeypot/overview https://anubis.techaro.lol/docs/admin/honeypot/overview
    - n1xis10t 11mo ago
      Probably any detail that you think is cool, I would be interested in reading about. When in doubt err on the side of too much detail. That was a good read. I hadn’t heard of spintax before, but I’ve thought of doing things like that. Also “pseudoprofound anti-content”, what a great term, that’s hilarious!
    - kstrauser 11mo ago
      As the owner of honeypot.net, I always appreciate seeing the name used as intended out in the wild.
    - 63stack 11mo ago
      This is amazing, I was just wondering about if it's possible to tie anubis together with iocaine, but it seems you already thought of that.
      - xena 11mo ago
        It's slightly different in subtle ways. If I recall iocaine makes you configure a subprocess that it executes to generate garbage. One rule I have for Anubis in the code is that fork()/exec() are banned. So the pluggable garbage generator is gonna be powered by CGI handlers compiled to WebAssembly. It should be fun!
- tommica 11mo ago
  Hi Xena! Your blog is amazing! Didn't realize you're working on Anubis - it's a really nice tool for the internet! Reminds me a bit of the ye' olde internet for some reason.
ramonga 11mo ago
what do people use to get keyword alerts in HN?
- n1xis10t 11mo ago
  I think that most people don't do this, and the ones that do have custom solutions. Xena's uses cron, but that's all I know. It's probably a custom shell script.
GaryBluto 11mo ago
[dead]

n1xis10t 11mo ago

That’s so many scrapers. There must be a ton of companies with very large document collections at this point, and it really sucks that they don’t at least do us the courtesy of indexing them and making them available for keyword search, but instead only do AI. It’s kind of crazy how much scraping goes on and how little search engine development goes on. I guess search engines aren’t fashionable. Reminds me of this article about search engines disappearing mysteriously: https://archive.org/details/search-timeline https://archive.org/details/search-timeline I try to share that article as much as possible, it’s interesting.

kstrauser 11mo ago
So! Much! Scraping! They were downloading every commit multiple times, and fetching every file as seen at each of those commits, and trying to download archives of all the code, and hitting `/me/my-repo/blame` endpoints as their IP's first-ever request to my server, and other unlikely stuff. My scraper dudes, it's a git repo. You can fetch the whole freaking thing if you wanna look at it. Of course, that would require work and context-aware processing on their end, and it's easier for them to shift the expense onto my little server and make me pay for their misbehavior.
- n1xis10t 11mo ago
  Crazy
miki123211 11mo ago
But there is a lot of search engine development going on, it's just that the results of the new search engines are fed straight into AI instead of displayed in the legacy 10-links-per-page view.
PeterStuer 11mo ago
Or some anti-ddos/bot companies using ultra cheap scraping services to annoy you enough to get you into their "free" anti bot protection, so they can charge the few real ai scrapers for access to your site.

anonymous908213 11mo ago

As someone on the browsing end, I love Anubis. I've only seen it a couple of times, but it sparks joy. It's rather refreshing compared to Cloudfare, which will usually make me immediately close the page and not bother with whatever content was behind it.

kstrauser 11mo ago
Same here, really. That's why I started using it. I'd seen it pop up for a moment on a few sites I'd visited, and it was so quirky and completely not disruptive that I didn't mind routing my legit users through it.
- n1xis10t 11mo ago
  So maybe there are more people who like the “anime catgirl” than there are who think it’s weird
  - kstrauser 11mo ago
    *anime jackalgirl ;-) Quite possibly. Or, in my case, I think it's more quirky and fun than weird. It's non-zero amounts of weird, sure, but far below my threshold of troublesome. I probably wouldn't put my business behind it. I'm A-OK with using it on personal and hobby projects. Frankly, anyone so delicate that they freak out at the utterly anodyne imagery is someone I don't want to deal with in my personal time. I can only abide so much pearl clutching when I'm not getting paid for it.
    - Imustaskforhelp 11mo ago
      For what its worth, I think that a UN/(Unicef?) website (not sure which one) did use anubis so maybe you can put it behind businesses too :)
    - D-Machine 11mo ago
      The Digital Research Alliance of Canada (the main organization unifying and handling all the main HPC compute clusters in Canada) now uses Anubis for their wiki. Granted this is not a business, but still! https://docs.alliancecan.ca/wiki/Technical_documentation https://docs.alliancecan.ca/wiki/Technical_documentation

buu700 11mo ago

It's actually a well established concept: https://youtu.be/p9KeopXHcf8 https://youtu.be/p9KeopXHcf8

tonymet 11mo ago

[flagged]

kstrauser 11mo ago
Which cartoon are you referring to? The version of Anubis I installed only has the G-rated default images.
- tonymet 11mo ago
  [flagged]
  - n1xis10t 11mo ago
    If you keep referring to non-explicit material as pornography, you will continue to confuse people. If you have an objection to the image other than it’s pornographic status, please word it clearly.
    - tonymet 11mo ago
      I was clear on the issue
  - kstrauser 11mo ago
    I'm being sincere here: I genuinely don't know what you're talking about. I'm referring to these default images: https://github.com/TecharoHQ/anubis/tree/main/docs/static/img/anubis https://github.com/TecharoHQ/anubis/tree/main/docs/static/im.... Do you mean something different?
    - tonymet 11mo ago
      Similar but yeah. Whatever prompts during the challenge . It’s creepy , out of context and inappropriate .

amypetrik8 11mo ago

>I love the insanity of this idea. Not saying it's a good idea, but it's a very highly entertaining one, and I like that! An even more insane idea -- minding the idea here is porn is radioactive to AI data training scrapers -- is there is something the powers that be view as far more disruptive and against community guidelineish than porn. And that would be wrongthink. The narratives. The historic narratives. The woke ideology. Anything related to an academic department whose field is <population subgroup> studies. Alls you need to do is plop in a little diatribe staunchly opposing any such enforced views and that AI bot will shoot away from your website and lightspeed

lelanthran 11mo ago
I like this better than of NSFW links; just include a (possible LLM generated) paragraph about not supporting transitions in minor children. Or perhaps that libraries that remove instructional booklets for how to have same-sex intercourse aren't actually banning the books. That sort of thing; nothing that 80% of people object to (so there's no problem if someone actually sees it), but something that definitely triggers the filters.
GoblinSlayer 11mo ago
I'm afraid AI bot and scraper are different things. Looks like poison is filtered after scraping no matter where it comes from, so there's no need to disable scraping you, because that's extra work.