3 ms·

Try Anubis: <https://anubis.techaro.lol https://anubis.techaro.lol> It's a reverse proxy that presents a PoC challenge to every new visitor. It shifts the init

Try Anubis: <https://anubis.techaro.lol https://anubis.techaro.lol> It's a reverse proxy that presents a PoC challenge to every new visitor. It shifts the initial cost of accessing your server's resources back at the client. Assuming your uplink can handle 300k clients requesting a single 70kb web page, it should solve most of your problems. For science, can you estimate your peak QPS?

marginalia_nu 1y ago
Anubis is a good choice because it whitelists legitimate and well behaved crawlers based on IP + user-agent. Cloudflare works as well in that regard but then you're MITM:ing all your visitors.
Imustaskforhelp 1y ago
Also, I was just watching brodie robertson video about how United Nations has this random search page of unesco which actually has anubis. Crazy how I remember the HN post where anubis's blog post was first made. Though, I always thought it was a bit funny with anime and it was made by frustration of (I think AWS? AI scrapers who won't follow general rules and it was constantly giving requests to his git server and it actually made his git server down I guess??) I didn't expect it to blow up to ... UN.
- xena 1y ago
  Her* It was frustration at AWS' Alexa team and their abuse of the commons. Amusingly if they had replied to my email before I wrote my shitpost of an implementation this all could have turned out vastly differently.
  - Imustaskforhelp 1y ago
    Oh I am so so sorry I didn't see your gender and assumed it to be a (he). { really sorry about that once again} Also didn't expect you to respond to my comment xD I went through the slow realization of while reading this comment that you are the creator of anubis and I had such a smile when I realized that you commented to me. Also, this project is really nice, but I actually want to ask, I haven't read the docs of anubis but could it be that the proof of work isn't wasted / it can be used for something (I know I might get downvoted because I am going to mention cryptocurrency, but nano currency has a proof of work required for each transaction, so if anubis actually does the proof of work as by nano standards, then theoretically that proof of work could atleast be some useful) Looking forward to your comment!
    - akaij 1y ago
      Useful as a for-profit cryptocurrency? I think zero chance. The only way I see anything like that incorporated is a folding@home kind of thing that could help humanity as a whole. Of course, if someone makes it work like you suggested, and it catches on, I will personally haunt your dreams forever. Don't give them any ideas.
martin82 1y ago
This looks very cool, but isn't it just a matter of months until all scrapers get updated and can easily beat this challenge and are able to compute modern JS stuff?

marginalia_nu 1y ago

Anubis is a good choice because it whitelists legitimate and well behaved crawlers based on IP + user-agent. Cloudflare works as well in that regard but then you're MITM:ing all your visitors.

Imustaskforhelp 1y ago

Also, I was just watching brodie robertson video about how United Nations has this random search page of unesco which actually has anubis. Crazy how I remember the HN post where anubis's blog post was first made. Though, I always thought it was a bit funny with anime and it was made by frustration of (I think AWS? AI scrapers who won't follow general rules and it was constantly giving requests to his git server and it actually made his git server down I guess??) I didn't expect it to blow up to ... UN.

xena 1y ago
Her* It was frustration at AWS' Alexa team and their abuse of the commons. Amusingly if they had replied to my email before I wrote my shitpost of an implementation this all could have turned out vastly differently.
- Imustaskforhelp 1y ago
  Oh I am so so sorry I didn't see your gender and assumed it to be a (he). { really sorry about that once again} Also didn't expect you to respond to my comment xD I went through the slow realization of while reading this comment that you are the creator of anubis and I had such a smile when I realized that you commented to me. Also, this project is really nice, but I actually want to ask, I haven't read the docs of anubis but could it be that the proof of work isn't wasted / it can be used for something (I know I might get downvoted because I am going to mention cryptocurrency, but nano currency has a proof of work required for each transaction, so if anubis actually does the proof of work as by nano standards, then theoretically that proof of work could atleast be some useful) Looking forward to your comment!
  - akaij 1y ago
    Useful as a for-profit cryptocurrency? I think zero chance. The only way I see anything like that incorporated is a folding@home kind of thing that could help humanity as a whole. Of course, if someone makes it work like you suggested, and it catches on, I will personally haunt your dreams forever. Don't give them any ideas.

martin82 1y ago

This looks very cool, but isn't it just a matter of months until all scrapers get updated and can easily beat this challenge and are able to compute modern JS stuff?