3 ms·

It's really surreal to see my project in the preview image like this. That's wild! If you want to try it: https://github.com/TecharoHQ/anubis https://github.com

by xena 2y ago

It's really surreal to see my project in the preview image like this. That's wild! If you want to try it: https://github.com/TecharoHQ/anubis https://github.com/TecharoHQ/anubis. So far I've noticed that it seems to actually work. I just deployed it to xeiaso.net as a way to see how it fails in prod for my blog.

diggan 2y ago
Nice work :) One piece of feedback: Could you add some explanation (for humans) what we're supposed to do and what is happening when met by that page? I know there is a loading animation widget thingy, but the first time I saw that page (some weeks ago at the Gnome issue tracker), it was proof-of-work'ing for like 20 seconds, and I wasn't sure what was going on, I initially thought I got blocked or that the captcha failed to load. Of course, now I understand what it is, but I'm not sure it's 100% clear when you just see the "checking if you're a bot" page in isolation.
- xena 2y ago
  > One piece of feedback: Could you add some explanation (for humans) what we're supposed to do and what is happening when met by that page? Will do! https://github.com/TecharoHQ/anubis/issues/25 https://github.com/TecharoHQ/anubis/issues/25
  - ranger_danger 2y ago
    also if you're using JShelter, which blocks Worker by default, there is no indication that it's never going to work, and the spinner just goes on forever doing nothing
    - xena 2y ago
      Noted! I filed a bug: https://github.com/TecharoHQ/anubis/issues/38 https://github.com/TecharoHQ/anubis/issues/38 All of this is placeholder wording, layouts, CSS, and more. It'll be fixed in time. This is teething pain that I will get through.
- hartator 2y ago
  Maybe a progress bar?
  - xena 2y ago
    There's no way to really make a progress bar make sense, it's a luck-based mechanic.
    - diggan 2y ago
      Maybe one of those (slightly misleading) progressbars that have a dynamic speed that gets slower and slower the closer to the finish it gets? Just to indicate that it's working towards something
    - isoprophlex 2y ago
      It'll be somewhat involved, but based on the difficulty vs the clients hashing speed you could say something probabilistic like "90% of the time, this window will be gone in xyz seconds from now"?
      - xena 2y ago
        Yeah, I have to get the data for that though! I'm gonna add that to the list.
    - blibble 2y ago
      more, easier proof of works and the law of large numbers will do the rest
      - yifanl 2y ago
        That's multiplying the work the server has to do by a large number so it can show a nicer progress bar. Seems very counter to the purpose.
    - wink 2y ago
      So just like the windows copy dialog. Progress bar it is.
clvx 2y ago
I really like this. I don't mind Internet acting like the Wild Wild West but I do mind there's no accountability. This is a nice way to pass the economic burden to the crawlers for sites who still want to stay freely available. You want the data, spend money on your side to get it. Even though the downside is your site could be delisted from search engines, there's no reason why you cannot register your service in a global or p2p indexer.
- lukan 2y ago
  "why you cannot register your service in a global or p2p indexer" Network effects anyone? So yes, we should work on a different way of indexing the web again, than via google, but easier said than done I think ..
isoprophlex 2y ago
Loving it, great work as always. Also > https://news.ycombinator.com/item?id=43422781 https://news.ycombinator.com/item?id=43422781 Integrate a way to calculate micro-amounts of the shitcoin of your choice and we might have the another actually legitimately useful application of cryptocurrencies on our hands..!
vhcr 2y ago
Anubis is only going to work as long as it doesn't gets famous, if that happens crawlers will start using GPUs / ASICs for the proof of work and it's game over.
- xena 2y ago
  Author of Anubis here. If that happens, I win.
  - eb0la 2y ago
    If that happens, count with me to use Anubis to factor large primes or whatever science needs as a background task.
    - enrico204 2y ago
      Actually, that is not a bad idea. @xena maybe Anubis v2 could make the client participate in some sort of SETI@HOME project, creating the biggest distributed cluster ever created :-D
      - programd 2y ago
        Oh come now, clearly Anubis should make the clients mine bitcoin as proof of work, with a split for the website and the author. Oh dear, somebody is going to implement this in about an hour, aren't they....
        clvx 2y ago
        a service that allows you expose and host your data in a private manner getting a cut from whatever token your endpoints have generated.
        grotorea 2y ago
        Just in case you didn't know, cryptominers in Javascript are already thing. Firefox even blocks them.
reginald78 2y ago
Maybe I'm missing something, but doesn't this mean the work has to be done by the client AND the server every time a challenge is issued? I think ideally you'd want work that was easy for the server and difficult for the server. And what is to stop being DDoS'd by clients that are challenged but neglect to perform the challenge? Regardless, I think something like this is the way forward if one doesn't want to throw privacy entirely out the window. client
- diggan 2y ago
  > I think ideally you'd want work that was easy for the server and difficult for the server. That's exactly how it works (easy for server, hard for client). Once the client completed the Proof-of-Work challenge, the server doesn't need to complete the same challenge, it only needs to validate that the results checks out. Similar to how in Proof-of-Work blockchains where coming up with the block hashes is difficult, but validating them isn't nearly as compute-intensive. This asymmetric computation requirement is probably the most fundamental property of Proof-of-Work, Wikipedia has more details if you're curious: https://en.wikipedia.org/wiki/Proof_of_work https://en.wikipedia.org/wiki/Proof_of_work Fun fact: it seems Proof-of-Work was used as a DoS preventing technique before it was used in Bitcoin/blockchains, so seems we've gone full circle :)
  - namaria 2y ago
    I think going full circle would be something like bitcoin being created on top of DoS prevention software and then eventually DoS prevention starting to use bitcoin. A tool being used for something than something else than the first something again is just... nothing? Happens all the time?
- xena 2y ago
  The magic of proof of work is that it's something that's really hard to do but easy to validate. Anubis' proof of work works like this: A sha256 hash is a bunch of bytes like this: 394d1cc82924c2368d4e34fa450c6b30d5d02f8ae4bb6310e2296593008ff89f We usually write it out in hex form, but that's literally what the bytes in ram look like. In a proof of work validation system, you take some base value (the "challenge") and a rapidly incrementing number (the "nonce"), so the thing you end up hashing is this: await sha256(`${challenge}${nonce}`); The "difficulty" is how many leading zeroes the generated hash needs to have. When a client requests to pass the challenge, they include the nonce they used. The server then only has to do one sha256 operation: the one that confirms that the challenge (generated from request metadata) and the nonce (provided by the client) match the difficulty number of leading zeroes. The other trick is that presenting the challenge page is super cheap. I wrote that page with templ (https://templ.guide https://templ.guide) so it compiles to native Go. This makes it as optimized as Go is modulo things like variable replacement. If this becomes a problem I plan to prerender things as much as possible. Rendering the challenge page from binary code or ram is always always always going to be so much cheaper than your webapp ever will be. I'm planning on adding things like changing out the hash in use, but right now sha256 is the best option because most CPUs in active deployment have instructions to accelerate sha256 hashing. This combined with webcrypto jumping to heavily optimized C++ and the JIT in JS being shockingly good means that this super naïve approach is probably the most efficient way to do things right now. I'm shocked that this all works so well and I'm so glad to see it take off like it has.
knowaveragejoe 2y ago
I love that I seem to stumble upon something by you randomly every so often. I'd just like to say that I enjoy your approach to explanations in blog form and will look further into Anubis!
GaggiX 2y ago
The AI anime girl has 6 fingers btw, combating AI bot with AI girls. Edit: I will probably send a pull request to fix it.
- xena 2y ago
  I'm commissioning an artist to make better assets. These are the placeholders that I used with the original rageware implementation. I never thought it would take off like this!
deleted 2y ago
[deleted]
pabs3 2y ago
Could you add an option for non-JS users? Maybe a Linux command-line we can paste the output of into a form.

by xena 2y ago

diggan 2y ago

Nice work :) One piece of feedback: Could you add some explanation (for humans) what we're supposed to do and what is happening when met by that page? I know there is a loading animation widget thingy, but the first time I saw that page (some weeks ago at the Gnome issue tracker), it was proof-of-work'ing for like 20 seconds, and I wasn't sure what was going on, I initially thought I got blocked or that the captcha failed to load. Of course, now I understand what it is, but I'm not sure it's 100% clear when you just see the "checking if you're a bot" page in isolation.

xena 2y ago
> One piece of feedback: Could you add some explanation (for humans) what we're supposed to do and what is happening when met by that page? Will do! https://github.com/TecharoHQ/anubis/issues/25 https://github.com/TecharoHQ/anubis/issues/25
- ranger_danger 2y ago
  also if you're using JShelter, which blocks Worker by default, there is no indication that it's never going to work, and the spinner just goes on forever doing nothing
  - xena 2y ago
    Noted! I filed a bug: https://github.com/TecharoHQ/anubis/issues/38 https://github.com/TecharoHQ/anubis/issues/38 All of this is placeholder wording, layouts, CSS, and more. It'll be fixed in time. This is teething pain that I will get through.
hartator 2y ago
Maybe a progress bar?
- xena 2y ago
  There's no way to really make a progress bar make sense, it's a luck-based mechanic.
  - diggan 2y ago
    Maybe one of those (slightly misleading) progressbars that have a dynamic speed that gets slower and slower the closer to the finish it gets? Just to indicate that it's working towards something
  - isoprophlex 2y ago
    It'll be somewhat involved, but based on the difficulty vs the clients hashing speed you could say something probabilistic like "90% of the time, this window will be gone in xyz seconds from now"?
    - xena 2y ago
      Yeah, I have to get the data for that though! I'm gonna add that to the list.
  - blibble 2y ago
    more, easier proof of works and the law of large numbers will do the rest
    - yifanl 2y ago
      That's multiplying the work the server has to do by a large number so it can show a nicer progress bar. Seems very counter to the purpose.
  - wink 2y ago
    So just like the windows copy dialog. Progress bar it is.

clvx 2y ago

I really like this. I don't mind Internet acting like the Wild Wild West but I do mind there's no accountability. This is a nice way to pass the economic burden to the crawlers for sites who still want to stay freely available. You want the data, spend money on your side to get it. Even though the downside is your site could be delisted from search engines, there's no reason why you cannot register your service in a global or p2p indexer.

lukan 2y ago
"why you cannot register your service in a global or p2p indexer" Network effects anyone? So yes, we should work on a different way of indexing the web again, than via google, but easier said than done I think ..

isoprophlex 2y ago

Loving it, great work as always. Also > https://news.ycombinator.com/item?id=43422781 https://news.ycombinator.com/item?id=43422781 Integrate a way to calculate micro-amounts of the shitcoin of your choice and we might have the another actually legitimately useful application of cryptocurrencies on our hands..!

vhcr 2y ago

Anubis is only going to work as long as it doesn't gets famous, if that happens crawlers will start using GPUs / ASICs for the proof of work and it's game over.

xena 2y ago
Author of Anubis here. If that happens, I win.
- eb0la 2y ago
  If that happens, count with me to use Anubis to factor large primes or whatever science needs as a background task.
  - enrico204 2y ago
    Actually, that is not a bad idea. @xena maybe Anubis v2 could make the client participate in some sort of SETI@HOME project, creating the biggest distributed cluster ever created :-D
    - programd 2y ago
      Oh come now, clearly Anubis should make the clients mine bitcoin as proof of work, with a split for the website and the author. Oh dear, somebody is going to implement this in about an hour, aren't they....
      - clvx 2y ago
        a service that allows you expose and host your data in a private manner getting a cut from whatever token your endpoints have generated.
      - grotorea 2y ago
        Just in case you didn't know, cryptominers in Javascript are already thing. Firefox even blocks them.

reginald78 2y ago

Maybe I'm missing something, but doesn't this mean the work has to be done by the client AND the server every time a challenge is issued? I think ideally you'd want work that was easy for the server and difficult for the server. And what is to stop being DDoS'd by clients that are challenged but neglect to perform the challenge? Regardless, I think something like this is the way forward if one doesn't want to throw privacy entirely out the window. client

diggan 2y ago
> I think ideally you'd want work that was easy for the server and difficult for the server. That's exactly how it works (easy for server, hard for client). Once the client completed the Proof-of-Work challenge, the server doesn't need to complete the same challenge, it only needs to validate that the results checks out. Similar to how in Proof-of-Work blockchains where coming up with the block hashes is difficult, but validating them isn't nearly as compute-intensive. This asymmetric computation requirement is probably the most fundamental property of Proof-of-Work, Wikipedia has more details if you're curious: https://en.wikipedia.org/wiki/Proof_of_work https://en.wikipedia.org/wiki/Proof_of_work Fun fact: it seems Proof-of-Work was used as a DoS preventing technique before it was used in Bitcoin/blockchains, so seems we've gone full circle :)
- namaria 2y ago
  I think going full circle would be something like bitcoin being created on top of DoS prevention software and then eventually DoS prevention starting to use bitcoin. A tool being used for something than something else than the first something again is just... nothing? Happens all the time?
xena 2y ago
The magic of proof of work is that it's something that's really hard to do but easy to validate. Anubis' proof of work works like this: A sha256 hash is a bunch of bytes like this: 394d1cc82924c2368d4e34fa450c6b30d5d02f8ae4bb6310e2296593008ff89f We usually write it out in hex form, but that's literally what the bytes in ram look like. In a proof of work validation system, you take some base value (the "challenge") and a rapidly incrementing number (the "nonce"), so the thing you end up hashing is this: await sha256(`${challenge}${nonce}`); The "difficulty" is how many leading zeroes the generated hash needs to have. When a client requests to pass the challenge, they include the nonce they used. The server then only has to do one sha256 operation: the one that confirms that the challenge (generated from request metadata) and the nonce (provided by the client) match the difficulty number of leading zeroes. The other trick is that presenting the challenge page is super cheap. I wrote that page with templ (https://templ.guide https://templ.guide) so it compiles to native Go. This makes it as optimized as Go is modulo things like variable replacement. If this becomes a problem I plan to prerender things as much as possible. Rendering the challenge page from binary code or ram is always always always going to be so much cheaper than your webapp ever will be. I'm planning on adding things like changing out the hash in use, but right now sha256 is the best option because most CPUs in active deployment have instructions to accelerate sha256 hashing. This combined with webcrypto jumping to heavily optimized C++ and the JIT in JS being shockingly good means that this super naïve approach is probably the most efficient way to do things right now. I'm shocked that this all works so well and I'm so glad to see it take off like it has.

knowaveragejoe 2y ago

I love that I seem to stumble upon something by you randomly every so often. I'd just like to say that I enjoy your approach to explanations in blog form and will look further into Anubis!

GaggiX 2y ago

The AI anime girl has 6 fingers btw, combating AI bot with AI girls. Edit: I will probably send a pull request to fix it.

xena 2y ago
I'm commissioning an artist to make better assets. These are the placeholders that I used with the original rageware implementation. I never thought it would take off like this!

deleted 2y ago

[deleted]

pabs3 2y ago

Could you add an option for non-JS users? Maybe a Linux command-line we can paste the output of into a form.