6 ms·

I'm not a part of the Gemini community, but if I had to guess: * caching is like alcohol: it is both the cause of and solution to all of life's problems * not

I'm not a part of the Gemini community, but if I had to guess: * caching is like alcohol: it is both the cause of and solution to all of life's problems * nothing Gemini serves should be so large as to need resumable downloads or compression In other words they seem like deliberate choices to prevent web-app-ification of Gemini.

api 5y ago
Cryptographic content-addressability (with inherent data deduplication) is superior to simple caching in every single way: efficiency, facilitation of distributed storage, and security. A link should point to a hash. Now you can go get that hash from the same host you used to get the link or any other host that stores it anywhere in the world. That hash could unpack to data or to a list of other hashes. Go get those. Where you get them doesn't matter one bit. You can verify the correctness of the data trivially. This is how BitTorrent works but it could work equally well on top of a simpler protocol.
- selfhoster11 5y ago
  Would IPFS about cover this use case?
  - api 5y ago
    IPFS uses this, but it's more complex than necessary and has a ton of added features that are not core to the principle. (Not saying they're all bad... just that the core principle doesn't require thousands of lines of code to implement.) Here's the bottom line: The cryptographic content addressing I described sounds complex but is actually super simple and can be implemented in a page or two of code that will almost never need to change because it is "correct." It could be added to the gemini protocol in an afternoon. Caching sounds simple but is horrifically complex and an endless source of bugs and failure modes.
    - selfhoster11 5y ago
      > It could be added to the gemini protocol in an afternoon. I'd actually like this to be a separate protocol that's co-hosted with Gemini. It would make Gemini simpler, and it would enable other protocols to reuse content addressable storage.
    - c-cube 5y ago
      A gemini-like but designed around a global content-addressed structure would be a lot more interesting, imho. It doesn't even need to be tied to gemini. If you know how to do that then you should do it!
- nullc 5y ago
  Content hash based distribution has downsides too: E.g. updating content can result in people still stuck on old versions because the links weren't updated. Content hash distribution can also easily have privacy problems. Say there is a site for rubber band fetishists that you don't want people to know you've visited. I can interrogate if you've visited this site by embedding some of its content-- if you load it instantly because you have it cached, then I can be fairly confident that you've visited that site before.
  - api 5y ago
    The first point is no different from regular dead links or outdated URLs. The second can be mitigated but is a legitimate issue.
    - samatman 5y ago
      Unfortunately, stale hashes are a worse problem in content-centric networks than dead links. So Alice links to some content in Bob's site, but Bob made an embarrassing error and wants to update it. Very well, and Alice has to update the hash pointing to Bob's content: but now any hash pointing to Alice's content is stale, and so on, ad infinitum. There are ways to solve this problem, but none of them force a linear forward history. Basically, you can request all Merkle chains featuring a base hash, and verify that Bob signed all of the branches, but only discipline can keep this collection in a linear order. This is probably fine in practice, just follow the longest branch. And now you have Secure Scuttlebutt!
  - remram 5y ago
    This last attack is possible regardless of whether content is addressed by hash or not, it only needs caching.
- hakfoo 5y ago
  This seems like it would be vulnerable to accidental or malicious hash collision generation. You need a hash plus some "authorized mirrors list" or some other anti-poisoning measure. Not sure offhand how BitTorrent does it.
  - telotortium 5y ago
    Unless you are using an older hash function that doesn't have sufficient collision or preimage resistance, that should not be an issue, because finding 2 files that hash to the same output is infeasible. It does point to the need to be able to upgrade the hash function in your protocol at some point, and preferably associate a set of hashes generated by different hash functions to the same file contents.
  - derefr 5y ago
    Any hash algorithm for which a single collision has ever been generated would be an inappropriate choice for a URN system. Re: malicious collisions, any modern hash algorithm currently in wide use, if "broken" by some kind of vulnerability, would cause far worse problems for the world than just some URN system breaking down. As such, literal man-millenia of cryptanalysis has gone into ensuring that there are no vulnerabilities in popular algorithms. Re: accidental collisions, please note that 128 bits of entropy (i.e. a UUID) is already enough to ensure no accidental collisions before the heat-death of the universe. And most modern hashes have ≥256-bit residues.
    - ltbarcly3 5y ago
      FYI UUIDs are 128bits but only have 122 bits of entropy. Still not going to get a collision with a good implementation.
      - api 5y ago
        UUIDs are long enough but are not cryptographic hashes. They can be deliberately collided pretty easily. A cryptographic hash like sha384 or blake2 makes it infeasible to come up with any input that would ever yield the same output. That’s what you would use for such a network.
kstrauser 5y ago
I think you’re right, and I think those complaints are a little ridiculous for a protocol primarily designed for publishing text. I have a tiny Gemini site because it’s fun to play with, even if its audience is tiny. My average file size is about 2KB. There’s practically zero benefit for caching or compression here, and much detriment toward making the protocol more complicated to support them (again, given the context of primarily moving small text files around).
- fsiefken 5y ago
  yes, i just wrote about this on the Gemini mailing list. Even with small 2K files there is a ~50% reduction in size (and more if you use dictionary based compression). If you are on an extremely slow link at sea that can mean a few seconds of extra loading time, which is significant. Of course in reality, few people are in this situation. But if you are designing a protocol where the fun and perhaps also the goal is optimizing for getting textual information across as efficiently as possible within any constraints of conceivable computing resources it's something I'd would want to consider. A simple check if a gmi is binary and if so decompress it would solve such an issue of lone gemini users on the moon or at sea.
tenebrisalietum 5y ago
Nothing stopping the client from caching locally. I think Lagrange and possibly the other clients display last download time and allow a refresh.

api 5y ago

Cryptographic content-addressability (with inherent data deduplication) is superior to simple caching in every single way: efficiency, facilitation of distributed storage, and security. A link should point to a hash. Now you can go get that hash from the same host you used to get the link or any other host that stores it anywhere in the world. That hash could unpack to data or to a list of other hashes. Go get those. Where you get them doesn't matter one bit. You can verify the correctness of the data trivially. This is how BitTorrent works but it could work equally well on top of a simpler protocol.

selfhoster11 5y ago
Would IPFS about cover this use case?
- api 5y ago
  IPFS uses this, but it's more complex than necessary and has a ton of added features that are not core to the principle. (Not saying they're all bad... just that the core principle doesn't require thousands of lines of code to implement.) Here's the bottom line: The cryptographic content addressing I described sounds complex but is actually super simple and can be implemented in a page or two of code that will almost never need to change because it is "correct." It could be added to the gemini protocol in an afternoon. Caching sounds simple but is horrifically complex and an endless source of bugs and failure modes.
  - selfhoster11 5y ago
    > It could be added to the gemini protocol in an afternoon. I'd actually like this to be a separate protocol that's co-hosted with Gemini. It would make Gemini simpler, and it would enable other protocols to reuse content addressable storage.
  - c-cube 5y ago
    A gemini-like but designed around a global content-addressed structure would be a lot more interesting, imho. It doesn't even need to be tied to gemini. If you know how to do that then you should do it!
nullc 5y ago
Content hash based distribution has downsides too: E.g. updating content can result in people still stuck on old versions because the links weren't updated. Content hash distribution can also easily have privacy problems. Say there is a site for rubber band fetishists that you don't want people to know you've visited. I can interrogate if you've visited this site by embedding some of its content-- if you load it instantly because you have it cached, then I can be fairly confident that you've visited that site before.
- api 5y ago
  The first point is no different from regular dead links or outdated URLs. The second can be mitigated but is a legitimate issue.
  - samatman 5y ago
    Unfortunately, stale hashes are a worse problem in content-centric networks than dead links. So Alice links to some content in Bob's site, but Bob made an embarrassing error and wants to update it. Very well, and Alice has to update the hash pointing to Bob's content: but now any hash pointing to Alice's content is stale, and so on, ad infinitum. There are ways to solve this problem, but none of them force a linear forward history. Basically, you can request all Merkle chains featuring a base hash, and verify that Bob signed all of the branches, but only discipline can keep this collection in a linear order. This is probably fine in practice, just follow the longest branch. And now you have Secure Scuttlebutt!
- remram 5y ago
  This last attack is possible regardless of whether content is addressed by hash or not, it only needs caching.
hakfoo 5y ago
This seems like it would be vulnerable to accidental or malicious hash collision generation. You need a hash plus some "authorized mirrors list" or some other anti-poisoning measure. Not sure offhand how BitTorrent does it.
- telotortium 5y ago
  Unless you are using an older hash function that doesn't have sufficient collision or preimage resistance, that should not be an issue, because finding 2 files that hash to the same output is infeasible. It does point to the need to be able to upgrade the hash function in your protocol at some point, and preferably associate a set of hashes generated by different hash functions to the same file contents.
- derefr 5y ago
  Any hash algorithm for which a single collision has ever been generated would be an inappropriate choice for a URN system. Re: malicious collisions, any modern hash algorithm currently in wide use, if "broken" by some kind of vulnerability, would cause far worse problems for the world than just some URN system breaking down. As such, literal man-millenia of cryptanalysis has gone into ensuring that there are no vulnerabilities in popular algorithms. Re: accidental collisions, please note that 128 bits of entropy (i.e. a UUID) is already enough to ensure no accidental collisions before the heat-death of the universe. And most modern hashes have ≥256-bit residues.
  - ltbarcly3 5y ago
    FYI UUIDs are 128bits but only have 122 bits of entropy. Still not going to get a collision with a good implementation.
    - api 5y ago
      UUIDs are long enough but are not cryptographic hashes. They can be deliberately collided pretty easily. A cryptographic hash like sha384 or blake2 makes it infeasible to come up with any input that would ever yield the same output. That’s what you would use for such a network.

kstrauser 5y ago

I think you’re right, and I think those complaints are a little ridiculous for a protocol primarily designed for publishing text. I have a tiny Gemini site because it’s fun to play with, even if its audience is tiny. My average file size is about 2KB. There’s practically zero benefit for caching or compression here, and much detriment toward making the protocol more complicated to support them (again, given the context of primarily moving small text files around).

fsiefken 5y ago
yes, i just wrote about this on the Gemini mailing list. Even with small 2K files there is a ~50% reduction in size (and more if you use dictionary based compression). If you are on an extremely slow link at sea that can mean a few seconds of extra loading time, which is significant. Of course in reality, few people are in this situation. But if you are designing a protocol where the fun and perhaps also the goal is optimizing for getting textual information across as efficiently as possible within any constraints of conceivable computing resources it's something I'd would want to consider. A simple check if a gmi is binary and if so decompress it would solve such an issue of lone gemini users on the moon or at sea.

tenebrisalietum 5y ago

Nothing stopping the client from caching locally. I think Lagrange and possibly the other clients display last download time and allow a refresh.