4 ms·
Cryptographic content-addressability (with inherent data deduplication) is superior to simple caching in every single way: efficiency, facilitation of distribut
by api 5y ago
Cryptographic content-addressability (with inherent data deduplication) is superior to simple caching in every single way: efficiency, facilitation of distributed storage, and security.
A link should point to a hash. Now you can go get that hash from the same host you used to get the link or any other host that stores it anywhere in the world. That hash could unpack to data or to a list of other hashes. Go get those. Where you get them doesn't matter one bit. You can verify the correctness of the data trivially.
This is how BitTorrent works but it could work equally well on top of a simpler protocol.
- selfhoster11 5y agoWould IPFS about cover this use case?
- api 5y agoIPFS uses this, but it's more complex than necessary and has a ton of added features that are not core to the principle. (Not saying they're all bad... just that the core principle doesn't require thousands of lines of code to implement.) Here's the bottom line: The cryptographic content addressing I described sounds complex but is actually super simple and can be implemented in a page or two of code that will almost never need to change because it is "correct." It could be added to the gemini protocol in an afternoon. Caching sounds simple but is horrifically complex and an endless source of bugs and failure modes.
- selfhoster11 5y ago> It could be added to the gemini protocol in an afternoon. I'd actually like this to be a separate protocol that's co-hosted with Gemini. It would make Gemini simpler, and it would enable other protocols to reuse content addressable storage.
- c-cube 5y agoA gemini-like but designed around a global content-addressed structure would be a lot more interesting, imho. It doesn't even need to be tied to gemini. If you know how to do that then you should do it!
- nullc 5y agoContent hash based distribution has downsides too: E.g. updating content can result in people still stuck on old versions because the links weren't updated. Content hash distribution can also easily have privacy problems. Say there is a site for rubber band fetishists that you don't want people to know you've visited. I can interrogate if you've visited this site by embedding some of its content-- if you load it instantly because you have it cached, then I can be fairly confident that you've visited that site before.
- api 5y agoThe first point is no different from regular dead links or outdated URLs. The second can be mitigated but is a legitimate issue.
- samatman 5y agoUnfortunately, stale hashes are a worse problem in content-centric networks than dead links. So Alice links to some content in Bob's site, but Bob made an embarrassing error and wants to update it. Very well, and Alice has to update the hash pointing to Bob's content: but now any hash pointing to Alice's content is stale, and so on, ad infinitum. There are ways to solve this problem, but none of them force a linear forward history. Basically, you can request all Merkle chains featuring a base hash, and verify that Bob signed all of the branches, but only discipline can keep this collection in a linear order. This is probably fine in practice, just follow the longest branch. And now you have Secure Scuttlebutt!
- remram 5y agoThis last attack is possible regardless of whether content is addressed by hash or not, it only needs caching.
- hakfoo 5y agoThis seems like it would be vulnerable to accidental or malicious hash collision generation. You need a hash plus some "authorized mirrors list" or some other anti-poisoning measure. Not sure offhand how BitTorrent does it.
- telotortium 5y agoUnless you are using an older hash function that doesn't have sufficient collision or preimage resistance, that should not be an issue, because finding 2 files that hash to the same output is infeasible. It does point to the need to be able to upgrade the hash function in your protocol at some point, and preferably associate a set of hashes generated by different hash functions to the same file contents.
- derefr 5y agoAny hash algorithm for which a single collision has ever been generated would be an inappropriate choice for a URN system. Re: malicious collisions, any modern hash algorithm currently in wide use, if "broken" by some kind of vulnerability, would cause far worse problems for the world than just some URN system breaking down. As such, literal man-millenia of cryptanalysis has gone into ensuring that there are no vulnerabilities in popular algorithms. Re: accidental collisions, please note that 128 bits of entropy (i.e. a UUID) is already enough to ensure no accidental collisions before the heat-death of the universe. And most modern hashes have ≥256-bit residues.
- ltbarcly3 5y agoFYI UUIDs are 128bits but only have 122 bits of entropy. Still not going to get a collision with a good implementation.
- api 5y agoUUIDs are long enough but are not cryptographic hashes. They can be deliberately collided pretty easily. A cryptographic hash like sha384 or blake2 makes it infeasible to come up with any input that would ever yield the same output. That’s what you would use for such a network.