6 ms·

Since the main use case for this utility is verifying network shell scripts, it would be interesting to see a query param convention, so we could use a tool suc

Since the main use case for this utility is verifying network shell scripts, it would be interesting to see a query param convention, so we could use a tool such as: > hashcurl http://load.this/script?hash= http://load.this/script?hash= QmUJPTFZnR2CPGAzmfdYPghgrFtYFB6pf1BqMvqfiPDam8 | sh

gojomo 12y ago
When the just-over-the-next-hilltop promised-land nirvana of content-centric networking arrives, the hash will be enough to locate & download the content – so you shouldn't even need an URL: $ hashcurl QmUJPTFZnR2CPGAzmfdYPghgrFtYFB6pf1BqMvqfiPDam8 | sh Maybe it's even a special filesystem path, that contains (but does not list) everything-that's-nameable-and-findable: $ sh /everything/QmUJPTFZnR2CPGAzmfdYPghgrFtYFB6pf1BqMvqfiPDam8 (BTW, personally not a fan of the opaque 'multihash' format, which obscures the algorithm-in-use to save a few characters.)
- andrewflnr 12y ago
  I'm not sure if it's really great or a really terrible, but I love the "/everything" idea.
  - whyrusleeping 12y ago
    you should check out ipfs.io then :) Disclaimer: I'm one of the devs
- stouset 12y ago
  It's not save a few characters. It's to allow for flexibility of encoding. "sha1" all but requires ASCII. Sometimes you're limited to hex, base64, etc.
  - gojomo 12y ago
    Not sure I understand; almost any context where "QmTpn…" can appear could also handle a "sha1:" human-readable prefix. Including, notably, this shell context. If another context is sufficiently different, it would be fine to use a controlled fixed-length binary-vocabulary there. (But, the ASCII bytes for "sha1:" would still be a pretty robust and largely self-documenting choice.)
    - _prometheus 12y ago
      oh the "Qm..." string is the base58 representation. you can have a binary packed version as well. check out more at https://github.com/jbenet/multihash/ https://github.com/jbenet/multihash/ the important thing here is standardizing the encoding, i.e. there should be a valid repr in {binary, hex, b32, b58, b64, ... bemoji ...} so that, say, if your input field _only takes hex_ you have a way to enter the hash. this problem surfaces when tools expect reprs to be in a particular encoding (all too common), and you dont have power or access to make the tool better (sadly all too common too).
      - gojomo 12y ago
        'sha1:' (0x736861313a) and other self-describing labels can be represented in hex (etc) just fine.
        stouset 12y ago
        So... it's readable in one out of dozens of possible encodings? And realistically, in that encoding (ASCII), the part following the readable prefix is unreadable garbage. In exchange for this convenience, you add a four bytes of unreadable prefix instead of one. I'm not sure I see the point. I deal with outputs from crypto functions on a daily basis. Never once have I intentionally rendered it as 8-byte ASCII. It's most often in hex. Mixing and matching encodings (e.g., prefixing a hex string with an ASCII string) for a single blob of data is silly and just causes headaches whenever you need to change encodings, either by having some data double-encoded or by having to specially handle the prefix separately.
        gojomo 12y ago
        My hunch is that the overwhelmingly dominant and important use case is where these identifiers appear in URLs, including URL fragments (and potentially in brand-new protocols). A few other important cases are also where they're visible to people, as in the example (or hypothetical) command-lines that kicked off this thread. In those cases, explicitness-at-a-glance helps, as will having one canonical encoding (such as b32, b64, or b58). Compared to that, adaptation to other constrained systems is a case-by-case issue. And if those systems are already capable of squeezing in these non-native slightly-longer hash-like-strings, then a few more bytes usually won't hurt, or if they do then whatever deep-in-the-bits coder (like yourself) who's shoehorning things in can handle compactification. The display/exchange format should be as casually readable as possible. Such an ASCII-name-inspired prefix is readable in all encodings. In some, it's just a magic number (but one that any coder can make educated guesses about); but in the one encoding that's likely most important for mutual comprehension between humans (user-exchanged strings and URLs), it's super-duper-readable. And yes, the prefix should in general have special handling: as the multihash project README notes in an important "warning", the prefix lacks the same distribution as the other bytes. Treating the whole thing as an opaque-but-still-reliable identifier invites indexing misoptimizations right off the bat, and then other later bugs, if any of the hash functions become deprecated, or a new hash is added with variant semantics.
- _prometheus 12y ago
  this already works. install ipfs: http://ipfs.io/docs/install http://ipfs.io/docs/install then: ipfs init ipfs daemon & sleep 20 # sorry this will go away ipfs mount sh /ipfs/QmTpnQL97XEHmyt54mgEwf5BN8gJWvw4sGgwQzjqtBwLX6 you can see it on the web at: http://gateway.ipfs.io/ipfs/QmTpnQL97XEHmyt54mgEwf5BN8gJWvw4sGgwQzjqtBwLX6 http://gateway.ipfs.io/ipfs/QmTpnQL97XEHmyt54mgEwf5BN8gJWvw4...
  - gojomo 12y ago
    That's great! I hope that IPFS, or something of its ilk, can be the promised-land to which I alluded. If I were to install it, how hard/breaking would it be to change the access-path to something more grandiosely descriptive like '/everything/'?
    - whyrusleeping 12y ago
      its a simple matter of changing the ipfs config file
      - _prometheus 12y ago
        <3
    - _prometheus 12y ago
      also, we want to push towards `/` being everything, shoving all your local crap into `/local` or `/localhost`
  - KMag 12y ago
    If any of you are designing a system like ipfs, please use Merkle tree roots as the identifiers (and make sure leaf nodes are hashed differently than inner nodes as in THEX, or better yet Joan Daemen's Sakura construction). The main reason is that at some point you probably want to support downloading from multiple untrusted sources. Properly implemented Merkle trees (such as using the Sakura construction) are provably as strong as the underlying hash algorithm and allow lightweight cryptographic proof that a given block belongs to the file in question and belongs in the claimed place. Gnutella uses the SHA-1 of the whole file the identifier and then just assumes the first peer to give it a Merkle tree root is giving it the correct root. Because there isn't guaranteed consistency between the identifier and the tree root, this makes it vulnerable to an attack where an attacker tries hard to be first, and give it a root for a corrupted version of the file and the peer wastes a lot of time and bandwidth re-downloading uncorrupted blocks that fail to verify. Alternatively, you could make the identifier the concatenation of a full-file hash and a Merkle tree root. However, since the Sakura construction (and several other constructions) are provably as strong as the underlying hash, this is a waste of space. Concatenating the SHA-256 of the file with a SHA-256 Merkle tree root only gives you as much strength as the first 257 bits of a SHA-512 Merkle tree root. If you can spare the space for the longer identifiers, you're better off using a longer hash instead of two shorter hashes. In other words, it's a waste of space to concatenate a whole-file hash with a Merkle tree root if you use a tree construction that's provably as strong as the underlying hash function.
- serendipitous 12y ago
  This sounds like DHT, which has existed for a while now. https://en.wikipedia.org/wiki/Distributed_hash_table https://en.wikipedia.org/wiki/Distributed_hash_table

gojomo 12y ago

When the just-over-the-next-hilltop promised-land nirvana of content-centric networking arrives, the hash will be enough to locate & download the content – so you shouldn't even need an URL: $ hashcurl QmUJPTFZnR2CPGAzmfdYPghgrFtYFB6pf1BqMvqfiPDam8 | sh Maybe it's even a special filesystem path, that contains (but does not list) everything-that's-nameable-and-findable: $ sh /everything/QmUJPTFZnR2CPGAzmfdYPghgrFtYFB6pf1BqMvqfiPDam8 (BTW, personally not a fan of the opaque 'multihash' format, which obscures the algorithm-in-use to save a few characters.)

andrewflnr 12y ago
I'm not sure if it's really great or a really terrible, but I love the "/everything" idea.
- whyrusleeping 12y ago
  you should check out ipfs.io then :) Disclaimer: I'm one of the devs
stouset 12y ago
It's not save a few characters. It's to allow for flexibility of encoding. "sha1" all but requires ASCII. Sometimes you're limited to hex, base64, etc.
- gojomo 12y ago
  Not sure I understand; almost any context where "QmTpn…" can appear could also handle a "sha1:" human-readable prefix. Including, notably, this shell context. If another context is sufficiently different, it would be fine to use a controlled fixed-length binary-vocabulary there. (But, the ASCII bytes for "sha1:" would still be a pretty robust and largely self-documenting choice.)
  - _prometheus 12y ago
    oh the "Qm..." string is the base58 representation. you can have a binary packed version as well. check out more at https://github.com/jbenet/multihash/ https://github.com/jbenet/multihash/ the important thing here is standardizing the encoding, i.e. there should be a valid repr in {binary, hex, b32, b58, b64, ... bemoji ...} so that, say, if your input field _only takes hex_ you have a way to enter the hash. this problem surfaces when tools expect reprs to be in a particular encoding (all too common), and you dont have power or access to make the tool better (sadly all too common too).
    - gojomo 12y ago
      'sha1:' (0x736861313a) and other self-describing labels can be represented in hex (etc) just fine.
      - stouset 12y ago
        So... it's readable in one out of dozens of possible encodings? And realistically, in that encoding (ASCII), the part following the readable prefix is unreadable garbage. In exchange for this convenience, you add a four bytes of unreadable prefix instead of one. I'm not sure I see the point. I deal with outputs from crypto functions on a daily basis. Never once have I intentionally rendered it as 8-byte ASCII. It's most often in hex. Mixing and matching encodings (e.g., prefixing a hex string with an ASCII string) for a single blob of data is silly and just causes headaches whenever you need to change encodings, either by having some data double-encoded or by having to specially handle the prefix separately.
        gojomo 12y ago
        My hunch is that the overwhelmingly dominant and important use case is where these identifiers appear in URLs, including URL fragments (and potentially in brand-new protocols). A few other important cases are also where they're visible to people, as in the example (or hypothetical) command-lines that kicked off this thread. In those cases, explicitness-at-a-glance helps, as will having one canonical encoding (such as b32, b64, or b58). Compared to that, adaptation to other constrained systems is a case-by-case issue. And if those systems are already capable of squeezing in these non-native slightly-longer hash-like-strings, then a few more bytes usually won't hurt, or if they do then whatever deep-in-the-bits coder (like yourself) who's shoehorning things in can handle compactification. The display/exchange format should be as casually readable as possible. Such an ASCII-name-inspired prefix is readable in all encodings. In some, it's just a magic number (but one that any coder can make educated guesses about); but in the one encoding that's likely most important for mutual comprehension between humans (user-exchanged strings and URLs), it's super-duper-readable. And yes, the prefix should in general have special handling: as the multihash project README notes in an important "warning", the prefix lacks the same distribution as the other bytes. Treating the whole thing as an opaque-but-still-reliable identifier invites indexing misoptimizations right off the bat, and then other later bugs, if any of the hash functions become deprecated, or a new hash is added with variant semantics.
_prometheus 12y ago
this already works. install ipfs: http://ipfs.io/docs/install http://ipfs.io/docs/install then: ipfs init ipfs daemon & sleep 20 # sorry this will go away ipfs mount sh /ipfs/QmTpnQL97XEHmyt54mgEwf5BN8gJWvw4sGgwQzjqtBwLX6 you can see it on the web at: http://gateway.ipfs.io/ipfs/QmTpnQL97XEHmyt54mgEwf5BN8gJWvw4sGgwQzjqtBwLX6 http://gateway.ipfs.io/ipfs/QmTpnQL97XEHmyt54mgEwf5BN8gJWvw4...
- gojomo 12y ago
  That's great! I hope that IPFS, or something of its ilk, can be the promised-land to which I alluded. If I were to install it, how hard/breaking would it be to change the access-path to something more grandiosely descriptive like '/everything/'?
  - whyrusleeping 12y ago
    its a simple matter of changing the ipfs config file
    - _prometheus 12y ago
      <3
  - _prometheus 12y ago
    also, we want to push towards `/` being everything, shoving all your local crap into `/local` or `/localhost`
- KMag 12y ago
  If any of you are designing a system like ipfs, please use Merkle tree roots as the identifiers (and make sure leaf nodes are hashed differently than inner nodes as in THEX, or better yet Joan Daemen's Sakura construction). The main reason is that at some point you probably want to support downloading from multiple untrusted sources. Properly implemented Merkle trees (such as using the Sakura construction) are provably as strong as the underlying hash algorithm and allow lightweight cryptographic proof that a given block belongs to the file in question and belongs in the claimed place. Gnutella uses the SHA-1 of the whole file the identifier and then just assumes the first peer to give it a Merkle tree root is giving it the correct root. Because there isn't guaranteed consistency between the identifier and the tree root, this makes it vulnerable to an attack where an attacker tries hard to be first, and give it a root for a corrupted version of the file and the peer wastes a lot of time and bandwidth re-downloading uncorrupted blocks that fail to verify. Alternatively, you could make the identifier the concatenation of a full-file hash and a Merkle tree root. However, since the Sakura construction (and several other constructions) are provably as strong as the underlying hash, this is a waste of space. Concatenating the SHA-256 of the file with a SHA-256 Merkle tree root only gives you as much strength as the first 257 bits of a SHA-512 Merkle tree root. If you can spare the space for the longer identifiers, you're better off using a longer hash instead of two shorter hashes. In other words, it's a waste of space to concatenate a whole-file hash with a Merkle tree root if you use a tree construction that's provably as strong as the underlying hash function.
serendipitous 12y ago
This sounds like DHT, which has existed for a while now. https://en.wikipedia.org/wiki/Distributed_hash_table https://en.wikipedia.org/wiki/Distributed_hash_table