4 ms·

Be warned that this reads everything until EOF into memory, doing something like: hashpipe QmUJPTFZnR2CPGAzmfdYPghgrFtYFB6pf1BqMvqfiPDam8 < large_file wil

by tyho 12y ago

Be warned that this reads everything until EOF into memory, doing something like: hashpipe QmUJPTFZnR2CPGAzmfdYPghgrFtYFB6pf1BqMvqfiPDam8 < large_file will produce unexpected behaviour.

_prometheus 12y ago
yeah, hashes are decided by every single bit including the last one, and not a single bit should be output until the hash matches. It currently buffers everything in memory, but might do this: https://github.com/jbenet/hashpipe/issues/1 https://github.com/jbenet/hashpipe/issues/1 (some settings dont have disk though). hashpipe is intended for most executable use cases (usually under <50MB)
- doomrobo 12y ago
  Loading everything into memory at once shouldn't be necessary to produce a hash of the entire input. All of the hash functions currently supported allow for incremental hashing. That means you can hash in blocks instead of all at once.
  - unsoundInput 12y ago
    The input still needs to be cached for eventual output in case the hash matches I assume.
    - throwaway_99837 12y ago
      Or, you know, do two passes.
    - doomrobo 12y ago
      In that case, you could still cache if you want but at least you now have the option to manage your memory instead of always crashing with large input
    - stouset 12y ago
      More importantly, so nothing is output if the hash doesn't match.
- stormbrew 12y ago
  You should be able to detect if the stream is seekable (by checking the result of 'lseek(fd,0,SEEK_CUR)') and only buffer if it's not. Of course, if you're really paranoid, the file could get changed out from under you. But honestly you're probably screwed either way with an attacker who can do that.
- gojomo 12y ago
  Another fun option would be to use a tree hash. The distributor of the content-to-be-verified would then envelope it in a format (to-be-defined) that includes proofs-up-to-root every N bytes. Then the verifier could stream, and know that everything it emits fits into the target hash, needing only N bytes of working space. Caveats: The source doing-the-enveloping will need two passes (and enough working space for the remainder-tree). An attacker could still choose the moment-when-content-goes-bad; in the envisioned use of immediately-executing the verifier output, this might leave things in a problematic/resource-consumptive state. (Scripts could be hardened against such partial-execution failures.)
geoelectric 12y ago
So too much in the pipe and things get wonky? Still a perfect name!
kazinator 12y ago
It doesn't have to read into memory; if that becomes a problem they can fix it. The hashing algorithm just needs a small, constant amount of state. While it is hashing, it can divert the data to a temporary file. Then it can just pass the file to its standard output. A "hashexec" command could also be produced which passes the name of this temporary file to a subprogram: whatever | hashexec <hash> arbitrary --command with args This "arbitrary" command's standard input is redirected from a temporary file created by hashexec (so no cat-like loop has to execute to feed the data). Also, since this is for scripts, there could be an argument which limits the size. This could have default value, say one megabyte. Anyone pulling down scripts which are anywhere near one megabyte has to add explicit overrides for the size: whatever | hashpipe --max=4M <hash> | ... The real problem isn't that hashpipe buffers everything, because the scripting language running the script will also do that; rather that because it buffers everything, it can be DDoS'ed with an infinite stream or whatever. By incorporating a size limit, hashpipe could provide an additional protection measure to the next pipeline element: it protects against content which doesn't match the hash, and against content which is too large. Idea: the size of the input could be encoded as a few digits of the hash. Then the argument is unnecessary. The hash itself tells you that the script is exactly 6059 bytes long; if you read 6060 bytes, the input is not the right one. Or the program could just stop reading at 6059 and check the hash at that point, and either pass on the 6059 bytes or error out.
- _prometheus 12y ago
  All great ideas! mind adding some to https://github.com/jbenet/hashpipe/issues https://github.com/jbenet/hashpipe/issues ? over time we can implement some
  - kazinator 12y ago
    I don't use github; I have my own server with git and cgit.

by tyho 12y ago

Be warned that this reads everything until EOF into memory, doing something like: hashpipe QmUJPTFZnR2CPGAzmfdYPghgrFtYFB6pf1BqMvqfiPDam8 < large_file will produce unexpected behaviour.

_prometheus 12y ago

yeah, hashes are decided by every single bit including the last one, and not a single bit should be output until the hash matches. It currently buffers everything in memory, but might do this: https://github.com/jbenet/hashpipe/issues/1 https://github.com/jbenet/hashpipe/issues/1 (some settings dont have disk though). hashpipe is intended for most executable use cases (usually under <50MB)

doomrobo 12y ago
Loading everything into memory at once shouldn't be necessary to produce a hash of the entire input. All of the hash functions currently supported allow for incremental hashing. That means you can hash in blocks instead of all at once.
- unsoundInput 12y ago
  The input still needs to be cached for eventual output in case the hash matches I assume.
  - throwaway_99837 12y ago
    Or, you know, do two passes.
  - doomrobo 12y ago
    In that case, you could still cache if you want but at least you now have the option to manage your memory instead of always crashing with large input
  - stouset 12y ago
    More importantly, so nothing is output if the hash doesn't match.
stormbrew 12y ago
You should be able to detect if the stream is seekable (by checking the result of 'lseek(fd,0,SEEK_CUR)') and only buffer if it's not. Of course, if you're really paranoid, the file could get changed out from under you. But honestly you're probably screwed either way with an attacker who can do that.
gojomo 12y ago
Another fun option would be to use a tree hash. The distributor of the content-to-be-verified would then envelope it in a format (to-be-defined) that includes proofs-up-to-root every N bytes. Then the verifier could stream, and know that everything it emits fits into the target hash, needing only N bytes of working space. Caveats: The source doing-the-enveloping will need two passes (and enough working space for the remainder-tree). An attacker could still choose the moment-when-content-goes-bad; in the envisioned use of immediately-executing the verifier output, this might leave things in a problematic/resource-consumptive state. (Scripts could be hardened against such partial-execution failures.)

geoelectric 12y ago

So too much in the pipe and things get wonky? Still a perfect name!

kazinator 12y ago

It doesn't have to read into memory; if that becomes a problem they can fix it. The hashing algorithm just needs a small, constant amount of state. While it is hashing, it can divert the data to a temporary file. Then it can just pass the file to its standard output. A "hashexec" command could also be produced which passes the name of this temporary file to a subprogram: whatever | hashexec <hash> arbitrary --command with args This "arbitrary" command's standard input is redirected from a temporary file created by hashexec (so no cat-like loop has to execute to feed the data). Also, since this is for scripts, there could be an argument which limits the size. This could have default value, say one megabyte. Anyone pulling down scripts which are anywhere near one megabyte has to add explicit overrides for the size: whatever | hashpipe --max=4M <hash> | ... The real problem isn't that hashpipe buffers everything, because the scripting language running the script will also do that; rather that because it buffers everything, it can be DDoS'ed with an infinite stream or whatever. By incorporating a size limit, hashpipe could provide an additional protection measure to the next pipeline element: it protects against content which doesn't match the hash, and against content which is too large. Idea: the size of the input could be encoded as a few digits of the hash. Then the argument is unnecessary. The hash itself tells you that the script is exactly 6059 bytes long; if you read 6060 bytes, the input is not the right one. Or the program could just stop reading at 6059 and check the hash at that point, and either pass on the 6059 bytes or error out.

_prometheus 12y ago
All great ideas! mind adding some to https://github.com/jbenet/hashpipe/issues https://github.com/jbenet/hashpipe/issues ? over time we can implement some
- kazinator 12y ago
  I don't use github; I have my own server with git and cgit.