4 ms·

yeah, hashes are decided by every single bit including the last one, and not a single bit should be output until the hash matches. It currently buffers everythi

yeah, hashes are decided by every single bit including the last one, and not a single bit should be output until the hash matches. It currently buffers everything in memory, but might do this: https://github.com/jbenet/hashpipe/issues/1 https://github.com/jbenet/hashpipe/issues/1 (some settings dont have disk though). hashpipe is intended for most executable use cases (usually under <50MB)

doomrobo 12y ago
Loading everything into memory at once shouldn't be necessary to produce a hash of the entire input. All of the hash functions currently supported allow for incremental hashing. That means you can hash in blocks instead of all at once.
- unsoundInput 12y ago
  The input still needs to be cached for eventual output in case the hash matches I assume.
  - throwaway_99837 12y ago
    Or, you know, do two passes.
  - doomrobo 12y ago
    In that case, you could still cache if you want but at least you now have the option to manage your memory instead of always crashing with large input
  - stouset 12y ago
    More importantly, so nothing is output if the hash doesn't match.
stormbrew 12y ago
You should be able to detect if the stream is seekable (by checking the result of 'lseek(fd,0,SEEK_CUR)') and only buffer if it's not. Of course, if you're really paranoid, the file could get changed out from under you. But honestly you're probably screwed either way with an attacker who can do that.
gojomo 12y ago
Another fun option would be to use a tree hash. The distributor of the content-to-be-verified would then envelope it in a format (to-be-defined) that includes proofs-up-to-root every N bytes. Then the verifier could stream, and know that everything it emits fits into the target hash, needing only N bytes of working space. Caveats: The source doing-the-enveloping will need two passes (and enough working space for the remainder-tree). An attacker could still choose the moment-when-content-goes-bad; in the envisioned use of immediately-executing the verifier output, this might leave things in a problematic/resource-consumptive state. (Scripts could be hardened against such partial-execution failures.)

doomrobo 12y ago

Loading everything into memory at once shouldn't be necessary to produce a hash of the entire input. All of the hash functions currently supported allow for incremental hashing. That means you can hash in blocks instead of all at once.

unsoundInput 12y ago
The input still needs to be cached for eventual output in case the hash matches I assume.
- throwaway_99837 12y ago
  Or, you know, do two passes.
- doomrobo 12y ago
  In that case, you could still cache if you want but at least you now have the option to manage your memory instead of always crashing with large input
- stouset 12y ago
  More importantly, so nothing is output if the hash doesn't match.

stormbrew 12y ago

You should be able to detect if the stream is seekable (by checking the result of 'lseek(fd,0,SEEK_CUR)') and only buffer if it's not. Of course, if you're really paranoid, the file could get changed out from under you. But honestly you're probably screwed either way with an attacker who can do that.

gojomo 12y ago

Another fun option would be to use a tree hash. The distributor of the content-to-be-verified would then envelope it in a format (to-be-defined) that includes proofs-up-to-root every N bytes. Then the verifier could stream, and know that everything it emits fits into the target hash, needing only N bytes of working space. Caveats: The source doing-the-enveloping will need two passes (and enough working space for the remainder-tree). An attacker could still choose the moment-when-content-goes-bad; in the envisioned use of immediately-executing the verifier output, this might leave things in a problematic/resource-consumptive state. (Scripts could be hardened against such partial-execution failures.)