11 ms·
To share files in a local network FTP or SMB/CIFS may be better suited. IPFS is content addressable storage by design on entire files. Reading limited set of c
by imhoguy 6y ago
To share files in a local network FTP or SMB/CIFS may be better suited.
IPFS is content addressable storage by design on entire files. Reading limited set of chunks poses some problem regarding data integrity and deduplication colisions. One bit flip in JPG may destroy the entire image - thankfuly with currently used hash functions the right and broken files will have different "fingerprint". One way hash functions have extremely low probability of collision with full file scan. However when you skip part of files the probalibity of collision is roughly equal to count of bytes not scaned divided by total file size.
- toomuchtodo 6y agoIt’s a shame modern file systems don’t generate and store common hash algorithms as file metadata.
- imhoguy 6y agoSome file systems like ZFS, btrfs support checksum on block level. However file level checksum is not trivial with random write file system. Imagine 1 byte is changed in 20GB file - that would require full file scan to update the checksum. Recalculating a file checksum from all block checksums could be a solution however far from any standard.
- brokenmachine 6y agoWouldn't a merkle tree, like torrents or dat use, avoid this? You'd only have to generate a hash of the changed chunk, then it's really low cost to update the root hash. https://datprotocol.github.io/book/ch01-02-merkle-tree.html https://datprotocol.github.io/book/ch01-02-merkle-tree.html
- imhoguy 6y agoYep, my last sentence covered also Markle tree.