3 ms·
Last I checked it was dumb. Possibly breaking block boundaries based on a rolling hash.
by fwip 8y ago
Last I checked it was dumb. Possibly breaking block boundaries based on a rolling hash.
- sideeffffect 8y agohttps://github.com/ipfs/go-ipfs-chunker https://github.com/ipfs/go-ipfs-chunker > go-ipfs-chunker provides the Splitter interface. IPFS splitters read data from a reader an create "chunks". These chunks are used to build the ipfs DAGs (Merkle Tree) and are the base unit to obtain the sums that ipfs uses to address content. > The package provides a SizeSplitter which creates chunks of equal size and it is used by default in most cases, and a rabin fingerprint chunker. This chunker will attempt to split data in a way that the resulting blocks are the same when the data has repetitive patterns, thus optimizing the resulting DAGs. I think they should use the rolling hash based chunking by default https://github.com/ipfs/go-ipfs-chunker/issues/13 https://github.com/ipfs/go-ipfs-chunker/issues/13