4 ms·

I recently tried to use IPFS to share 20 gigabytes of data on a local network, using ipfs-desktop. Apparently it hashes the entire file, meaning it will read i

by jokoon 6y ago

I recently tried to use IPFS to share 20 gigabytes of data on a local network, using ipfs-desktop. Apparently it hashes the entire file, meaning it will read it entirely, so I killed it because it took way too long. Apparently there are other methods for fingerprinting large files, like reading a limited set of chunks. I asked on IRC but I did not get a thorough answer.

imhoguy 6y ago
To share files in a local network FTP or SMB/CIFS may be better suited. IPFS is content addressable storage by design on entire files. Reading limited set of chunks poses some problem regarding data integrity and deduplication colisions. One bit flip in JPG may destroy the entire image - thankfuly with currently used hash functions the right and broken files will have different "fingerprint". One way hash functions have extremely low probability of collision with full file scan. However when you skip part of files the probalibity of collision is roughly equal to count of bytes not scaned divided by total file size.
- toomuchtodo 6y ago
  It’s a shame modern file systems don’t generate and store common hash algorithms as file metadata.
  - imhoguy 6y ago
    Some file systems like ZFS, btrfs support checksum on block level. However file level checksum is not trivial with random write file system. Imagine 1 byte is changed in 20GB file - that would require full file scan to update the checksum. Recalculating a file checksum from all block checksums could be a solution however far from any standard.
    - brokenmachine 6y ago
      Wouldn't a merkle tree, like torrents or dat use, avoid this? You'd only have to generate a hash of the changed chunk, then it's really low cost to update the root hash. https://datprotocol.github.io/book/ch01-02-merkle-tree.html https://datprotocol.github.io/book/ch01-02-merkle-tree.html
      - imhoguy 6y ago
        Yep, my last sentence covered also Markle tree.
fit2rule 6y ago
I don't understand why you got impatient? Just let it finish and then you can share, easily enough across your entire network.

by jokoon 6y ago

imhoguy 6y ago

To share files in a local network FTP or SMB/CIFS may be better suited. IPFS is content addressable storage by design on entire files. Reading limited set of chunks poses some problem regarding data integrity and deduplication colisions. One bit flip in JPG may destroy the entire image - thankfuly with currently used hash functions the right and broken files will have different "fingerprint". One way hash functions have extremely low probability of collision with full file scan. However when you skip part of files the probalibity of collision is roughly equal to count of bytes not scaned divided by total file size.

toomuchtodo 6y ago
It’s a shame modern file systems don’t generate and store common hash algorithms as file metadata.
- imhoguy 6y ago
  Some file systems like ZFS, btrfs support checksum on block level. However file level checksum is not trivial with random write file system. Imagine 1 byte is changed in 20GB file - that would require full file scan to update the checksum. Recalculating a file checksum from all block checksums could be a solution however far from any standard.
  - brokenmachine 6y ago
    Wouldn't a merkle tree, like torrents or dat use, avoid this? You'd only have to generate a hash of the changed chunk, then it's really low cost to update the root hash. https://datprotocol.github.io/book/ch01-02-merkle-tree.html https://datprotocol.github.io/book/ch01-02-merkle-tree.html
    - imhoguy 6y ago
      Yep, my last sentence covered also Markle tree.

fit2rule 6y ago

I don't understand why you got impatient? Just let it finish and then you can share, easily enough across your entire network.