6 ms·

How you will calculate hash of file, when it broken, to lookup for?

01HNNWZ0MV43FF 2y ago
You could do a rolling hash and say that a chunk with a given hash should appear between two other chunks of certain hashes
- arijun 2y ago
  That seems like a recipe for nefarious code insertion.
  - 01HNNWZ0MV43FF 2y ago
    oh shit yeah it does lol
everfree 2y ago
Just hash it before it's broken.
- jonhohle 2y ago
  Maybe this is a joke that’s over my head, but the OP wants a system where damaged media can be repaired. They have the damaged media so there’s no way to make a hash of the content they want.
  - OnlyMortal 2y ago
    How far would error correction go?
selcuka 2y ago
Just use the sector number(s) of the damaged parts.
alex_duf 2y ago
if you store the merkle tree that was used to download it, you'll be able to know exactly which chunk of the file got a bit flip.
rakoo 2y ago
You have all the hashes in the .torrent file. All you need is a regular check with it (but then the .torrent file itself has to be stored on a storage that resists bit flipping)
- arijun 2y ago
  If you’re worried about bit-flipping, you could just store multiple copies of the hash and then do voting, since it’s small. If you’re worried about correlated sources of error that helps less, though.
- Dibby053 2y ago
  >storage [...] bit flipping As someone with no storage expertise I'm curious, does anyone know the likelyhood of an error resulting in a bit flip rather than an unreadable sector? Memory bit flips during I/O are another thing but I'd expect a modern HDD/SSD to return an error if it isn't sure about what it's reading.
  - halfcat 2y ago
    Not sure if this is what you mean, but most HDD vendors publish reliability data like “Non-recoverable read errors per bits read”: https://documents.westerndigital.com/content/dam/doc-library/en_us/assets/public/western-digital/product/internal-drives/wd-red-hdd/product-brief-western-digital-wd-red-hdd.pdf https://documents.westerndigital.com/content/dam/doc-library...
    - Dibby053 2y ago
      Thanks for the link. I think that 10^14 figure is the likelyhood of the disk error correction failing to produce a valid result from the underlying media, returning a read error and adding the block to pending bad sectors. A typical read error that is caught by the OS and prompts the user to replace drives. What I understand by bit flip is a corruption that gets past that check (ie the "flips balance themselves" and produce a valid ECC) and returns bad data to the OS without producing any errors. Only a few filesystems that make their own checksums (like ZFS) would catch this failure mode. It's one reason I still use ZFS despite the downsides, so I wonder if I'm being too cautious about something that essentially can't happen.

You could do a rolling hash and say that a chunk with a given hash should appear between two other chunks of certain hashes

arijun 2y ago
That seems like a recipe for nefarious code insertion.
- 01HNNWZ0MV43FF 2y ago
  oh shit yeah it does lol

everfree 2y ago

Just hash it before it's broken.

jonhohle 2y ago
Maybe this is a joke that’s over my head, but the OP wants a system where damaged media can be repaired. They have the damaged media so there’s no way to make a hash of the content they want.
- OnlyMortal 2y ago
  How far would error correction go?

selcuka 2y ago

Just use the sector number(s) of the damaged parts.

alex_duf 2y ago

if you store the merkle tree that was used to download it, you'll be able to know exactly which chunk of the file got a bit flip.

rakoo 2y ago

You have all the hashes in the .torrent file. All you need is a regular check with it (but then the .torrent file itself has to be stored on a storage that resists bit flipping)

arijun 2y ago
If you’re worried about bit-flipping, you could just store multiple copies of the hash and then do voting, since it’s small. If you’re worried about correlated sources of error that helps less, though.
Dibby053 2y ago
>storage [...] bit flipping As someone with no storage expertise I'm curious, does anyone know the likelyhood of an error resulting in a bit flip rather than an unreadable sector? Memory bit flips during I/O are another thing but I'd expect a modern HDD/SSD to return an error if it isn't sure about what it's reading.
- halfcat 2y ago
  Not sure if this is what you mean, but most HDD vendors publish reliability data like “Non-recoverable read errors per bits read”: https://documents.westerndigital.com/content/dam/doc-library/en_us/assets/public/western-digital/product/internal-drives/wd-red-hdd/product-brief-western-digital-wd-red-hdd.pdf https://documents.westerndigital.com/content/dam/doc-library...
  - Dibby053 2y ago
    Thanks for the link. I think that 10^14 figure is the likelyhood of the disk error correction failing to produce a valid result from the underlying media, returning a read error and adding the block to pending bad sectors. A typical read error that is caught by the OS and prompts the user to replace drives. What I understand by bit flip is a corruption that gets past that check (ie the "flips balance themselves" and produce a valid ECC) and returns bad data to the OS without producing any errors. Only a few filesystems that make their own checksums (like ZFS) would catch this failure mode. It's one reason I still use ZFS despite the downsides, so I wonder if I'm being too cautious about something that essentially can't happen.