5 ms·

Indeed. It's true that the data may be corrupted before hitting any disk[1], but once it has hit the disks (>1), it's extremely unlikely that you'll ever hit a

Indeed. It's true that the data may be corrupted before hitting any disk[1], but once it has hit the disks (>1), it's extremely unlikely that you'll ever hit a similar bit error where it'll mistakenly choose the wrong disk block to recover from. The main point of e.g. ZFS or Btrfs checksumming is that a) at least it isn't getting worse, and b) I can tell if it's getting worse. [1] ... but if the bits are not generated by the machine that actually saving to disk, how do you know they weren't corrupted along the way? The number of people who religiously check PGP signatures/SHA256sums or whatever is miniscule.

ruleabidinguser 10y ago
What are you doing where you're actually checking checksums periodically and detcting when things get worse? That seems like a lot of work to set up.
- boomboomsubban 10y ago
  They are using ZFS, scrubbing is one command.
- lomnakkus 10y ago
  zpool scrub (This may be a myth: It's not something you should actually do that often because actually reading the media may dregrade it.)
  - derefr 10y ago
    Either way, it'd be fine on SSDs, then?
    - lomnakkus 10y ago
      Hm? As long as the error correction technology on your chosen SSD of choice stands up, I guess... yes? What, exactly, are you asking?
      - derefr 10y ago
        My thought was "reading SSDs doesn't degrade them, so there's no disadvantage to constant scrubbing." Unless I've misunderstood what you mean by "reading."
        lomnakkus 10y ago
        Ah, right, fair point. AFAIUI SSD storage does degrade a tiny(!) amount when reading, but magnetic storage degrades quite a bit more. (That's where that came from, draw your own conlusions :).)
  - kscz 10y ago
    or for btrfs users out there: btrfs scrub start /mnt/volume_name
derefr 10y ago
> The number of people who religiously check PGP signatures/SHA256sums or whatever is miniscule. • If you transfer things around using BitTorrent, it'll ensure you always end up with a file that hashes correctly to the sum it originally had when the .torrent file was constructed. • Many archive formats (zip, rar, and 7z, at least) contain checksums, and archival utilities validate those checksums during extraction, refusing to extract broken files. "Self-extracting archive" executables that use these formats inherit this property. • Some common disk-image formats (dmg, wim) embed a checksum that checks the whole disk-image during mount, and will refuse to mount a bad one. (I believe you can then try to "repair" the disk image with your OS's disk-repair utility, if you have no other copies.) • Web pages increasingly use Sub-Resource Integrity attributes on things like .css and .js files, protecting them (though not the page itself) from errors. • ISO files don't embed checks, but all the common package formats (Windows .cab and .msi; Linux .deb and .rpm; macOS .pkg) on installer ISOs embed their own checksums and often signatures. • git repos are 'protected' insofar as you won't be able to sync mis-hashed objects from a remote, so they won't spread. Really, looking over all that, it's only 1. plain binary executables, and 2. "media files" (images, audio, video)—and only when retrieved over a "dumb" protocol, rather than a pre-baked-manifest protocol like BitTorrent or zsync—that are "risky" and in need of explicit checksum comparison.
- lomnakkus 10y ago
  True. Still mostly immaterial in the grand scheme of things, I think? EDIT: Should say, as a point of interest: Even though .zip's were protected back in the Good Old Days, that didn't really matter because we all got corrupted (expanded from .zip) .mp3's because of those fucking RTL3xxsomething cards that would just transmit things perfectly and then corrupt the checksum (or whichever way 'round). Ugh. One of the few times I've actually hated engineers. (Don't get me wrong. I really do want more of these checks to be pervasive. We start with our local file systems.)
- comex 10y ago
  Both macOS/iOS and Windows use code signing for executables, which should guard against most types of corruption. Web pages (and anything else) transmitted over HTTPS are protected from corruption in transit by TLS's hashing (which is vastly stronger than the checksums at lower levels of the network stack), though that doesn't help if the server has faulty memory or storage. PNG has built-in checksums, though other image formats don't (JPEG). Not sure about video.

ruleabidinguser 10y ago

What are you doing where you're actually checking checksums periodically and detcting when things get worse? That seems like a lot of work to set up.

boomboomsubban 10y ago
They are using ZFS, scrubbing is one command.
lomnakkus 10y ago
zpool scrub (This may be a myth: It's not something you should actually do that often because actually reading the media may dregrade it.)
- derefr 10y ago
  Either way, it'd be fine on SSDs, then?
  - lomnakkus 10y ago
    Hm? As long as the error correction technology on your chosen SSD of choice stands up, I guess... yes? What, exactly, are you asking?
    - derefr 10y ago
      My thought was "reading SSDs doesn't degrade them, so there's no disadvantage to constant scrubbing." Unless I've misunderstood what you mean by "reading."
      - lomnakkus 10y ago
        Ah, right, fair point. AFAIUI SSD storage does degrade a tiny(!) amount when reading, but magnetic storage degrades quite a bit more. (That's where that came from, draw your own conlusions :).)
- kscz 10y ago
  or for btrfs users out there: btrfs scrub start /mnt/volume_name

derefr 10y ago

> The number of people who religiously check PGP signatures/SHA256sums or whatever is miniscule. • If you transfer things around using BitTorrent, it'll ensure you always end up with a file that hashes correctly to the sum it originally had when the .torrent file was constructed. • Many archive formats (zip, rar, and 7z, at least) contain checksums, and archival utilities validate those checksums during extraction, refusing to extract broken files. "Self-extracting archive" executables that use these formats inherit this property. • Some common disk-image formats (dmg, wim) embed a checksum that checks the whole disk-image during mount, and will refuse to mount a bad one. (I believe you can then try to "repair" the disk image with your OS's disk-repair utility, if you have no other copies.) • Web pages increasingly use Sub-Resource Integrity attributes on things like .css and .js files, protecting them (though not the page itself) from errors. • ISO files don't embed checks, but all the common package formats (Windows .cab and .msi; Linux .deb and .rpm; macOS .pkg) on installer ISOs embed their own checksums and often signatures. • git repos are 'protected' insofar as you won't be able to sync mis-hashed objects from a remote, so they won't spread. Really, looking over all that, it's only 1. plain binary executables, and 2. "media files" (images, audio, video)—and only when retrieved over a "dumb" protocol, rather than a pre-baked-manifest protocol like BitTorrent or zsync—that are "risky" and in need of explicit checksum comparison.

lomnakkus 10y ago
True. Still mostly immaterial in the grand scheme of things, I think? EDIT: Should say, as a point of interest: Even though .zip's were protected back in the Good Old Days, that didn't really matter because we all got corrupted (expanded from .zip) .mp3's because of those fucking RTL3xxsomething cards that would just transmit things perfectly and then corrupt the checksum (or whichever way 'round). Ugh. One of the few times I've actually hated engineers. (Don't get me wrong. I really do want more of these checks to be pervasive. We start with our local file systems.)
comex 10y ago
Both macOS/iOS and Windows use code signing for executables, which should guard against most types of corruption. Web pages (and anything else) transmitted over HTTPS are protected from corruption in transit by TLS's hashing (which is vastly stronger than the checksums at lower levels of the network stack), though that doesn't help if the server has faulty memory or storage. PNG has built-in checksums, though other image formats don't (JPEG). Not sure about video.