5 ms·

There are programs with which you can add any desired amount of redundancy to your backup archives, so that they would survive corruption that does not affect a

There are programs with which you can add any desired amount of redundancy to your backup archives, so that they would survive corruption that does not affect a greater amount of data than the added redundancy. For instance, on Linux there is par2cmdline. For all my backups, I create pax archives, which are then compressed, then encrypted, then expanded with par2create, then aggregated again in a single pax file (the legacy tar file formats are not good for faithfully storing all metadata of modern file systems and each kind of tar program may have proprietary non-portable extensions to handle this, therefore I use only the pax file format). Besides that, important data should be replicated and stored on 2 or even 3 SSDs/HDDs/tapes, which should preferably be stored themselves in different locations.

antonkochubey 1y ago
Unfortunately some SSD controllers plainly refuse to read data they consider corrupted, even if you have extra parity that could potentially restore corrupted data, your entire drive might refuse to read.
- lazide 1y ago
  Huh? The issue being discussed is random blocks, yes? If your entire drive is bricked, that is an entirely different issue.
  - jeremyvisser 1y ago
    Here’s the thing. That SSD controller is the interface between you and those blocks. If it decides, by some arbitrary measurement, as defined by some logic within its black box firmware, that it should stop returning all blocks, then it will do so, and you have almost no recourse. This is a very common failure mode of SSDs. As a consequence of some failed blocks (likely exceeding a number of failed blocks, or perhaps the controller’s own storage failed), drives will commonly brick themselves. Perhaps you haven’t seen it happen, or your SSD doesn’t do this, or perhaps certain models or firmwares don’t, but some certainly do, both from my own experience, and countless accounts I’ve read elsewhere, so this is more common than you might realise.
    - reactordev 1y ago
      This is correct, you still have to go through firmware to gain access to the block/page on “disk” and if the firmware decides the block is invalid than it fails. You can sidestep this by bypassing the controller on a test bench though. Pinning wires to the chips. At that point it’s no longer an SSD.
    - lazide 1y ago
      Yes, and? HDD controllers dying and head crashes are a thing too. At least in the ‘bricked’ case it’s a trivial RMA - corrupt blocks tend to be a harder fight. And since ‘bricked’ is such a trivial RMA, manufacturers have more of an incentive to fix it or go broke, or avoid it in the first place. This is why backups are important now; and always have been.
      - mort96 1y ago
        We're not talking about the SSD controller dying. The SSD controller in the hypothetical situation that's being described is working as intended.
        lazide 1y ago
        Not as far as I can tell, where intended is ‘as any user would reasonably expect’. Bricking the drive (can’t even read) because of too many errors is not what most users would ever want. Some would (enterprise maybe), but even then they’d want deterministic data deletes too, which doesn’t sound like are happening.
        mort96 1y ago
        You can argue that controllers shouldn't behave that way. But they do, it's not a bug, and it's not a dead controller. It's a perfectly functional controller's response to dead blocks.
        lazide 1y ago
mywittyname 1y ago
This is fine, but I'd prefer an option to transparently add parity bits to the drive, even if it means losing access to capacity. Personally, I keep backups of critical data on a platter disk NAS, so I'm not concerned about losing critical data off of an SSD. However, I did recently have to reinstall Windows on a computer because of a randomly corrupted system file. Which is something this feature would have prevented.
casenmgreen 1y ago
Thank you for this. I had no knowledge of pax, or that par was an open standard, and I care about what they help with. Going to switch over to using both in my backups.
- adrian_b 1y ago
  For handling pax archives, I recommend the "libarchive" package, which is available in many Linux distributions, even if it originally comes from FreeBSD. Among other utilities, it installs the "bsdtar" program, which you can use in your scripts like this: bsdtar --create --verbose --format=pax --file="${DIRECTORY}".pax "${DIRECTORY}" || exit And for extraction: bsdtar --extract --preserve-permissions --verbose --file="${DIRECTORY}".pax The bsdtar program has options for compressing and/or encrypting the archives, for the case when you do not want to use directly other external programs. "par2create" creates multiple files from the (normally compressed and encrypted) archive file, for storing the added redundancy. I make a directory where I move those files, then I use a second time bsdtar (obviously without any compression or encryption) to aggregate those files in a single archive with redundancy. The libarchive package can also be taken directly from: https://github.com/libarchive/libarchive https://github.com/libarchive/libarchive "libarchive" handles correctly all kinds of file metadata, e.g. extended file attributes and high-resolution file timestamps, which not all archiving utilities do. Many Linux utilities, with the default command-line options or when they have not been compiled from their source with adequate compilation options, which happens in some Linux distributions, may silently lose some of the file metadata, when copying, moving or archiving.
  - compsciphd 1y ago
    there's no reason that you have to create multiple files for par2 if you are storing the recovery data with the protected data. It only was split into files of varying size due to its source in protecting usenet posted binaries to allow users to not have to download the entire recovery data when they only needed a portion.

antonkochubey 1y ago

Unfortunately some SSD controllers plainly refuse to read data they consider corrupted, even if you have extra parity that could potentially restore corrupted data, your entire drive might refuse to read.

lazide 1y ago
Huh? The issue being discussed is random blocks, yes? If your entire drive is bricked, that is an entirely different issue.
- jeremyvisser 1y ago
  Here’s the thing. That SSD controller is the interface between you and those blocks. If it decides, by some arbitrary measurement, as defined by some logic within its black box firmware, that it should stop returning all blocks, then it will do so, and you have almost no recourse. This is a very common failure mode of SSDs. As a consequence of some failed blocks (likely exceeding a number of failed blocks, or perhaps the controller’s own storage failed), drives will commonly brick themselves. Perhaps you haven’t seen it happen, or your SSD doesn’t do this, or perhaps certain models or firmwares don’t, but some certainly do, both from my own experience, and countless accounts I’ve read elsewhere, so this is more common than you might realise.
  - reactordev 1y ago
    This is correct, you still have to go through firmware to gain access to the block/page on “disk” and if the firmware decides the block is invalid than it fails. You can sidestep this by bypassing the controller on a test bench though. Pinning wires to the chips. At that point it’s no longer an SSD.
  - lazide 1y ago
    Yes, and? HDD controllers dying and head crashes are a thing too. At least in the ‘bricked’ case it’s a trivial RMA - corrupt blocks tend to be a harder fight. And since ‘bricked’ is such a trivial RMA, manufacturers have more of an incentive to fix it or go broke, or avoid it in the first place. This is why backups are important now; and always have been.
    - mort96 1y ago
      We're not talking about the SSD controller dying. The SSD controller in the hypothetical situation that's being described is working as intended.
      - lazide 1y ago
        Not as far as I can tell, where intended is ‘as any user would reasonably expect’. Bricking the drive (can’t even read) because of too many errors is not what most users would ever want. Some would (enterprise maybe), but even then they’d want deterministic data deletes too, which doesn’t sound like are happening.
        mort96 1y ago
        You can argue that controllers shouldn't behave that way. But they do, it's not a bug, and it's not a dead controller. It's a perfectly functional controller's response to dead blocks.
        lazide 1y ago

mywittyname 1y ago

This is fine, but I'd prefer an option to transparently add parity bits to the drive, even if it means losing access to capacity. Personally, I keep backups of critical data on a platter disk NAS, so I'm not concerned about losing critical data off of an SSD. However, I did recently have to reinstall Windows on a computer because of a randomly corrupted system file. Which is something this feature would have prevented.

casenmgreen 1y ago

Thank you for this. I had no knowledge of pax, or that par was an open standard, and I care about what they help with. Going to switch over to using both in my backups.

adrian_b 1y ago
For handling pax archives, I recommend the "libarchive" package, which is available in many Linux distributions, even if it originally comes from FreeBSD. Among other utilities, it installs the "bsdtar" program, which you can use in your scripts like this: bsdtar --create --verbose --format=pax --file="${DIRECTORY}".pax "${DIRECTORY}" || exit And for extraction: bsdtar --extract --preserve-permissions --verbose --file="${DIRECTORY}".pax The bsdtar program has options for compressing and/or encrypting the archives, for the case when you do not want to use directly other external programs. "par2create" creates multiple files from the (normally compressed and encrypted) archive file, for storing the added redundancy. I make a directory where I move those files, then I use a second time bsdtar (obviously without any compression or encryption) to aggregate those files in a single archive with redundancy. The libarchive package can also be taken directly from: https://github.com/libarchive/libarchive https://github.com/libarchive/libarchive "libarchive" handles correctly all kinds of file metadata, e.g. extended file attributes and high-resolution file timestamps, which not all archiving utilities do. Many Linux utilities, with the default command-line options or when they have not been compiled from their source with adequate compilation options, which happens in some Linux distributions, may silently lose some of the file metadata, when copying, moving or archiving.
- compsciphd 1y ago
  there's no reason that you have to create multiple files for par2 if you are storing the recovery data with the protected data. It only was split into files of varying size due to its source in protecting usenet posted binaries to allow users to not have to download the entire recovery data when they only needed a portion.