4 ms·

> data needs to be validated regularly and cycled to a new media every 5-10 years in order to ensure it’s safe and easily accessible This is what I do. I hate

• justinclift 4y ago
  Hang on, that sounds like you're copying critical data to a single disk? That's not actually the case is it?!?!?!
  • Juke_Ellington 4y ago
    It makes sense if you keep the old disks around until they kick it. You can always have 3-5 copies around in a decently readable state
    • favorited 4y ago
      If you're not periodically checking the data for corruption, and only moving from a single drive to a replacement single drive, eventually you'll have some corrupted data which gets copied from drive to drive without you noticing. 4 TB drives are dirt cheap. If someone would really be "lost" without this data, having some redundancy would be inexpensive and easy.
      • lazide 4y ago
        If stored on ZFS, at least it would be validated each time it was copied.
        • kadoban 4y ago
          If you keep a ZFS mirror of the most-recent N drives, and store them separately, that should be pretty good. I forget how ZFS behaves if a mirror is missing drives though, if some are off-site. Hopefully it's smart enough to let you do that and just rotate through.
          • lazide 4y ago
            ZFS doesn’t handle that well generally. You’d have better luck making a full (manual) copy most likely (ZFS send/recv, or mirror then un-mirror even better), assuming you’d run a scrub after. Or manually make checksum files I guess. I’ve done that, less ‘magic’ that way.
            • jjav 4y ago
              > ZFS doesn’t handle that well generally. Can you expand on that? The purpose and benefit of a zfs mirror is that every disk in the mirror contains everything. So it's expensive in space usage, but great for reliability. As long as any one of the disks in a mirror survives, you can recover everything.
              • lazide 4y ago
                The issue is that a mirror in ZFS is assumed to be ‘live’, unless you explicitly sever the relationship (unmirror it). Having tons of unreachable devices (because you have them sitting on a shelf somewhere) makes things unhappy operationally. So if you have one ‘live’ copy, create a mirror, then sever the relationship before taking the external device offline, it’s fine. Even taking it offline sometimes when it’s a normal live mirror is fine (though it will complain of course, depending on how it’s done). But if you want to make copies, so add a bunch of mirrors, taking them offline ‘forever’ (really over multiple boot cycles) it makes the running ZFS instance angry because it expects those mirror copies to still be accessible somewhere (after all, ZFS is a live filesystem) and will keep trying to use/access them, and won’t be able to. I don’t think you’ll lose data or brick anything, but it will be a hassle at some point. Also, if you reconnect those old instances, it will try to resilver the older copies (and hence modify them). Which is not what I would want for an archive, unless I manually told it to do so anyway. Which is easy enough to do of course even after severing the mirror relationship later, albeit with more disk I/O. I’ve done this kind of archiving before, there is built in ZFS support for making a new zpool when splitting mirrors this way, and it works well. The way I ended up doing it was primary/secondary disks (both external hard drives). Setup as a mirror, copy archive data over. Split the mirror, now you have two (slightly differently named) unmirrored zpools with identical data that can both be mounted at the same time, scrubbed independently, etc. Having one of them ‘live’ and the other one the archived copy (on the external disk) would be trivial, and allows you to zpool export the archived/external copy, name it with the date it was made, etc. - which is what you want to make everyone happy. P.S. if doing this, be REALLY careful about what kernel/OS/ZFS features you are using with your pools or you can end up with an unmountable ZFS copy! (As I found out). Built-in ZFS encryption is a major footgun here. Better to use dmcrypt style encryption for several reasons.
        • jjav 4y ago
          Also, run zpool scrub regularly to detect and repair any corruption early.
• jjav 4y ago
  > data needs to be validated regularly and cycled to a new media every 5-10 years in order to ensure it’s safe and easily accessible I used to do that but found it to be a gamble. I have files back to the 80s, so I rotated them from 5.25" floppies to 3.5" floppies to zip drives to CD-R and the DVD-R. But it's a fragile system, files can get corrupted somewhere along the line and if I didn't migrate in time it can be hard to go back. For instance I lost a handful of files during the iomega zip drive phase when the drive died and I had no way to recover (and the files weren't that important to try to source a new iomega drive). Now I simply keep everything online in a big zfs mirror pool.
• ilyt 4y ago
  You'd at least need to check checksums of all of them post-copy and preferably store it in error-resistant (RAID5/6 or other error correction) way. Else you might just be copying the errors that sneak in. It might not even be the source hard drive producing it just transient bit flip in RAM