6 ms·

For a homelab use-case, I backed out of creating a ZFS mirrored storage pool for network storage. It wasn't that the steps were too complicated or that ZFS let

For a homelab use-case, I backed out of creating a ZFS mirrored storage pool for network storage. It wasn't that the steps were too complicated or that ZFS let me down as such. Just that RAID in general and ZFS put my files in a blackbox, and that I didn't really need a significant boost in network reads. I settled for plain old ext4 and periodic rsync mirror files to another disk (I don't mind some interim data loss + writes are fast). Use SFTP or sshfs for accessing the drive.

isatty 5y ago
Any RAID solution that does block/bit level striping will be a “blackbox” as you describe it. That’s not a bad thing (nor is your approach) but pros and cons.
- dkh 5y ago
  Yup, and as far as "blackbox" solutions go, if something horrible should happen to your system, you want something that works like ZFS as opposed to, say, some high-performance but proprietary hardware RAID. As someone who has a couple times needed to rip 100tb of ZFS disks out of something, put them into a different machine with a different architecture or OS in some random order, and access everything without having lost anything, it's hard for me to overstate how great it is that such a thing is even possible.
  - pdimitar 5y ago
    How exactly did you do that? I'm learning ZFS still and I'm planning to move some of the disks in my first NAS to a better machine. Which ZFS commands?
    - Filligree 5y ago
      Nothing special is necessary. All you need to do is import the pool as normal. There’s a potential problem, in that you can’t import a poll on an OS version that lacks the feature flags that are enabled on the pool. The way to solve that is to choose a common subset when creating the pool.
      - sennight 5y ago
        Unless they fixed it, another edge case is a change in processor endianness. I started working on a patch a long time ago, but it necessitated changes to code that had remained unchanged since the original release... not a blamelog I'm totally comfortable with.
        Filligree 5y ago
        I hadn’t heard of that one. Does send |receive still work?
        sennight 5y ago
        Nope, the system would instantly panic - presumably in order to protect data integrity... imagine the damage possible otherwise. It was actually a fairly ponderous debugging effort, as I remember it, because it was occurring on an already shaky POWER9 setup I was using to port other stuff. ZFS worked great for any vdev setup on any OS compiled with the same endianness, but plugging in so much as a thumb drive previously provisioned from a different endian host... insta-console-vomit. Definitely an uncommon problem though :)
        rincebrain 5y ago
        What? CPU endianness switches work fine, IME - I have pools I originated on SPARC Solaris that work fine on x86, I just got a patch landed for an edge case in one recent feature not interoperating properly between endiannesses but other than that it's worked fine for me. What kind of issue did you see?
        sennight 5y ago
        > CPU endianness switches work fine Thats great news, hopefully that rare potential problem got addressed in the last year or so. Steps to recreate: compile linux (maybe FreeBSD as well, that is a little more fuzzy) for LE. Boot new kernel and setup a new pool. Recompile for BE. Boot new kernel and enjoy bootloop... lol, actually things are starting to come more into focus... page size played a role in the whole thing. As I said, edge case :) No potential for data loss though.
    - Volundr 5y ago
      `zpool export` on the machine your taking the array out of and `zpool import <pool name>` on the receiving end will do the job in most cases.
  - flyinghamster 5y ago
    Hopefully, the pool was exported before you moved the disks. I came very close to losing data (but ultimately didn't) when I yanked a pool out of an OpenIndiana system and tried to import it on Ubuntu without exporting it first. ZFS really didn't like finding the disks on devices with radically different names than where it had left off. It ended up re-importing OK when I brought it back to the OpenIndiana box. Whew! I then did an export, and Ubuntu was then able to import the pool.
    - boomboomsubban 5y ago
      When you import without exporting, all you risk losing is the last few seconds of data transactions. If a motherboard dying was enough to lose all your data people wouldn't really recommend ZFS. So hopefully the pool was exported, but the worst case scenario isn't very bad.
    - flexd 5y ago
      I make my pools with the serial number /dev/disk/by-id/ names, like ata-WDC_WD30EFRX-68EUZN0_WD-XXXXXXXXXXX. That is not going to change between different machines as far as I know, I might be wrong about this. I made some bad choices (in hindsight) when making this pool for my uses, but it's still going strong many years later. [1] This current pool (which is now old, 50k+ power on hours on these disk) has survived a motherboard dying randomly and one or two SSD failures with the OS on. Always just installed a OS on a new SSD and it's been picked up just fine. [1] I went with RAID-Z2 for 6x3TB WD Reds where I probably should have made mirrored pools or something like it to gain more space? It's been a while since I looked at it. Can't really expand this pool or add more storage without replacing each disk one by one with something bigger. I could make another pool with new disks but I'd lose another 20% to parity.
      - flyinghamster 5y ago
        I use those names as well, at least now that I entirely use ZoL. That wasn't an option back in the day with OpenSolaris and Illumos. You'd be using names like c0t0d0 instead, and woe to you if it found the wrong disks in the wrong places (it would just refuse to import claiming the pool was faulted). Caveat: things may have changed since then - it was at least six years ago.
sennight 5y ago
> I settled for plain old ext4 and periodic rsync mirror files to another disk This isn't a mirror, not in the sense that most would understand. No performance boost, no protection from bit rot, unnecessarily wasteful in every potential metric. If you wanted to intentionally design a system to propagate errors and render backups useless - you would start with this kind of setup. It makes sense to avoid all the RAID related problems associated with hardware solutions... but I'm drawing a blank on rational reasons to do what you've described. Super weird boot manager + physical space constraints?
- sirius87 5y ago
  Two standard external USB disks of different ages, connected to a powered USB hub attached to a ROC-Raspberry Pi. The older one is the "live" one that serves all data requests. The newer disk is usually attached to the hub, but it's also disconnected during summer. If the "live" disk fails, the other disk will eventually replace it. > render backups useless Like I mentioned, data loss isn't a concern. This is mainly hosting code repositories and media files. What's my use-case? My laptop, that has a paltry 256GB SSD, keeps running out of disk space, and I often find myself plonking files that aren't super critical on the network drive.
  - sennight 5y ago
    > but it's also disconnected during summer. I knew there was a reason I was getting a "can't send mail more than 500 miles" vibe... Are we talking small form factor spinning rust, or thumb drives? Because if you thought that ZFS was a black box - look into flash wear leveling algorithms, I'm hard pressed to think of any storage media more prone to spontaneous bitflips. Unless you are checksuming before and after every movement of data - your files are going to silently get corrupted in ways you won't notice, until something breaks. Anyway, I've had way more instances of bitflips than drive failures - and I've had to deal with that kind of data corruption escaping detection and making it into backup archives. That is why I obnoxiously promote ZFS to the degree I do... it totally eliminates that risk automatically. No joke, this thread prompted me to run a manual pool scrub a month in advance for an 8TB archive pool. Since it is archival, it gets few reads - which means it gets few automatic checksum verifications outside of the scheduled bi-monthly scrub. Well the scrub is only half way through, but it already shows that at some point in the last month a 128KB block on a single 3.5" disk got silently corrupted. Data corruption nipped in the bud, thanks to ZFS <insert whatever Sun's jingle was>.

isatty 5y ago

Any RAID solution that does block/bit level striping will be a “blackbox” as you describe it. That’s not a bad thing (nor is your approach) but pros and cons.

dkh 5y ago
Yup, and as far as "blackbox" solutions go, if something horrible should happen to your system, you want something that works like ZFS as opposed to, say, some high-performance but proprietary hardware RAID. As someone who has a couple times needed to rip 100tb of ZFS disks out of something, put them into a different machine with a different architecture or OS in some random order, and access everything without having lost anything, it's hard for me to overstate how great it is that such a thing is even possible.
- pdimitar 5y ago
  How exactly did you do that? I'm learning ZFS still and I'm planning to move some of the disks in my first NAS to a better machine. Which ZFS commands?
  - Filligree 5y ago
    Nothing special is necessary. All you need to do is import the pool as normal. There’s a potential problem, in that you can’t import a poll on an OS version that lacks the feature flags that are enabled on the pool. The way to solve that is to choose a common subset when creating the pool.
    - sennight 5y ago
      Unless they fixed it, another edge case is a change in processor endianness. I started working on a patch a long time ago, but it necessitated changes to code that had remained unchanged since the original release... not a blamelog I'm totally comfortable with.
      - Filligree 5y ago
        I hadn’t heard of that one. Does send |receive still work?
        sennight 5y ago
        Nope, the system would instantly panic - presumably in order to protect data integrity... imagine the damage possible otherwise. It was actually a fairly ponderous debugging effort, as I remember it, because it was occurring on an already shaky POWER9 setup I was using to port other stuff. ZFS worked great for any vdev setup on any OS compiled with the same endianness, but plugging in so much as a thumb drive previously provisioned from a different endian host... insta-console-vomit. Definitely an uncommon problem though :)
      - rincebrain 5y ago
        What? CPU endianness switches work fine, IME - I have pools I originated on SPARC Solaris that work fine on x86, I just got a patch landed for an edge case in one recent feature not interoperating properly between endiannesses but other than that it's worked fine for me. What kind of issue did you see?
        sennight 5y ago
        > CPU endianness switches work fine Thats great news, hopefully that rare potential problem got addressed in the last year or so. Steps to recreate: compile linux (maybe FreeBSD as well, that is a little more fuzzy) for LE. Boot new kernel and setup a new pool. Recompile for BE. Boot new kernel and enjoy bootloop... lol, actually things are starting to come more into focus... page size played a role in the whole thing. As I said, edge case :) No potential for data loss though.
  - Volundr 5y ago
    `zpool export` on the machine your taking the array out of and `zpool import <pool name>` on the receiving end will do the job in most cases.
- flyinghamster 5y ago
  Hopefully, the pool was exported before you moved the disks. I came very close to losing data (but ultimately didn't) when I yanked a pool out of an OpenIndiana system and tried to import it on Ubuntu without exporting it first. ZFS really didn't like finding the disks on devices with radically different names than where it had left off. It ended up re-importing OK when I brought it back to the OpenIndiana box. Whew! I then did an export, and Ubuntu was then able to import the pool.
  - boomboomsubban 5y ago
    When you import without exporting, all you risk losing is the last few seconds of data transactions. If a motherboard dying was enough to lose all your data people wouldn't really recommend ZFS. So hopefully the pool was exported, but the worst case scenario isn't very bad.
  - flexd 5y ago
    I make my pools with the serial number /dev/disk/by-id/ names, like ata-WDC_WD30EFRX-68EUZN0_WD-XXXXXXXXXXX. That is not going to change between different machines as far as I know, I might be wrong about this. I made some bad choices (in hindsight) when making this pool for my uses, but it's still going strong many years later. [1] This current pool (which is now old, 50k+ power on hours on these disk) has survived a motherboard dying randomly and one or two SSD failures with the OS on. Always just installed a OS on a new SSD and it's been picked up just fine. [1] I went with RAID-Z2 for 6x3TB WD Reds where I probably should have made mirrored pools or something like it to gain more space? It's been a while since I looked at it. Can't really expand this pool or add more storage without replacing each disk one by one with something bigger. I could make another pool with new disks but I'd lose another 20% to parity.
    - flyinghamster 5y ago
      I use those names as well, at least now that I entirely use ZoL. That wasn't an option back in the day with OpenSolaris and Illumos. You'd be using names like c0t0d0 instead, and woe to you if it found the wrong disks in the wrong places (it would just refuse to import claiming the pool was faulted). Caveat: things may have changed since then - it was at least six years ago.

sennight 5y ago

> I settled for plain old ext4 and periodic rsync mirror files to another disk This isn't a mirror, not in the sense that most would understand. No performance boost, no protection from bit rot, unnecessarily wasteful in every potential metric. If you wanted to intentionally design a system to propagate errors and render backups useless - you would start with this kind of setup. It makes sense to avoid all the RAID related problems associated with hardware solutions... but I'm drawing a blank on rational reasons to do what you've described. Super weird boot manager + physical space constraints?

sirius87 5y ago
Two standard external USB disks of different ages, connected to a powered USB hub attached to a ROC-Raspberry Pi. The older one is the "live" one that serves all data requests. The newer disk is usually attached to the hub, but it's also disconnected during summer. If the "live" disk fails, the other disk will eventually replace it. > render backups useless Like I mentioned, data loss isn't a concern. This is mainly hosting code repositories and media files. What's my use-case? My laptop, that has a paltry 256GB SSD, keeps running out of disk space, and I often find myself plonking files that aren't super critical on the network drive.
- sennight 5y ago
  > but it's also disconnected during summer. I knew there was a reason I was getting a "can't send mail more than 500 miles" vibe... Are we talking small form factor spinning rust, or thumb drives? Because if you thought that ZFS was a black box - look into flash wear leveling algorithms, I'm hard pressed to think of any storage media more prone to spontaneous bitflips. Unless you are checksuming before and after every movement of data - your files are going to silently get corrupted in ways you won't notice, until something breaks. Anyway, I've had way more instances of bitflips than drive failures - and I've had to deal with that kind of data corruption escaping detection and making it into backup archives. That is why I obnoxiously promote ZFS to the degree I do... it totally eliminates that risk automatically. No joke, this thread prompted me to run a manual pool scrub a month in advance for an 8TB archive pool. Since it is archival, it gets few reads - which means it gets few automatic checksum verifications outside of the scheduled bi-monthly scrub. Well the scrub is only half way through, but it already shows that at some point in the last month a 128KB block on a single 3.5" disk got silently corrupted. Data corruption nipped in the bud, thanks to ZFS <insert whatever Sun's jingle was>.