4 ms·

This is primarily aimed at those running very large vdevs comprised of lots of disks. FTA: > The dRAID offers a solution for large arrays, vdevs with fewer tha

This is primarily aimed at those running very large vdevs comprised of lots of disks. FTA: > The dRAID offers a solution for large arrays, vdevs with fewer than 20 spindles will have limited benefits from the new option. The performance and resilver result will be similar to RAIDZ for small numbers of spindles. If you're doing a 6-8 disk array, this probably isn't relevant to your interests. If you're doing a 20+ disk array, well firstly, damn, you've been lucky if you've avoided any serious issues. Secondly, you really want to look into this ASAP.

tutfbhuf 6y ago
> If you're doing a 20+ disk array ... Then you might want to consider ceph.
- secabeen 6y ago
  You might, but you might not. I have an 140 disk array that just works. Ceph would require multiple nodes, lots of tuning, maintenance, etc.
  - tutfbhuf 6y ago
    But multiple nodes offers you resilience against node failures not just drive failures. And with Ceph you have the flexibility to add or remove single disks as you like. And you have multiple storage interface not just FS, but also S3 or block device if you like.
    - mbreese 6y ago
      But that's a lot of extra overhead too. You need monitors, OSDs, metadata servers, etc... If you're comfortable with the node being a single point of failure, you can do a lot with a single storage node connected to a JBOD and serving data over NFS. And even if the node is the unit of failure, you can attach a SAS JBOD to multiple servers if you'd like. In my position, in addition to the added cost, I just don't have the physical rack space for the extra servers required for ceph.
      - tutfbhuf 6y ago
        > don't have the physical rack space for the extra servers required for ceph Sure, it depends but the 140 drives (mentioned by secabeen) alone will take up a huge amount of space. Maybe I just wanted to point out that there might be a point when you have maxed out the storage capability of a single node and might want to consider a distributed storage setup.
        secabeen 6y ago
        They take about 14U total. (3x4U 60-drive JBOds, and a 2U server.) Big, but not huge.
    - secabeen 6y ago
      And I need none of those things. This is bulk storage over CIFS. I have two systems in two different buildings, and we bought enough capacity to exceed the projected lifetime of the system. It should run till drives start reaching EOL, at which point, we'll migrate to a replacement, or shut it down.
mbreese 6y ago
I’m incredibly interested in this. I run multiple 60+ disk arrays, but never with enough of a budget for the extra servers required for a better distributed system. Basically these are single large JBODs that store large genomics data. (Cheap and big are the only priorities, speed is a distant third). Right now, I use ZFS with multiple raidz2 or raidz3 vdevs with a number of hot spares waiting to take over. Resilvering can take a day for a 4TB disk. I’m terrified how long a 12TB disk will take (hence the move to raidz3). Disk loss is a normal fact of life, and it’s something we prepare for. But this would help me sleep better at night. I’m seriously excited about this and I’m definitely the target audience for this.
- secabeen 6y ago
  I'm in the same boat, but I don't run hot-spares. If a drive fails, I want a human in the loop before a resilver begins. Ive seen too many communication/cabling errors over the years, and I'd rather try to confirm the root cause before assuming that the problem is a bad disk.
  - rsync 6y ago
    I really like this line of thought and I think your instinct is very good here. Remember - if you are in the middle of a potentially failing array, every single bit of parity data is pure gold and if you fail out a drive, that drive worth of parity is gone forever. So if you fail out a drive but it was actually weird bus errors getting spammed by some other drive that is actually failing, now you're in trouble ... While on the topic, in filesystem emergencies, don't forget the ability to mount, and even run, read-only. Your filesystem might be puking on mount and backing you into a corner, but perhaps if it were mounted read-only ...
- rsync 6y ago
  Even with 16TB disks I would not worry about 12 or 15 disk vdevs running in raidz3. Which is to say, beat the shit out of the disks before deploying them[1] to weed out early failures, then aggressively fail them out at the first sign of trouble[2]. We don't use hot spares - we just very aggressively pull drives that give us any meaningful smart errors ... [1] perhaps with the badblocks tool ... [2] smartctl

tutfbhuf 6y ago

> If you're doing a 20+ disk array ... Then you might want to consider ceph.

secabeen 6y ago
You might, but you might not. I have an 140 disk array that just works. Ceph would require multiple nodes, lots of tuning, maintenance, etc.
- tutfbhuf 6y ago
  But multiple nodes offers you resilience against node failures not just drive failures. And with Ceph you have the flexibility to add or remove single disks as you like. And you have multiple storage interface not just FS, but also S3 or block device if you like.
  - mbreese 6y ago
    But that's a lot of extra overhead too. You need monitors, OSDs, metadata servers, etc... If you're comfortable with the node being a single point of failure, you can do a lot with a single storage node connected to a JBOD and serving data over NFS. And even if the node is the unit of failure, you can attach a SAS JBOD to multiple servers if you'd like. In my position, in addition to the added cost, I just don't have the physical rack space for the extra servers required for ceph.
    - tutfbhuf 6y ago
      > don't have the physical rack space for the extra servers required for ceph Sure, it depends but the 140 drives (mentioned by secabeen) alone will take up a huge amount of space. Maybe I just wanted to point out that there might be a point when you have maxed out the storage capability of a single node and might want to consider a distributed storage setup.
      - secabeen 6y ago
        They take about 14U total. (3x4U 60-drive JBOds, and a 2U server.) Big, but not huge.
  - secabeen 6y ago
    And I need none of those things. This is bulk storage over CIFS. I have two systems in two different buildings, and we bought enough capacity to exceed the projected lifetime of the system. It should run till drives start reaching EOL, at which point, we'll migrate to a replacement, or shut it down.

mbreese 6y ago

I’m incredibly interested in this. I run multiple 60+ disk arrays, but never with enough of a budget for the extra servers required for a better distributed system. Basically these are single large JBODs that store large genomics data. (Cheap and big are the only priorities, speed is a distant third). Right now, I use ZFS with multiple raidz2 or raidz3 vdevs with a number of hot spares waiting to take over. Resilvering can take a day for a 4TB disk. I’m terrified how long a 12TB disk will take (hence the move to raidz3). Disk loss is a normal fact of life, and it’s something we prepare for. But this would help me sleep better at night. I’m seriously excited about this and I’m definitely the target audience for this.

secabeen 6y ago
I'm in the same boat, but I don't run hot-spares. If a drive fails, I want a human in the loop before a resilver begins. Ive seen too many communication/cabling errors over the years, and I'd rather try to confirm the root cause before assuming that the problem is a bad disk.
- rsync 6y ago
  I really like this line of thought and I think your instinct is very good here. Remember - if you are in the middle of a potentially failing array, every single bit of parity data is pure gold and if you fail out a drive, that drive worth of parity is gone forever. So if you fail out a drive but it was actually weird bus errors getting spammed by some other drive that is actually failing, now you're in trouble ... While on the topic, in filesystem emergencies, don't forget the ability to mount, and even run, read-only. Your filesystem might be puking on mount and backing you into a corner, but perhaps if it were mounted read-only ...
rsync 6y ago
Even with 16TB disks I would not worry about 12 or 15 disk vdevs running in raidz3. Which is to say, beat the shit out of the disks before deploying them[1] to weed out early failures, then aggressively fail them out at the first sign of trouble[2]. We don't use hot spares - we just very aggressively pull drives that give us any meaningful smart errors ... [1] perhaps with the badblocks tool ... [2] smartctl