3 ms·

To add to the downsides, you can't expand RAIDZ vdevs. If you start with a 6 disk RAIDZ2 and want to add a couple more drives, you can't. The only way to add c

by Veratyr 10y ago

To add to the downsides, you can't expand RAIDZ vdevs. If you start with a 6 disk RAIDZ2 and want to add a couple more drives, you can't. The only way to add capacity to the pool is to add an entire new vdev. Unfortunately I don't think there's an expandable file system that handles parity/erasure coding as well as bitrot that can be used right now. BTRFS might be usable after the RAID5/6 rewrite and Ceph might be more usable after Bluestore comes along (at the moment erasure coding performance seems to be pretty bad, even with an SSD cache) but in the present there's nothing. I really hope https://www.patreon.com/bcachefs https://www.patreon.com/bcachefs manages to fix this.

ThatPlayer 10y ago
I've been looking at SnapRAID and UnRAID which I believe use their own parity and handles expansions. Mostly because I'm currently using BTRFS RAID6 and might move off it. I also have high hopes for bcachefs. Currently using bcache on my laptop and it's been great. BTRFS doesn't really have an easy SSD cache like ZFS, and it's a hassle to convert over to a Bcache backed BTRFS setup.
ymse 10y ago
> Ceph might be more usable after Bluestore comes along (at the moment erasure coding performance seems to be pretty bad, even with an SSD cache) I just saturated a 10Gb link in a k=4,m=2 EC configuration on Ceph (Haswell). Too lazy to set up concurrent clients. What do you mean by pretty bad? This is a Hammer cluster without SSD journals or cache. Bluestore was recently released with Jewel, but shouldn't affect EC performance drastically, if at all.
- Veratyr 10y ago
  Maybe I'm doing something horribly wrong. My setup was: - 6x4TB HDD - 2x120GB SSD - OSD per disk - k=4,m=2 erasure coded HDD pool - 1 copy replicated (not) SSD pool - SSD pool writeback cache for HDD pool I was consistently seeing the SSD writeback cache fill up then the disks thrashed at 100% IO utilization, limiting incoming writes to ~40MB/s from memory.
  - ymse 10y ago
    Oops, didn't see this reply until now. It's no longer recommended to use a SSD cache pool, much for the reasons you describe. You will be a lot better off using bcache or similar, on top of the OSD device.
goda90 10y ago
You can expand capacity by replacing all the drives with bigger ones. It's obviously expensive and time consuming(has to be done one disk at a time so that each one can be resilvered) but it is possible.
Terribledactyl 10y ago
The amount and type of work that would be required to take a 6disk to a 7disk (in place) is quite hard. If you can take some downtime, you'd really want to snap, send to a new larger system, go down, do the final copy, and come back up. That or work with more vdevs, but then you might have unevenly striped data. And there are filesystems that are expandable and meet your other requirements, they just aren't cheap. IBM GPFS is one I work with regularly.
- teraflop 10y ago
  Mdadm (Linux's built-in software RAID) can expand a RAID array online, and has been able to do so for about a decade. It's not fundamentally that complex of an operation, but there are enough details to be careful of that I don't see why it makes sense to reimplement it in every filesystem.
  - Terribledactyl 10y ago
    It's the checksumming, striping, and parity, especially in context of CoW snapshots that complicate this operation.
    - mappu 10y ago
      LVM2 is a filesystem-agnostic layer that gives you CoW snapshots on top of mdadm's striping/parity. From your list, that just leaves checksumming for bitrot, which you can do at the RAID level via mdadm check, or at the filesystem level with sha1sum --tag / sha1sum -c (although maybe that's not as nice as ZFS/btrfs).
      - Annatar 10y ago
        RAID controllers cannot be relied upon to do check sums, as most RAID controllers do not possess this capability, and of those which do, their firmware is opaque and buggy, and therefore cannot be trusted. If you care about your data, never use hardware RAID, and always use ZFS or Oracle ASM. mdadm(8) is a good start on GNU/Linux if one is forced to run on an (older) operating system version with only ext3 or XFS, however while it provides administration consistency and scales to many systems, it does not provide data checksumming, and therefore no data corruption detection and no self healing capabilities like Oracle ASM or like (Open)ZFS.
nmjohn 10y ago
> To add to the downsides, you can't expand RAIDZ vdevs. > If you start with a 6 disk RAIDZ2 and want to add a couple more drives, you can't. The only way to add capacity to the pool is to add an entire new vdev. This is mostly true, and in many situations for all practical purposes is true. Not fully understanding this recently cost me a few hundred dollars (2x 8TB external drives) when I needed to expand my 4 disk raidz2 to a 6 disk raidz2. However, what _is_ possible is expanding a pool by incrementally replacing drives with larger capacity equivalents. This wouldn't work in my situation as I already was using the largest consumer drives available - but the next time I need to expand in a few years it may be a possibility. > ... expandable file system that handles parity/erasure coding as well as bitrot that can be used right now Unfortunately I believe this to be the case. However in practice, so long as you know up front your data store is not expandable, it's fairly easy to work around without it being too much of a hassle. If having two zpools when you need to expand is a burden for your use case because the data necessarily will be partitioned across pools (ie: in multiple directories), there are tools that will present multiple filesystems as a single mount point to linux. Alternatively, mirrored vdevs [0] as opposed to a classic raidz2 configuration are much easier to expand, as you only have to replace 2 drives to expand your total storage space (a single vdev) as opposed to having to replace all of your drives with higher capacity drives as with raidz2. > and Ceph might be more usable After all that rambling about zfs, the main thing I wanted to touch on: I'd caution most people to not use ceph unless you already know you need it. For the average user, it is introducing so much unnecessary complexity that will just create future headaches when using it in a non-distributed manner (if zfs is an alternative, it must be non-distributed). Which isn't to say ceph isn't useful, it's just built to be optimal at solving a slightly different problem than a filesystem on a single compute can solve. [0]: http://jrs-s.net/2015/02/06/zfs-you-should-use-mirror-vdevs-not-raidz/ http://jrs-s.net/2015/02/06/zfs-you-should-use-mirror-vdevs-...
MikusR 10y ago
REFS on top of Storage Spaces
tmd83 10y ago
Yeah. I have read about zfs to some degree before but somehow missed this until a couple of days ago. For all its flexibility this looks a very bad limitation. I think this and rebuild time (again related to vdev width) makes zfs very easy to mess up. The goal of bachefs seem to very lofty though there isn't much detail on the implication of raid design. It seemed to suggest it solves the zfs fragmentation problem, gives better r/w performance with a simpler codebase. Anyone who understand the programmers guide well enough to comment on the viability of the architecture? Does it really solves all the problem of a non-distributed huge filesystem capable of working from a single disk to a big array?

by Veratyr 10y ago

ThatPlayer 10y ago

I've been looking at SnapRAID and UnRAID which I believe use their own parity and handles expansions. Mostly because I'm currently using BTRFS RAID6 and might move off it. I also have high hopes for bcachefs. Currently using bcache on my laptop and it's been great. BTRFS doesn't really have an easy SSD cache like ZFS, and it's a hassle to convert over to a Bcache backed BTRFS setup.

ymse 10y ago

> Ceph might be more usable after Bluestore comes along (at the moment erasure coding performance seems to be pretty bad, even with an SSD cache) I just saturated a 10Gb link in a k=4,m=2 EC configuration on Ceph (Haswell). Too lazy to set up concurrent clients. What do you mean by pretty bad? This is a Hammer cluster without SSD journals or cache. Bluestore was recently released with Jewel, but shouldn't affect EC performance drastically, if at all.

Veratyr 10y ago
Maybe I'm doing something horribly wrong. My setup was: - 6x4TB HDD - 2x120GB SSD - OSD per disk - k=4,m=2 erasure coded HDD pool - 1 copy replicated (not) SSD pool - SSD pool writeback cache for HDD pool I was consistently seeing the SSD writeback cache fill up then the disks thrashed at 100% IO utilization, limiting incoming writes to ~40MB/s from memory.
- ymse 10y ago
  Oops, didn't see this reply until now. It's no longer recommended to use a SSD cache pool, much for the reasons you describe. You will be a lot better off using bcache or similar, on top of the OSD device.

goda90 10y ago

You can expand capacity by replacing all the drives with bigger ones. It's obviously expensive and time consuming(has to be done one disk at a time so that each one can be resilvered) but it is possible.

Terribledactyl 10y ago

The amount and type of work that would be required to take a 6disk to a 7disk (in place) is quite hard. If you can take some downtime, you'd really want to snap, send to a new larger system, go down, do the final copy, and come back up. That or work with more vdevs, but then you might have unevenly striped data. And there are filesystems that are expandable and meet your other requirements, they just aren't cheap. IBM GPFS is one I work with regularly.

teraflop 10y ago
Mdadm (Linux's built-in software RAID) can expand a RAID array online, and has been able to do so for about a decade. It's not fundamentally that complex of an operation, but there are enough details to be careful of that I don't see why it makes sense to reimplement it in every filesystem.
- Terribledactyl 10y ago
  It's the checksumming, striping, and parity, especially in context of CoW snapshots that complicate this operation.
  - mappu 10y ago
    LVM2 is a filesystem-agnostic layer that gives you CoW snapshots on top of mdadm's striping/parity. From your list, that just leaves checksumming for bitrot, which you can do at the RAID level via mdadm check, or at the filesystem level with sha1sum --tag / sha1sum -c (although maybe that's not as nice as ZFS/btrfs).
    - Annatar 10y ago
      RAID controllers cannot be relied upon to do check sums, as most RAID controllers do not possess this capability, and of those which do, their firmware is opaque and buggy, and therefore cannot be trusted. If you care about your data, never use hardware RAID, and always use ZFS or Oracle ASM. mdadm(8) is a good start on GNU/Linux if one is forced to run on an (older) operating system version with only ext3 or XFS, however while it provides administration consistency and scales to many systems, it does not provide data checksumming, and therefore no data corruption detection and no self healing capabilities like Oracle ASM or like (Open)ZFS.

nmjohn 10y ago

> To add to the downsides, you can't expand RAIDZ vdevs. > If you start with a 6 disk RAIDZ2 and want to add a couple more drives, you can't. The only way to add capacity to the pool is to add an entire new vdev. This is mostly true, and in many situations for all practical purposes is true. Not fully understanding this recently cost me a few hundred dollars (2x 8TB external drives) when I needed to expand my 4 disk raidz2 to a 6 disk raidz2. However, what _is_ possible is expanding a pool by incrementally replacing drives with larger capacity equivalents. This wouldn't work in my situation as I already was using the largest consumer drives available - but the next time I need to expand in a few years it may be a possibility. > ... expandable file system that handles parity/erasure coding as well as bitrot that can be used right now Unfortunately I believe this to be the case. However in practice, so long as you know up front your data store is not expandable, it's fairly easy to work around without it being too much of a hassle. If having two zpools when you need to expand is a burden for your use case because the data necessarily will be partitioned across pools (ie: in multiple directories), there are tools that will present multiple filesystems as a single mount point to linux. Alternatively, mirrored vdevs [0] as opposed to a classic raidz2 configuration are much easier to expand, as you only have to replace 2 drives to expand your total storage space (a single vdev) as opposed to having to replace all of your drives with higher capacity drives as with raidz2. > and Ceph might be more usable After all that rambling about zfs, the main thing I wanted to touch on: I'd caution most people to not use ceph unless you already know you need it. For the average user, it is introducing so much unnecessary complexity that will just create future headaches when using it in a non-distributed manner (if zfs is an alternative, it must be non-distributed). Which isn't to say ceph isn't useful, it's just built to be optimal at solving a slightly different problem than a filesystem on a single compute can solve. [0]: http://jrs-s.net/2015/02/06/zfs-you-should-use-mirror-vdevs-not-raidz/ http://jrs-s.net/2015/02/06/zfs-you-should-use-mirror-vdevs-...

MikusR 10y ago

REFS on top of Storage Spaces

tmd83 10y ago

Yeah. I have read about zfs to some degree before but somehow missed this until a couple of days ago. For all its flexibility this looks a very bad limitation. I think this and rebuild time (again related to vdev width) makes zfs very easy to mess up. The goal of bachefs seem to very lofty though there isn't much detail on the implication of raid design. It seemed to suggest it solves the zfs fragmentation problem, gives better r/w performance with a simpler codebase. Anyone who understand the programmers guide well enough to comment on the viability of the architecture? Does it really solves all the problem of a non-distributed huge filesystem capable of working from a single disk to a big array?