6 ms·

This might be somewhat off topic but I'm desperate. I've been looking for a way to store files: - Using parity rather than mirroring. I'm happy to deal with so

This might be somewhat off topic but I'm desperate. I've been looking for a way to store files: - Using parity rather than mirroring. I'm happy to deal with some loss of IOPS in exchange for extra usable storage. - That deals with bitrot. - That I can migrate to without somehow moving all of my files somewhere first (i.e. supports addition/removal of disks). - Is stable (doesn't frequently crash or lose data) - Is free or has transparent pricing (not "Contact Sales"). - Ideally, supports arbitrary stripe width (i.e. 2 blocks data + 1 block parity on a 6 disk array) Unfortunately it doesn't appear that a solution for this exists: - ZFS doesn't support addition of disks unless you're happy to put a RAID0 on top of your RAID5/6 and it doesn't support removal of disks at all when parity is involved. It is possible to migrate by putting giant sparse files on the existing storage, filling the filesystem, removing a sparse file, removing a disk from the original FS and "replacing" the sparse file with the actual disk but this is somewhat risky. - BTRFS has critical bugs and has been unstable even with my RAID1 filesystem. - Ceph mostly works but I always seem to run into bugs that nobody else sees. - I couldn't even figure out how to get GlusterFS to create a volume. - MDADM/hardware RAID don't deal with bitrot. - Minio has hard coded N/2 data N/2 parity erasure coding, which destroys IOPS and drastically reduces capacity in exchange for an obscene level of resiliency I don't need. - FlexRAID either isn't realtime or doesn't deal with bitrot depending which version you choose. - Windows storage spaces are slow as a dog (4 disks = 25MB/s write). - QuoByte, the successor to XtreemFS has erasure coding but has "Contact Us" pricing and trial. - Openstack Swift is complex as hell. - BcacheFS seems extremely promising but it's still in development and EC isn't available yet. I'm currently down to fixing bugs in Ceph, modifying Minio, evaluating Tahoe-LAFS and EMC ScaleIO or building my own solution.

sfoskett 9y ago
There kinda isn't such a thing at this point. I sure wish there was! And this is the point behind the article: Wouldn't it be awesome if ZFS had continued developing as the super filesystem it looked like last decade? It would do all this and more! Frankly if ZFS supported re-balancing ("re-RAID-ing") and hybrid pools, it would be pretty much everything we all need. I do hope ZFS or Btrfs or something gets there.
- Veratyr 9y ago
  It's so close though. BTRFS and Ceph both have what I want, they're just unstable. The BcacheFS dev has told me he's close but not quite there yet. Hopefully one of the other solutions will work, otherwise I'll just have to build it.
  - jamiek88 9y ago
    bcachefs really does seem as if it will be our holy grail, we have very similar needs. They have a patreon if you wish to contribute that way.
    - Veratyr 9y ago
      I already contribute and help out with testing and the occasional docs on IRC :)
MikusR 9y ago
For storage spaces try fixed size, you can later enlarge the volumes, faster.
- Veratyr 9y ago
  I was using fixed size, it didn't help. Read speeds were okay but write always capped out at 25MB/s.
davrosthedalek 9y ago
Have you considered snapraid? It's offline though, but seems to tick all your boxes otherwise.
- Veratyr 9y ago
  Yes, I've looked into SnapRAID but I'm not a fan of offline and IIRC I didn't have enough memory to run it last time I tried it out.
rodgerd 9y ago
XFS has gained (some) checksumming and CoW support. At this rate XFS will end up evolving to add all the features btrfs promised before the latter makes them stable.
amorphid 9y ago
Re: contact sales, I totally hear what you're saying. It's not always a bad idea to talk to a sales person, and a good one can steer you towards the options you'd actually want. But yeah, I generally don't want to talk to a salesperson either.
- el_isma 9y ago
  Yes, but would it kill them to just put a pricing there? Nowadays you can get the list price of sending a Rocket into space[1] but not how much some software is going to cost you [1] http://www.spacex.com/about/capabilities http://www.spacex.com/about/capabilities in case you need one
emikulic 9y ago
Have you seen Quantcast File System? http://quantcast.github.io/qfs/ http://quantcast.github.io/qfs/
- Veratyr 9y ago
  I hadn't and it looks good but it'll only allow 3 recovery stripes or none, which isn't optimal for me.
555h 9y ago
You can probably achieve what you're looking for by stacking a few filesystems. For example, you could create a separate ZFS pool/vdev with a single full-disk zvol on each disk. Then use mdadm to create a RAID array of the zvols. Then put ext4 (or whatever) on the mdadm array. I've done something similar for the purpose of getting FDE with ZFS in linux. It can be a little finicky, but it's definitely workable. One ZFS-specific caveat (which may conflict with your desire to get high storage efficiency): you way need to prevent your ZFS pools from filling up too much [1]. You can either enable discard/TRIM on the whole stack, so the top level FS (e.g. ext4) can let ZFS know when a block is actually free. Or alternative just to limit your zvols to 85% (for example) of their respective pools. The latter is my preference, because there was originally a bug with discard in zfs and it's not immediately clear if it's totally fixed (although my fstrim tests seemed to work out fine). [1] https://www.reddit.com/r/zfs/comments/3vtur4/what_exactly_happens_when_you_go_over_80_on/cxqu7uh/ https://www.reddit.com/r/zfs/comments/3vtur4/what_exactly_ha...
- Veratyr 9y ago
  Hmm, that actually sounds workable. I could even format the mdadm device as ZFS too if I really wanted. I am somewhat worried about performance, have you had any issues with that?
  - 555h 9y ago
    I haven't had any issues with performance, but then again my requirement was just "reasonable performance". I ran some quick benchmarks (data below). Obviously this is far from rigorous, but maybe it'll be useful. In previous tests I found that volblocksize=128K was optimal for my stack -- which is why the last benchmarks use that setting. Every additional ZFS filesystem in the stack may reduce storage efficiency (minimum free space requirements [1]; metadata & checksum overhead [2][3]) -- that's why I used ext4 as the top layer instead of another ZFS. [1] (as mentioned before) https://www.reddit.com/r/zfs/comments/3vtur4/what_exactly_happens_when_you_go_over_80_on/cxqu7uh/ https://www.reddit.com/r/zfs/comments/3vtur4/what_exactly_ha... [2] https://news.ycombinator.com/item?id=14756360 https://news.ycombinator.com/item?id=14756360 [3] https://forums.freenas.org/index.php?threads/what-is-the-exact-checksum-size-overhead.28187/#post-183802 https://forums.freenas.org/index.php?threads/what-is-the-exa... Test setup: debian stable kernel 4.9.0-3-amd64 zfs 0.6.5.9-5 ZFS "pool": mirror with 2x 7200rpm drives Benchmark command: for i in `seq 1 10`; do sync; dd if=/dev/zero of=DEST bs=1M count=1024 conv=fdatasync; done zfs mirror -> dataset Data (MB/s): 125,115,104,135,148,170,135,151,118,119 Mean (MB/s): 132.0 Std.dev.: 19.9 zfs mirror -> zvol (volblocksize=8K [default]) Data (MB/s): 150,115,127,125,122,118,105,118,124,128 Mean (MB/s): 123.2 Std.dev.: 11.6 zfs mirror -> zvol (volblocksize=128K) Data (MB/s): 68.5,112,115,114,94.3,85.1,83.1,98.4,120,108 Mean (MB/s): 99.8 Std.dev.: 16.9 zfs mirror -> zvol (volblocksize=128K) -> luks -> ext4 (my stack) Data (MB/s): 130,94.4,109,139,138,125,94.9,124,134,133 Mean (MB/s): 122.1 Std.dev.: 16.8 edit: formatting
lmm 9y ago
> ZFS doesn't support addition of disks unless you're happy to put a RAID0 on top of your RAID5/6 and it doesn't support removal of disks at all when parity is involved. It is possible to migrate by putting giant sparse files on the existing storage, filling the filesystem, removing a sparse file, removing a disk from the original FS and "replacing" the sparse file with the actual disk but this is somewhat risky. It may be "RAID0 on top of your RAID5/6" but it's all integrated and works smoothly. What I do is have two raidz2 vdevs of 4 disks each, one twice the size of the other, and alternate which one I upgrade (i.e. I started with 4x250gb disks and 4x500gb disks, a few years later replaced the 250gb ones with 1tb ones, then the 500gb ones with 2tb ones, and most recently the 1tb ones with 4tb ones). But yeah I am now stuck with at least 8 disks and if I wanted to migrate off them I'd have to do so all in one go.
- Veratyr 9y ago
  > It may be "RAID0 on top of your RAID5/6" but it's all integrated and works smoothly. I know that it does but I don't like the way you lose your entire pool if a single vdev fails. I'd have less of a problem with it if a pool could distribute data such that a vdev failure results in the loss of only what was on that vdev.
  - lmm 9y ago
    Shrug. I figure losing a random half of my files is pretty much as bad as losing all of them (if I wanted to make an explicit split into two halves I could use two separate pools).

sfoskett 9y ago

There kinda isn't such a thing at this point. I sure wish there was! And this is the point behind the article: Wouldn't it be awesome if ZFS had continued developing as the super filesystem it looked like last decade? It would do all this and more! Frankly if ZFS supported re-balancing ("re-RAID-ing") and hybrid pools, it would be pretty much everything we all need. I do hope ZFS or Btrfs or something gets there.

Veratyr 9y ago
It's so close though. BTRFS and Ceph both have what I want, they're just unstable. The BcacheFS dev has told me he's close but not quite there yet. Hopefully one of the other solutions will work, otherwise I'll just have to build it.
- jamiek88 9y ago
  bcachefs really does seem as if it will be our holy grail, we have very similar needs. They have a patreon if you wish to contribute that way.
  - Veratyr 9y ago
    I already contribute and help out with testing and the occasional docs on IRC :)

MikusR 9y ago

For storage spaces try fixed size, you can later enlarge the volumes, faster.

Veratyr 9y ago
I was using fixed size, it didn't help. Read speeds were okay but write always capped out at 25MB/s.

davrosthedalek 9y ago

Have you considered snapraid? It's offline though, but seems to tick all your boxes otherwise.

Veratyr 9y ago
Yes, I've looked into SnapRAID but I'm not a fan of offline and IIRC I didn't have enough memory to run it last time I tried it out.

rodgerd 9y ago

XFS has gained (some) checksumming and CoW support. At this rate XFS will end up evolving to add all the features btrfs promised before the latter makes them stable.

amorphid 9y ago

Re: contact sales, I totally hear what you're saying. It's not always a bad idea to talk to a sales person, and a good one can steer you towards the options you'd actually want. But yeah, I generally don't want to talk to a salesperson either.

el_isma 9y ago
Yes, but would it kill them to just put a pricing there? Nowadays you can get the list price of sending a Rocket into space[1] but not how much some software is going to cost you [1] http://www.spacex.com/about/capabilities http://www.spacex.com/about/capabilities in case you need one

emikulic 9y ago

Have you seen Quantcast File System? http://quantcast.github.io/qfs/ http://quantcast.github.io/qfs/

Veratyr 9y ago
I hadn't and it looks good but it'll only allow 3 recovery stripes or none, which isn't optimal for me.

555h 9y ago

You can probably achieve what you're looking for by stacking a few filesystems. For example, you could create a separate ZFS pool/vdev with a single full-disk zvol on each disk. Then use mdadm to create a RAID array of the zvols. Then put ext4 (or whatever) on the mdadm array. I've done something similar for the purpose of getting FDE with ZFS in linux. It can be a little finicky, but it's definitely workable. One ZFS-specific caveat (which may conflict with your desire to get high storage efficiency): you way need to prevent your ZFS pools from filling up too much [1]. You can either enable discard/TRIM on the whole stack, so the top level FS (e.g. ext4) can let ZFS know when a block is actually free. Or alternative just to limit your zvols to 85% (for example) of their respective pools. The latter is my preference, because there was originally a bug with discard in zfs and it's not immediately clear if it's totally fixed (although my fstrim tests seemed to work out fine). [1] https://www.reddit.com/r/zfs/comments/3vtur4/what_exactly_happens_when_you_go_over_80_on/cxqu7uh/ https://www.reddit.com/r/zfs/comments/3vtur4/what_exactly_ha...

Veratyr 9y ago
Hmm, that actually sounds workable. I could even format the mdadm device as ZFS too if I really wanted. I am somewhat worried about performance, have you had any issues with that?
- 555h 9y ago
  I haven't had any issues with performance, but then again my requirement was just "reasonable performance". I ran some quick benchmarks (data below). Obviously this is far from rigorous, but maybe it'll be useful. In previous tests I found that volblocksize=128K was optimal for my stack -- which is why the last benchmarks use that setting. Every additional ZFS filesystem in the stack may reduce storage efficiency (minimum free space requirements [1]; metadata & checksum overhead [2][3]) -- that's why I used ext4 as the top layer instead of another ZFS. [1] (as mentioned before) https://www.reddit.com/r/zfs/comments/3vtur4/what_exactly_happens_when_you_go_over_80_on/cxqu7uh/ https://www.reddit.com/r/zfs/comments/3vtur4/what_exactly_ha... [2] https://news.ycombinator.com/item?id=14756360 https://news.ycombinator.com/item?id=14756360 [3] https://forums.freenas.org/index.php?threads/what-is-the-exact-checksum-size-overhead.28187/#post-183802 https://forums.freenas.org/index.php?threads/what-is-the-exa... Test setup: debian stable kernel 4.9.0-3-amd64 zfs 0.6.5.9-5 ZFS "pool": mirror with 2x 7200rpm drives Benchmark command: for i in `seq 1 10`; do sync; dd if=/dev/zero of=DEST bs=1M count=1024 conv=fdatasync; done zfs mirror -> dataset Data (MB/s): 125,115,104,135,148,170,135,151,118,119 Mean (MB/s): 132.0 Std.dev.: 19.9 zfs mirror -> zvol (volblocksize=8K [default]) Data (MB/s): 150,115,127,125,122,118,105,118,124,128 Mean (MB/s): 123.2 Std.dev.: 11.6 zfs mirror -> zvol (volblocksize=128K) Data (MB/s): 68.5,112,115,114,94.3,85.1,83.1,98.4,120,108 Mean (MB/s): 99.8 Std.dev.: 16.9 zfs mirror -> zvol (volblocksize=128K) -> luks -> ext4 (my stack) Data (MB/s): 130,94.4,109,139,138,125,94.9,124,134,133 Mean (MB/s): 122.1 Std.dev.: 16.8 edit: formatting

lmm 9y ago

> ZFS doesn't support addition of disks unless you're happy to put a RAID0 on top of your RAID5/6 and it doesn't support removal of disks at all when parity is involved. It is possible to migrate by putting giant sparse files on the existing storage, filling the filesystem, removing a sparse file, removing a disk from the original FS and "replacing" the sparse file with the actual disk but this is somewhat risky. It may be "RAID0 on top of your RAID5/6" but it's all integrated and works smoothly. What I do is have two raidz2 vdevs of 4 disks each, one twice the size of the other, and alternate which one I upgrade (i.e. I started with 4x250gb disks and 4x500gb disks, a few years later replaced the 250gb ones with 1tb ones, then the 500gb ones with 2tb ones, and most recently the 1tb ones with 4tb ones). But yeah I am now stuck with at least 8 disks and if I wanted to migrate off them I'd have to do so all in one go.

Veratyr 9y ago
> It may be "RAID0 on top of your RAID5/6" but it's all integrated and works smoothly. I know that it does but I don't like the way you lose your entire pool if a single vdev fails. I'd have less of a problem with it if a pool could distribute data such that a vdev failure results in the loss of only what was on that vdev.
- lmm 9y ago
  Shrug. I figure losing a random half of my files is pretty much as bad as losing all of them (if I wanted to make an explicit split into two halves I could use two separate pools).