9 ms·

As a heavy user of ZFS and Linux, what else is there that even comes close to what ZFS offers? I want cheap and reliable snapshots, export & import of file sys

As a heavy user of ZFS and Linux, what else is there that even comes close to what ZFS offers? I want cheap and reliable snapshots, export & import of file systems like ZFS datasets, simple compression, caching facilities(like SLOG and ARC) and decent performance.

zippergz 7y ago
I know this isn't an option for everyone, but this is part of why I run FreeBSD instead of Linux for servers where I need ZFS.
- ggm 7y ago
  This isn't why I started running FreeBSD, but it is also one of the reasons I continue to run FreeBSD.
  - mekster 7y ago
    Yes, I run Linux for business but keep a FreeBSD personally so I'm used to it in case I need zfs for business.
fgonzag 7y ago
Bcachefs is probably the only thing that will get there. The codebase is clean and we'll mantained, built from solid technology (bcache) and will include most of the ZFS niceties. I just wish more companies would sponsor de project and stop wasting money on BTRFS
- jessant 7y ago
  Snapshots don't seem to be done yet.
  - fgonzag 7y ago
    That's exactly why I said it's probably the only one that will get there.
    - Rapzid 7y ago
      Heh, BTRFS deja vu. Been hearing about the ZFS alternative "not quite there, but catching up" for about as long as high-speed rail. I wonder which will arrive first :)
      - fgonzag 7y ago
        BTRFS is never going to become stable. Ever. Just take a quick dive into the codebase. Bcachefs has never had an unrecoverable data error AFAIK, even though it isn't even considered stable enough to merge into the kernel. The bcache on disk format won't be considered stable until he merges his code into mainline, though he doesn't feel he will need to adjust it further. Features that currently work: Full data checksumming Compression Multiple device support Tiering/writeback caching RAID1/RAID10 All of these are stable, tested, and mostly bug free. Honestly, once the code gets mainlined you'll be able to start using it very quickly. Main issue right now is performance, as it about as slow as BTRFS, which isn't inspiring. However the author has stated that he's going for correctness first, then he'll begin optimizing.
  - aseipp 7y ago
    Kent has admitted (many times) that snapshots are one of the more difficult features to add in a reliable and safe way, and will require significant work to do right, especially for what he wants to see them do (I assume "really damn fast and low overhead" is a major one, plus some other tricks he has up his sleeve.) So he has intentionally not tackled them yet, instead going after a slew of other features first. Reflink, full checksum, replication, caching, compression, native encryption, etc. All of that works today. Snapshots are a huge feature for sure, but it's not like bcachefs is completely incapable without them. There was a very recent update he gave in late December (2019) that mentioned he's actively chipping away at the roadblocks for snapshots.
  - ZoomZoomZoom 7y ago
    They're being worked on ATM: Dec 29, 2019 "Just finished a major rework that gets us a step closer to snapshots: the btree code is incrementally being changed to handle extents like regular keys." https://www.patreon.com/posts/towards-32698961 https://www.patreon.com/posts/towards-32698961
- reacharavindh 7y ago
  Yes, I’m eagerly waiting for Bcachefs to get there at some point, but it is several years away (rightly so, because it is hard and the developer is doing an amazing job) if my understanding of its state is correct. I have heard of durability issues with btrfs, and do not want to touch it if it fails with its primary job.
- lifty 7y ago
  I’m looking forward to bcachefs becoming feature complete and upstreamed. We finally have a good chance of having a modern and reliable FS in the Linux kernel. My wish list includes snapshots and per volume encryption.
- donmcronald 7y ago
  What if the main purpose of BTRFS is to have something "good enough" so no one starts working on a project that can compete with large commercial storage offerings? Does anyone remember the parity patches they rejected in 2014? > Your work is very very good, it just doesn’t fit our business case. I haven't followed it much. Does it have anything more than mirroring (that's stable) these days?
- forgotpwd16 7y ago
  >stop wasting money on BTRFS You're saying they should stop supporting a project that was considered stable by the time the other started being developed. Why do that? What makes Bcachefs a better choice?
  - fgonzag 7y ago
    Take a cursory look into both codebases, the stability of the every feature at launch and on maintenance. It's not hard to see BTRFS is a doomed project. Bcachefs is more like PostgreSQL, the developer doesn't add features until he has a solid design that's well thought out. Hence why he hasn't implemented snapshots. I don't think too many people consider it stable enough for production, either. (Unless you count a very limited subset of its functionality). I rather run Bcachefs today than Btrfs, by a mile. At least with bcachefs I won't lose my data.
    - phire 7y ago
      If you are on BTRFS and you encounter an unrecoverable bug (which seems to be reasonably common), the developers will most likely recommend you wipe the drive and restore from backups (because you had backups, right?) Even if the data is still on the drive and a bugfix would make the filesystem recoverable again, they don't have the time/knowledge/resources to untangle that codebase and make fixes. Even BTRFS developers don't trust the filesystem with their own data. If you are on Bcachefs and you encounter an unrecoverable bug, the developer will ask for some logs, or reproduction steps, or potentially even remote debugging access to your corrupt filesystem. And then he will fix the bug, releasing a new version that can read/repair your filesystem. He knows his codebase like the back of hand. In my research, I couldn't find any examples of someone actually losing data due to Bcachefs. All the bugs appeared to be "data has been written to drive, but bug prevented reading" While I would still hesitate to trust Bcachefs, I would trust it way more than BTRFS.
      - XorNot 7y ago
        Just want to note that bcachefs looks great (I was sort of tangentially aware but hadn't dedicated significant attention to it). Definitely something to try out (backing up my home servers is just about to reach viability for me, so I'd definitely consider switching to it in that use case). Thanks!
- XorNot 7y ago
  Which is why ZFS is still a thing today - there are no other alternatives. Everything is coming "soon" while ZFS is actually here and clocking up a stability track-record.
- ksec 7y ago
  >Bcachefs is probably the only thing that will get there. Or Bcachefs is probably the only thing that might get there. The amount of engineering hours went into ZFS is insane. It is easy to get a project that has 80% similarity on the surface, but then you spend the same amount of time from 0 - 80% on the last 20% and edge cases. ZFS has been battle tested by many. Rsync is on ZFS. The amount of Petabyte stored in ZFS safely over the years gives peace of mind. Speaking of Rsync, normally a topic of ZFS on HN will have him resurface. Hasn't seen any reply from him yet.
jessant 7y ago
Does btrfs met your requirements?
- nosequel 7y ago
  btrfs is not at all reliable, so if you care about your files staying working files, it probably doesn't meet your requirements. It is like the MongoDB 0.1 of filesystems.
  - jessant 7y ago
    Seems pretty reliable these days. Are you commenting based upon personal experience? If so, when was it that you used btrfs?
    - reacharavindh 7y ago
      When it comes to file systems “pretty reliable” these days does not sound very good. Reliability had to have been a fundamental requirement for design of a file system. If not, it sounds like putting lipstick on a pig. Redhat throwing towel on their support for development does not instill confidence either. Nothing personally against Btrfs. Just an end user making a file system choice saying what I care about.
      - jessant 7y ago
        re Redhat deprecating btrfs: > People are making a bigger deal of this than it is. Since I left Red Hat in 2012 there hasn't been another engineer to pick up the work, and it is _a lot_ of work. https://news.ycombinator.com/item?id=14909843 https://news.ycombinator.com/item?id=14909843
    - yjftsjthsd-h 7y ago
      I have a laptop running opensuse, with root on btrfs. Twice I have had to reinstall because it managed to corrupt the file system.
- Youden 7y ago
xupybd 7y ago
I agree that ZFS has a lot to offer. But the legal difficulties in merging ZFS support into the mainline kernal are understandable. It's a shame but I think he is making the right call.
- shawnz 7y ago
  Merging into the mainline kernel is not what the person he is replying to was even asking for. All they were asking is for Linux to stop putting APIs behind DRM that prevents non-GPL modules like ZFS from using them. That doesn't mean ZFS must be bundled with Linux. I think everyone is in agreement that ZFS can't be included in the mainline kernel. The question is just if users should be able to install and use it themselves or not.
  - xupybd 7y ago
    Thanks, I should have read more into this. The follow up actually clears things up pretty well. https://www.realworldtech.com/forum/?threadid=189711&curpostid=189842 https://www.realworldtech.com/forum/?threadid=189711&curpost...
- stopreformation 7y ago
  Kernal? If you can merge zfs support into 8KB kernal then you are not a mere mortal, so no need to worry about any legal difficulties.
bayindirh 7y ago
Honestly asking, how Btrfs compares to ZFS? There's also Lustre but it's a different beast altogether for a different scenario.
- dsr_ 7y ago
  On the surface, btrfs is pretty close to zfs. Once you actually use them, you discover all the ways that btrfs is a pain and zfs is a (minor) joy: - snapshot management - online scrub - data integrity - disk management I lost data from perfectly healthy-appearing btrfs systems twice. I've never lost data on maintained zfs systems, and I now trust a lot more data to zfs than I ever have to btrfs.
  - the8472 7y ago
    At least disk management is far easier with btrfs. You can restripe at will while zfs has severe limitations around resizing, adding and removing devices. Granted, at enterprise scale this hardly matters because you can just send-receive to rebuild pools if you have enough spares, but for consumer-grade deployments it's a non-negligible annoyance.
    - HorstG 7y ago
      Actually, this matters a lot in many enterprises. Beancounters hate excess capacities, so there are never enough spares and everything is always almost full. Maybe SV is different...
    - p_l 7y ago
      Restriping is source of unsafety, though. A lot of ZFS data safety comes from the fact it doesn't support overwriting anything, making it so that normal operation can't introduce unrecoverable corruption. In fact, all writes are done through snapshots.
      - the8472 7y ago
        ZFS wanted to have that too (the mythical block pointer rewrite) but it never happend, instead they add clunky workarounds like indirection tables for.
the8472 7y ago
btrfs + dm-cache? throw in dm-raid if you want raid5.
montjoy 7y ago
Hardware RAID controllers can do most if not all of these things.
- AtHeartEngineer 7y ago
  The problem with most of these is you have to bring the system down to do maintenance. You can do a scrub on zfs while it's up.
  - HorstG 7y ago
    Most non-hobbyist RAID hardware does online-scrub just fine (not that I would recommend wasting money on such hw). Btw, ZFS scrub is not only a RAID-block-check but also a partial fsck, so its not really comparable.
    - montjoy 7y ago
      We used the LSI 9286CV-8e (or dell equivalent) which was somewhere between $1000-$1500 back in the day. Worth it compared to babysitting any software RAID IMO.
- ggm 7y ago
  I've lost more data in hardware RAID than in ZFS but I have lost data in both. Hardware RAID has very poor longevity. Vendor support and battery backup replacement collide in BIOS and host management badly. Disclaimer: I work on Dell rackmounts, which means rather than native SAS I am 'Dells hack on SAS' which is a problem and I know its possible to 'downgrade' back to native.
  - montjoy 7y ago
    Yeah we started ordering the ones with the supercap so we didn’t have to replace batteries anymore. Somewhat recently I dealt with LSI and Dell cards. Longevity seemed just fine for a normal 3 year server lifecycle. The only time we had an issue is when the power went down in the data center. The power spike fried a few of the cards. Luckily we had spares. Way way back I dealt with the Compaq/hp smartarrays. Those were awful. Also anything consumer grade is awful.
m4rtink 7y ago
XFS on LVM thin pool LV should give you a very robust fs, cheap CoW snapshots, multi device support. If you want, you can make the thin pool be on RAID via LVM RAID under the thin pool. For import export, IIRC XFS has support for it and you can dump/import LV snapshots to get atomicity. For caching there is LVM cache, should be again possible to combine with thinpool & RAID. Or you can use it separately for normal LV. All this is functionality tested by years of production use. For compression/deduplication, that is AFAIK work in progress upstream based on the open sourced VDO code.
- reacharavindh 7y ago
  Interesting combination of tools I have used independently but never as a replacement of my beloved ZFS. Never made snapshots with LVM. Always used LVM as a way to carve up logical storage from a pool of physical devices but nothing more. I need to RTFM on how snapshotting would work there - could I restore just a few files from an hour ago while letting everything else be as they are? With ZFS, I use RAM as read chace(ARC) and an Optane disk as sync write cache(SLOG). I wonder if LVM cache would let me do such a thing. Again, a pointer for more manual reading for me. Compression is a nice to have for me at this moment. Good to know that it is being worked on at the LVM layer.
  - m4rtink 7y ago
    IIRC you can mount any of the snapshots & copy files from it without influencing the others & the thin LV itself. As for RAM caching, I'm not sure LVM would allow LVM cache residing on ram disk PV, but isin't regular Linux transparent FS access RAM caching sufficient actually ? For some reading about LVM thin provisioning: http://man7.org/linux/man-pages/man7/lvmthin.7.html http://man7.org/linux/man-pages/man7/lvmthin.7.html https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/logical_volume_manager_administration/thinly_provisioned_volume_creation https://access.redhat.com/documentation/en-us/red_hat_enterp...
- nickik 7y ago
  Call me when somebody like a major cloud provider has used this system to drive millions of hard-drives. I'm not gone patch my data security together like that. There is difference between 'all these tools have been used in production' and 'this is an integrated tool that has been used for 15+ years in the biggest storage installations in the world'.
arminiusreturns 7y ago
So besides the obvious btrfs answer, what about ceph as clustered storage with very fast connectivity? There is also BeeGFS, I haven't used it but /r/datahoarders sometimes touts it. Not for linux but I have been keeping an eye on M Dillons DragonFly BSD where he has been working on HAMMER2, which is very interesting. I don't know much but bcachefs has been making more waves lately also. I think the bottom line is that people need to have good backup in place regardless.
freedomben 7y ago
Stratis and VDO have a lot of promise, although it's still a little early. The approach that Stratis has taken is refreshing. It's very simple and reuses lots of already existing stuff so by the time it's released it will already be mature (since the underlying code has been running for many years). Once a little more guidance comes out about how to properly use VDO and Stratis together, I'll move my personal stuff to it.

zippergz 7y ago

I know this isn't an option for everyone, but this is part of why I run FreeBSD instead of Linux for servers where I need ZFS.

ggm 7y ago
This isn't why I started running FreeBSD, but it is also one of the reasons I continue to run FreeBSD.
- mekster 7y ago
  Yes, I run Linux for business but keep a FreeBSD personally so I'm used to it in case I need zfs for business.

fgonzag 7y ago

Bcachefs is probably the only thing that will get there. The codebase is clean and we'll mantained, built from solid technology (bcache) and will include most of the ZFS niceties. I just wish more companies would sponsor de project and stop wasting money on BTRFS

jessant 7y ago
Snapshots don't seem to be done yet.
- fgonzag 7y ago
  That's exactly why I said it's probably the only one that will get there.
  - Rapzid 7y ago
    Heh, BTRFS deja vu. Been hearing about the ZFS alternative "not quite there, but catching up" for about as long as high-speed rail. I wonder which will arrive first :)
    - fgonzag 7y ago
      BTRFS is never going to become stable. Ever. Just take a quick dive into the codebase. Bcachefs has never had an unrecoverable data error AFAIK, even though it isn't even considered stable enough to merge into the kernel. The bcache on disk format won't be considered stable until he merges his code into mainline, though he doesn't feel he will need to adjust it further. Features that currently work: Full data checksumming Compression Multiple device support Tiering/writeback caching RAID1/RAID10 All of these are stable, tested, and mostly bug free. Honestly, once the code gets mainlined you'll be able to start using it very quickly. Main issue right now is performance, as it about as slow as BTRFS, which isn't inspiring. However the author has stated that he's going for correctness first, then he'll begin optimizing.
- aseipp 7y ago
  Kent has admitted (many times) that snapshots are one of the more difficult features to add in a reliable and safe way, and will require significant work to do right, especially for what he wants to see them do (I assume "really damn fast and low overhead" is a major one, plus some other tricks he has up his sleeve.) So he has intentionally not tackled them yet, instead going after a slew of other features first. Reflink, full checksum, replication, caching, compression, native encryption, etc. All of that works today. Snapshots are a huge feature for sure, but it's not like bcachefs is completely incapable without them. There was a very recent update he gave in late December (2019) that mentioned he's actively chipping away at the roadblocks for snapshots.
- ZoomZoomZoom 7y ago
  They're being worked on ATM: Dec 29, 2019 "Just finished a major rework that gets us a step closer to snapshots: the btree code is incrementally being changed to handle extents like regular keys." https://www.patreon.com/posts/towards-32698961 https://www.patreon.com/posts/towards-32698961
reacharavindh 7y ago
Yes, I’m eagerly waiting for Bcachefs to get there at some point, but it is several years away (rightly so, because it is hard and the developer is doing an amazing job) if my understanding of its state is correct. I have heard of durability issues with btrfs, and do not want to touch it if it fails with its primary job.
lifty 7y ago
I’m looking forward to bcachefs becoming feature complete and upstreamed. We finally have a good chance of having a modern and reliable FS in the Linux kernel. My wish list includes snapshots and per volume encryption.
donmcronald 7y ago
What if the main purpose of BTRFS is to have something "good enough" so no one starts working on a project that can compete with large commercial storage offerings? Does anyone remember the parity patches they rejected in 2014? > Your work is very very good, it just doesn’t fit our business case. I haven't followed it much. Does it have anything more than mirroring (that's stable) these days?
forgotpwd16 7y ago
>stop wasting money on BTRFS You're saying they should stop supporting a project that was considered stable by the time the other started being developed. Why do that? What makes Bcachefs a better choice?
- fgonzag 7y ago
  Take a cursory look into both codebases, the stability of the every feature at launch and on maintenance. It's not hard to see BTRFS is a doomed project. Bcachefs is more like PostgreSQL, the developer doesn't add features until he has a solid design that's well thought out. Hence why he hasn't implemented snapshots. I don't think too many people consider it stable enough for production, either. (Unless you count a very limited subset of its functionality). I rather run Bcachefs today than Btrfs, by a mile. At least with bcachefs I won't lose my data.
  - phire 7y ago
    If you are on BTRFS and you encounter an unrecoverable bug (which seems to be reasonably common), the developers will most likely recommend you wipe the drive and restore from backups (because you had backups, right?) Even if the data is still on the drive and a bugfix would make the filesystem recoverable again, they don't have the time/knowledge/resources to untangle that codebase and make fixes. Even BTRFS developers don't trust the filesystem with their own data. If you are on Bcachefs and you encounter an unrecoverable bug, the developer will ask for some logs, or reproduction steps, or potentially even remote debugging access to your corrupt filesystem. And then he will fix the bug, releasing a new version that can read/repair your filesystem. He knows his codebase like the back of hand. In my research, I couldn't find any examples of someone actually losing data due to Bcachefs. All the bugs appeared to be "data has been written to drive, but bug prevented reading" While I would still hesitate to trust Bcachefs, I would trust it way more than BTRFS.
    - XorNot 7y ago
      Just want to note that bcachefs looks great (I was sort of tangentially aware but hadn't dedicated significant attention to it). Definitely something to try out (backing up my home servers is just about to reach viability for me, so I'd definitely consider switching to it in that use case). Thanks!
XorNot 7y ago
Which is why ZFS is still a thing today - there are no other alternatives. Everything is coming "soon" while ZFS is actually here and clocking up a stability track-record.
ksec 7y ago
>Bcachefs is probably the only thing that will get there. Or Bcachefs is probably the only thing that might get there. The amount of engineering hours went into ZFS is insane. It is easy to get a project that has 80% similarity on the surface, but then you spend the same amount of time from 0 - 80% on the last 20% and edge cases. ZFS has been battle tested by many. Rsync is on ZFS. The amount of Petabyte stored in ZFS safely over the years gives peace of mind. Speaking of Rsync, normally a topic of ZFS on HN will have him resurface. Hasn't seen any reply from him yet.

jessant 7y ago

Does btrfs met your requirements?

nosequel 7y ago
btrfs is not at all reliable, so if you care about your files staying working files, it probably doesn't meet your requirements. It is like the MongoDB 0.1 of filesystems.
- jessant 7y ago
  Seems pretty reliable these days. Are you commenting based upon personal experience? If so, when was it that you used btrfs?
  - reacharavindh 7y ago
    When it comes to file systems “pretty reliable” these days does not sound very good. Reliability had to have been a fundamental requirement for design of a file system. If not, it sounds like putting lipstick on a pig. Redhat throwing towel on their support for development does not instill confidence either. Nothing personally against Btrfs. Just an end user making a file system choice saying what I care about.
    - jessant 7y ago
      re Redhat deprecating btrfs: > People are making a bigger deal of this than it is. Since I left Red Hat in 2012 there hasn't been another engineer to pick up the work, and it is _a lot_ of work. https://news.ycombinator.com/item?id=14909843 https://news.ycombinator.com/item?id=14909843
  - yjftsjthsd-h 7y ago
    I have a laptop running opensuse, with root on btrfs. Twice I have had to reinstall because it managed to corrupt the file system.
Youden 7y ago

xupybd 7y ago

I agree that ZFS has a lot to offer. But the legal difficulties in merging ZFS support into the mainline kernal are understandable. It's a shame but I think he is making the right call.

shawnz 7y ago
Merging into the mainline kernel is not what the person he is replying to was even asking for. All they were asking is for Linux to stop putting APIs behind DRM that prevents non-GPL modules like ZFS from using them. That doesn't mean ZFS must be bundled with Linux. I think everyone is in agreement that ZFS can't be included in the mainline kernel. The question is just if users should be able to install and use it themselves or not.
- xupybd 7y ago
  Thanks, I should have read more into this. The follow up actually clears things up pretty well. https://www.realworldtech.com/forum/?threadid=189711&curpostid=189842 https://www.realworldtech.com/forum/?threadid=189711&curpost...
stopreformation 7y ago
Kernal? If you can merge zfs support into 8KB kernal then you are not a mere mortal, so no need to worry about any legal difficulties.

bayindirh 7y ago

Honestly asking, how Btrfs compares to ZFS? There's also Lustre but it's a different beast altogether for a different scenario.

dsr_ 7y ago
On the surface, btrfs is pretty close to zfs. Once you actually use them, you discover all the ways that btrfs is a pain and zfs is a (minor) joy: - snapshot management - online scrub - data integrity - disk management I lost data from perfectly healthy-appearing btrfs systems twice. I've never lost data on maintained zfs systems, and I now trust a lot more data to zfs than I ever have to btrfs.
- the8472 7y ago
  At least disk management is far easier with btrfs. You can restripe at will while zfs has severe limitations around resizing, adding and removing devices. Granted, at enterprise scale this hardly matters because you can just send-receive to rebuild pools if you have enough spares, but for consumer-grade deployments it's a non-negligible annoyance.
  - HorstG 7y ago
    Actually, this matters a lot in many enterprises. Beancounters hate excess capacities, so there are never enough spares and everything is always almost full. Maybe SV is different...
  - p_l 7y ago
    Restriping is source of unsafety, though. A lot of ZFS data safety comes from the fact it doesn't support overwriting anything, making it so that normal operation can't introduce unrecoverable corruption. In fact, all writes are done through snapshots.
    - the8472 7y ago
      ZFS wanted to have that too (the mythical block pointer rewrite) but it never happend, instead they add clunky workarounds like indirection tables for.

the8472 7y ago

btrfs + dm-cache? throw in dm-raid if you want raid5.

montjoy 7y ago

Hardware RAID controllers can do most if not all of these things.

AtHeartEngineer 7y ago
The problem with most of these is you have to bring the system down to do maintenance. You can do a scrub on zfs while it's up.
- HorstG 7y ago
  Most non-hobbyist RAID hardware does online-scrub just fine (not that I would recommend wasting money on such hw). Btw, ZFS scrub is not only a RAID-block-check but also a partial fsck, so its not really comparable.
  - montjoy 7y ago
    We used the LSI 9286CV-8e (or dell equivalent) which was somewhere between $1000-$1500 back in the day. Worth it compared to babysitting any software RAID IMO.
ggm 7y ago
I've lost more data in hardware RAID than in ZFS but I have lost data in both. Hardware RAID has very poor longevity. Vendor support and battery backup replacement collide in BIOS and host management badly. Disclaimer: I work on Dell rackmounts, which means rather than native SAS I am 'Dells hack on SAS' which is a problem and I know its possible to 'downgrade' back to native.
- montjoy 7y ago
  Yeah we started ordering the ones with the supercap so we didn’t have to replace batteries anymore. Somewhat recently I dealt with LSI and Dell cards. Longevity seemed just fine for a normal 3 year server lifecycle. The only time we had an issue is when the power went down in the data center. The power spike fried a few of the cards. Luckily we had spares. Way way back I dealt with the Compaq/hp smartarrays. Those were awful. Also anything consumer grade is awful.

m4rtink 7y ago

XFS on LVM thin pool LV should give you a very robust fs, cheap CoW snapshots, multi device support. If you want, you can make the thin pool be on RAID via LVM RAID under the thin pool. For import export, IIRC XFS has support for it and you can dump/import LV snapshots to get atomicity. For caching there is LVM cache, should be again possible to combine with thinpool & RAID. Or you can use it separately for normal LV. All this is functionality tested by years of production use. For compression/deduplication, that is AFAIK work in progress upstream based on the open sourced VDO code.

reacharavindh 7y ago
Interesting combination of tools I have used independently but never as a replacement of my beloved ZFS. Never made snapshots with LVM. Always used LVM as a way to carve up logical storage from a pool of physical devices but nothing more. I need to RTFM on how snapshotting would work there - could I restore just a few files from an hour ago while letting everything else be as they are? With ZFS, I use RAM as read chace(ARC) and an Optane disk as sync write cache(SLOG). I wonder if LVM cache would let me do such a thing. Again, a pointer for more manual reading for me. Compression is a nice to have for me at this moment. Good to know that it is being worked on at the LVM layer.
- m4rtink 7y ago
  IIRC you can mount any of the snapshots & copy files from it without influencing the others & the thin LV itself. As for RAM caching, I'm not sure LVM would allow LVM cache residing on ram disk PV, but isin't regular Linux transparent FS access RAM caching sufficient actually ? For some reading about LVM thin provisioning: http://man7.org/linux/man-pages/man7/lvmthin.7.html http://man7.org/linux/man-pages/man7/lvmthin.7.html https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/logical_volume_manager_administration/thinly_provisioned_volume_creation https://access.redhat.com/documentation/en-us/red_hat_enterp...
nickik 7y ago
Call me when somebody like a major cloud provider has used this system to drive millions of hard-drives. I'm not gone patch my data security together like that. There is difference between 'all these tools have been used in production' and 'this is an integrated tool that has been used for 15+ years in the biggest storage installations in the world'.

arminiusreturns 7y ago

So besides the obvious btrfs answer, what about ceph as clustered storage with very fast connectivity? There is also BeeGFS, I haven't used it but /r/datahoarders sometimes touts it. Not for linux but I have been keeping an eye on M Dillons DragonFly BSD where he has been working on HAMMER2, which is very interesting. I don't know much but bcachefs has been making more waves lately also. I think the bottom line is that people need to have good backup in place regardless.

freedomben 7y ago

Stratis and VDO have a lot of promise, although it's still a little early. The approach that Stratis has taken is refreshing. It's very simple and reuses lots of already existing stuff so by the time it's released it will already be mature (since the underlying code has been running for many years). Once a little more guidance comes out about how to properly use VDO and Stratis together, I'll move my personal stuff to it.