5 ms·

Is ZFS worth considering at this point? Btrfs seems to be quite featureful and stable, with a plethora of grocers and retailers all over the world using it. Wal

Is ZFS worth considering at this point? Btrfs seems to be quite featureful and stable, with a plethora of grocers and retailers all over the world using it. Wal-Mart, Kroger (and subsidiaries), all of IBM's old point of sale customers (now Toshiba SuperPOS iirc) all use it on things as small as deli scales all the way to running their point of sale systems and backends.

noja 8y ago
btrfs is not stable in some configurations (i.e. raid-5 like configs). How are Walmart using it?
- etskinner 8y ago
  They just don't use Raid 5/6
  - rleigh 8y ago
    That's by far from the only failure case to worry about. In addition to random dataloss, the most frequent failure cause I dealt with was complete unbalancing making the filesystem read-only. This happened every ~48 hours of continuous use! I've not read that this is fixed yet, but I haven't used it in the recent past due to this making it unsuitable for production use.
regularfry 8y ago
Btrfs used to have really nasty behaviour on nearly-full drives. Has that been fixed?
- unhammer 8y ago
  I've used it for backups for five years now. My USB backup drives have gotten full quite a lot of times because of old snapshots (rsync isn't able to detect moving directories), and I've had zero problems with it – just have to delete some old snapshots and wait a bit and then there's space again. I don't use RAID5, which is where I've heard btrfs is dangerous, just for the snapshots and dedupe (with https://github.com/jbruchon/jdupes/ https://github.com/jbruchon/jdupes/ )
- jstimpfle 8y ago
  It was used at my old job on a 1TB backup drive, containing lots of hourly snapshots from around 100 other machines that also had btrfs on it (btrfs send). The snapshots were also frequently thinned (i.e. only a few of the older ones kept) to make room for new ones. The filesystem broke at least two times in two years when it was nearly full. None of the fixes we found with web searches worked. We went with formatting and copying the backups from another location, losing only a little bit of data that was not too important. That was around 2015-2017, and the version of Btrfs was about what's in Debian jessie.
lclarkmichalek 8y ago
Facebook also use it on millions of machines.
- mrweasel 8y ago
  Has Facebook ever made it completely clear what they use btrfs for? I believe it was in one of the episodes of bsdnow where they talked about it "only" being used for administrative systems.
  - lclarkmichalek 8y ago
    I think that's a couple of years old. If you want more modern info, I think https://www.youtube.com/watch?v=ME4RhxwPO5A https://www.youtube.com/watch?v=ME4RhxwPO5A (skip to about 8m30) is pretty good, as is https://code.fb.com/open-source/linux/ https://code.fb.com/open-source/linux/. We also talk a lot about the benefits of having cgroup2 work well with BTRFS, which (imo) isn't something that particularly matters with administrative systems, but more for data processing/serving systems. I don't think we've released numbers or percentages, but it's safe to say every request you make to Facebook.com is processed by 1 or more machines with a btrfs filesystem. We've got a ton of value out of it, I hope we talk more about its successes publicly soon :) (I work at fb)
    - exikyut 8y ago
      Here's 8:23 pre-queued: https://youtu.be/ME4RhxwPO5A?t=503 https://youtu.be/ME4RhxwPO5A?t=503 That video's discussing btrfs layering, which is very neat. I am _very_ curious what sort of experience FB has had with btrfs. If FB has decided to go all-in with btrfs, you probably have the most accurate raw data there is to have. Of course, any hyper-scale deployment of a technology is going to produce remarkable/exponential statistics, and these will (sadly/annoyingly) need careful parsing to normalize in a way that is generally accessible and unsurprising. (We live in a very knee-jerk world, and all. Sigh) All this to say, I at least am looking forward to whatever sorts of numbers you end up being able to share. -- After considering the bit about btrfs in the video, and your mention of "1 or more machines", I wonder if you aren't using btrfs on things like load balancer type systems - or, more abstractly, node configurations that can always safely be "thrown away", whether by hardware failure or explicit decommission. But then I realize the chances are you probably use such a model (individual hardware failure must be acceptable; everything important must be n-way redundant) because nothing else makes sense at scale. And then I wonder... exactly what point in that spectrum does btrfs fit in? I wonder if/how you can answer that question.
ayush--s 8y ago
zfs used to have a better compression algorithm (lz4) before btrfs landed zstd with kernel version 4.15. Btrfs still doesn't have stable support for RAID 5+6 https://btrfs.wiki.kernel.org/index.php/RAID56 https://btrfs.wiki.kernel.org/index.php/RAID56
- rleigh 8y ago
  zfs is also getting zstandard: e.g. https://reviews.freebsd.org/D11124?id=30301 https://reviews.freebsd.org/D11124?id=30301 Doesn't appear to be merged yet, but it's coming.
zielmicha 8y ago
- fsync on btrfs is extremly slow. The situation has improved in last 3 years, but it is still much slower than ZFS. I just did a simple test and BTFS is 12x slower than ZFS (on my customer SSD, for small writes). - RAID5/6 mode is only experimental in btrfs, while it is really stable in ZFS. - I don't have concrete data for that, but in my experience, BTRFS has high latency (>1 second) even for small file operations when under load. - While that should not be a problem for production systems, I have some crappy hardware where BTRFS oopses or corrupts data, while other filesystems (Ext4, ZFS) work fine.
- jstimpfle 8y ago
  Yes the 12x slower seems about right. I remember Debian installations taking >1h while ext4 took < 10min. (Installations and package operations are particularly bad since dpkg does many fsync()s when unpacking). I think that was on a spinning drive, though. Anyway at my old job (where we had mainly spinning drives) we used apt and dpkg only with eatmydata. eatmydata is a command which uses LD_PRELOAD hackery to remove fsync() systemcalls.
  - rleigh 8y ago
    This was really painful. Back when I was a Debian developer tracking unstable, I would update every evening and it would sometimes take multiple hours to complete. The fsync performance is a truly awful design flaw, and while eatmydata does make it faster, it's a terrible and dangerious hack! (Which everyone used!) I even had to add eatmydata support to the schroot tool as a command-prefix option to allow every command to be run via eatmydata when using btrfs snapshots. (I've since dropped btrfs support entirely; it was too unreliable with large snapshot turnover rapidly unbalancing the filesystem. Unusable in production.)
  - andoma 8y ago
    I believe the correct solution to this problem would be that the installers would snapshot the system, install all packages without any fsync()ing at all, then finally one sync() and remove the snapshot. Optionally keeping snapshots if user want to roll back from a broken upgrade (for whatever reason). Again, as others have written here, btrfs is great if your software plays along with it, otherwise it might not be so great.
swinglock 8y ago
Is Btrfs worth considering? ZFS is one of the most advanced, stable and feature-complete filesystems there is, and is cross-platform. Btrfs is best known for being flakey and in mainline. Had there not been uncertainties about licensing which prevented ZFS from being in mainline there wouldn't be any Btrfs. How can you ever really trust a filesystem developed by adding features first, trying to add stability and usability later anyway?
- Wowfunhappy 8y ago
  I don't even really understand why ZFS being outside mainline is such an issue. On Debian Stretch, I installed ZFS like any other package, and didn't even realize the kernel was getting recompiled until I read about the licensing situation later on. I did think it was odd that installing the ZFS package took such a long time, but that was a one-time wait.
lrem 8y ago
If you're busy and can't be arsed to research past "this has been rock-stable for a decade, so should be safe to use", ZFS is the best choice out there.
xte 8y ago
btrfs is designed with a developers POV, zfs with a sysadmin POV. This means that the former is a pain to use, the latter works really well.
paulie_a 8y ago
If you care about data not randomly failing, btrfs is not an option. While working in that specific industry it seems like there is a motivation to specifically go out of their way to make things unreliable, extra complicated and utilize bad technologies. The fact that IBM and Toshiba on POS is frightening.

noja 8y ago

btrfs is not stable in some configurations (i.e. raid-5 like configs). How are Walmart using it?

etskinner 8y ago
They just don't use Raid 5/6
- rleigh 8y ago
  That's by far from the only failure case to worry about. In addition to random dataloss, the most frequent failure cause I dealt with was complete unbalancing making the filesystem read-only. This happened every ~48 hours of continuous use! I've not read that this is fixed yet, but I haven't used it in the recent past due to this making it unsuitable for production use.

regularfry 8y ago

Btrfs used to have really nasty behaviour on nearly-full drives. Has that been fixed?

unhammer 8y ago
I've used it for backups for five years now. My USB backup drives have gotten full quite a lot of times because of old snapshots (rsync isn't able to detect moving directories), and I've had zero problems with it – just have to delete some old snapshots and wait a bit and then there's space again. I don't use RAID5, which is where I've heard btrfs is dangerous, just for the snapshots and dedupe (with https://github.com/jbruchon/jdupes/ https://github.com/jbruchon/jdupes/ )
jstimpfle 8y ago
It was used at my old job on a 1TB backup drive, containing lots of hourly snapshots from around 100 other machines that also had btrfs on it (btrfs send). The snapshots were also frequently thinned (i.e. only a few of the older ones kept) to make room for new ones. The filesystem broke at least two times in two years when it was nearly full. None of the fixes we found with web searches worked. We went with formatting and copying the backups from another location, losing only a little bit of data that was not too important. That was around 2015-2017, and the version of Btrfs was about what's in Debian jessie.

lclarkmichalek 8y ago

Facebook also use it on millions of machines.

mrweasel 8y ago
Has Facebook ever made it completely clear what they use btrfs for? I believe it was in one of the episodes of bsdnow where they talked about it "only" being used for administrative systems.
- lclarkmichalek 8y ago
  I think that's a couple of years old. If you want more modern info, I think https://www.youtube.com/watch?v=ME4RhxwPO5A https://www.youtube.com/watch?v=ME4RhxwPO5A (skip to about 8m30) is pretty good, as is https://code.fb.com/open-source/linux/ https://code.fb.com/open-source/linux/. We also talk a lot about the benefits of having cgroup2 work well with BTRFS, which (imo) isn't something that particularly matters with administrative systems, but more for data processing/serving systems. I don't think we've released numbers or percentages, but it's safe to say every request you make to Facebook.com is processed by 1 or more machines with a btrfs filesystem. We've got a ton of value out of it, I hope we talk more about its successes publicly soon :) (I work at fb)
  - exikyut 8y ago
    Here's 8:23 pre-queued: https://youtu.be/ME4RhxwPO5A?t=503 https://youtu.be/ME4RhxwPO5A?t=503 That video's discussing btrfs layering, which is very neat. I am _very_ curious what sort of experience FB has had with btrfs. If FB has decided to go all-in with btrfs, you probably have the most accurate raw data there is to have. Of course, any hyper-scale deployment of a technology is going to produce remarkable/exponential statistics, and these will (sadly/annoyingly) need careful parsing to normalize in a way that is generally accessible and unsurprising. (We live in a very knee-jerk world, and all. Sigh) All this to say, I at least am looking forward to whatever sorts of numbers you end up being able to share. -- After considering the bit about btrfs in the video, and your mention of "1 or more machines", I wonder if you aren't using btrfs on things like load balancer type systems - or, more abstractly, node configurations that can always safely be "thrown away", whether by hardware failure or explicit decommission. But then I realize the chances are you probably use such a model (individual hardware failure must be acceptable; everything important must be n-way redundant) because nothing else makes sense at scale. And then I wonder... exactly what point in that spectrum does btrfs fit in? I wonder if/how you can answer that question.

ayush--s 8y ago

zfs used to have a better compression algorithm (lz4) before btrfs landed zstd with kernel version 4.15. Btrfs still doesn't have stable support for RAID 5+6 https://btrfs.wiki.kernel.org/index.php/RAID56 https://btrfs.wiki.kernel.org/index.php/RAID56

rleigh 8y ago
zfs is also getting zstandard: e.g. https://reviews.freebsd.org/D11124?id=30301 https://reviews.freebsd.org/D11124?id=30301 Doesn't appear to be merged yet, but it's coming.

zielmicha 8y ago

- fsync on btrfs is extremly slow. The situation has improved in last 3 years, but it is still much slower than ZFS. I just did a simple test and BTFS is 12x slower than ZFS (on my customer SSD, for small writes). - RAID5/6 mode is only experimental in btrfs, while it is really stable in ZFS. - I don't have concrete data for that, but in my experience, BTRFS has high latency (>1 second) even for small file operations when under load. - While that should not be a problem for production systems, I have some crappy hardware where BTRFS oopses or corrupts data, while other filesystems (Ext4, ZFS) work fine.

jstimpfle 8y ago
Yes the 12x slower seems about right. I remember Debian installations taking >1h while ext4 took < 10min. (Installations and package operations are particularly bad since dpkg does many fsync()s when unpacking). I think that was on a spinning drive, though. Anyway at my old job (where we had mainly spinning drives) we used apt and dpkg only with eatmydata. eatmydata is a command which uses LD_PRELOAD hackery to remove fsync() systemcalls.
- rleigh 8y ago
  This was really painful. Back when I was a Debian developer tracking unstable, I would update every evening and it would sometimes take multiple hours to complete. The fsync performance is a truly awful design flaw, and while eatmydata does make it faster, it's a terrible and dangerious hack! (Which everyone used!) I even had to add eatmydata support to the schroot tool as a command-prefix option to allow every command to be run via eatmydata when using btrfs snapshots. (I've since dropped btrfs support entirely; it was too unreliable with large snapshot turnover rapidly unbalancing the filesystem. Unusable in production.)
- andoma 8y ago
  I believe the correct solution to this problem would be that the installers would snapshot the system, install all packages without any fsync()ing at all, then finally one sync() and remove the snapshot. Optionally keeping snapshots if user want to roll back from a broken upgrade (for whatever reason). Again, as others have written here, btrfs is great if your software plays along with it, otherwise it might not be so great.

swinglock 8y ago

Is Btrfs worth considering? ZFS is one of the most advanced, stable and feature-complete filesystems there is, and is cross-platform. Btrfs is best known for being flakey and in mainline. Had there not been uncertainties about licensing which prevented ZFS from being in mainline there wouldn't be any Btrfs. How can you ever really trust a filesystem developed by adding features first, trying to add stability and usability later anyway?

Wowfunhappy 8y ago
I don't even really understand why ZFS being outside mainline is such an issue. On Debian Stretch, I installed ZFS like any other package, and didn't even realize the kernel was getting recompiled until I read about the licensing situation later on. I did think it was odd that installing the ZFS package took such a long time, but that was a one-time wait.

lrem 8y ago

If you're busy and can't be arsed to research past "this has been rock-stable for a decade, so should be safe to use", ZFS is the best choice out there.

xte 8y ago

btrfs is designed with a developers POV, zfs with a sysadmin POV. This means that the former is a pain to use, the latter works really well.

paulie_a 8y ago

If you care about data not randomly failing, btrfs is not an option. While working in that specific industry it seems like there is a motivation to specifically go out of their way to make things unreliable, extra complicated and utilize bad technologies. The fact that IBM and Toshiba on POS is frightening.