7 ms·

I've been using btrfs since about 2011, and I've stopped using ext4 / xfs / zfs everwhere since about 2014. From 2012-2014 it was mostly breakage every other m

by zanny 10y ago

I've been using btrfs since about 2011, and I've stopped using ext4 / xfs / zfs everwhere since about 2014. From 2012-2014 it was mostly breakage every other month. From 2014-2016, it was semi-annual issues. For the last ~18 months I have had ~30 machines running btrfs with no issues, some servers, some personal computers. The release notes are boring, the bugs are boring, and to me its definitely in a state I would strongly consider trusting it with any workload. I worry that btrfs is just going to remain doomed. It wasn't stable half a decade ago, so it - for some reason - cannot be more stable now. But it has seen so much work put into it to make it as mature as it is now, and in my experience it is pretty damn mature now. All I want to see is another year and a half of perfect stability before I would start arguing to drop zfs entirely.

_ecqc 10y ago
Are you running BTRFS with its built in RAID? That's been the biggest blocker for me. There have been numerous RAID bugs that have caused data-loss and I believe at least one of them is still unpatched.
- kzrdude 10y ago
  I think Chris Mason is open with that raid support is not really stable yet, right? Status page says so: https://btrfs.wiki.kernel.org/index.php/Status https://btrfs.wiki.kernel.org/index.php/Status
  - aarmenaa 10y ago
    As far as I know they consider RAID0, 1, and 10 to be stable. Last time I used it rebuilds were substantially slower than ZFS or mdraid. Rebuild performance seems to one of a few issues that BTRFS has had trouble solving. RAID 5 and 6 were declared stable last year, only to have that retracted when some fatal flaw was discovered that would apparently cause data loss if you needed to rebuild.
    - mshook 10y ago
      It's mostly true but the issues as far as I know are: - RAID 1 with more than 2 disks is not what you think it is: the data will be mirrored but only once, no matter how many disks you have (meaning if you have a mirror with 3 disks, you only have 2 copies of your data). Because in BTRFS lingo, RAID 1 means '2 copies of the data' https://btrfs.wiki.kernel.org/index.php/FAQ#What_are_the_differences_among_MD-RAID_.2F_device_mapper_.2F_btrfs_raid.3F https://btrfs.wiki.kernel.org/index.php/FAQ#What_are_the_dif... which is not what people expect from RAID 1 with more than 2 disks - RAID 1 always needs 2 disks to be working, if not you can't mount the thing... Well you can, but only once... https://btrfs.wiki.kernel.org/index.php/Gotchas#raid1_volumes_only_mountable_once_RW_if_degraded https://btrfs.wiki.kernel.org/index.php/Gotchas#raid1_volume... - RAID 10 inherits these special RAID 1 cases as a result Most of that stuff is described here: https://btrfs.wiki.kernel.org/index.php/Status https://btrfs.wiki.kernel.org/index.php/Status So as they say on the status page: "mostly ok"
- lomnakkus 10y ago
  My main issue isn't actually a single thing that's wrong -- it's the completely and utterly haphazard way many of the features in btrfs have been "designed"[1]. Some of these problem they've had seem, to me at least, to be a fundamental lack of a coherent design. That does not bode well for stability, even 10(?) years after its first version. bcachefs seems to have a much more coherent design. [1] "Oh, yeah, I don't know how to handle this code path yet, let's stick a BUG_ON in there! I'm sure we'll figure something out later."
jdc 10y ago
I lost data on my workstation running brtfs in 2016, but would give it another shot if I knew I could expect reliability from it. Makes me wonder if anyone's tried something like Jepsen for filesystems.
- kev009 10y ago
  https://github.com/kdave/xfstests https://github.com/kdave/xfstests is the old standard. Folks have also been using AFL to fuzz. In ZFS, zdb is often used for similar purposes.
  - koverstreet 10y ago
    xfstests is still the standard, on Linux - https://git.kernel.org/pub/scm/fs/xfs/xfstests-dev.git https://git.kernel.org/pub/scm/fs/xfs/xfstests-dev.git I use it pretty much every day.
rrauenza 10y ago
I've been using btrfs since about 2014 on CentOS7. I only use it with mirroring+compression. No snapshots. I mostly went with it for data scrubbing and compression. My experience has been mixed, but haven't had any data loss. There was a bug for a while regarding free space so occasionally the system would seem to be full but wasn't ....and it was a real pain to correct it. I now have a cron job that does a monthly btrfs balance along with a mount -oremount,clear_cache. I also run the latest kernels from http://elrepo.org http://elrepo.org instead of the CentOS7 kernels so that I get the latest patches.
- Zardoz84 10y ago
  Why the remount,clear_cache ? I'm doing a monthly balance before I got problems on a few servers.
  - rrauenza 10y ago
    It's a little black magic -- didn't have time to completely research (this is a home NAS system.) I believe there was a bug in the clear space cache. This could cause the system to think it didn't have free blocks... you'd have to mount another device to create more space in order to rebalance and fix. Eventually I saw a bug fix report about a corruption in the cache... I never investigated to see if my current kernel has the fix.
- gigatexal 10y ago
  ZFS gets you all of that and raid and snapshots and has been tested time and time again. Why run BRTFS then?
  - tylerjd 10y ago
    ZFS is not GPL? I dunno, I like feeling safe when I do kernel upgrades knowing even if for whatever reason my ZFS module doesn't compile/work under the new kernel, I won't be left without a root FS. I have been running BTRFS for 5 years with very little issues, and enjoy not having to compile a new dkms module with every kernel.
noja 10y ago
Me too. I love it, but only with raid-1 or no raid. btrfs as a filesystem will be dead within the next 12 months unless they fix raid-5 and raid-6.
- Zardoz84 10y ago
  I'm using BTRFS on many VMs running Ubuntu servers, and I find that expanding the virtual hard-disk, on Proxmox, without stooping the VM it's far trivial.
  - arde 10y ago
    I think you missed a key word there so the meaning is lost. Is it "far from trivial" or "far more trivial"? You also surely meant stopping, not stooping.
- jl6 10y ago
  I think RAID 5/6 will be dead before btrfs.
  - noja 10y ago
    How are you doing redundancy then? Raid-1?
    - jl6 10y ago
      Me personally? Yes, RAID-1.
mrmondo 10y ago
Likewise, I've run it on hundreds of machines over the past three years without issue. I do continue to use EXT4 for database hosts as it far outperforms BTRFS with PostgreSQL from what I've seen.
owaislone 10y ago
I had a bad experience with in around 2014-15 on Ubuntu. 3 different laptops in my house suddenly stopped working (didn't boot at all) within the time-span of a year, 1 laptop faced the problem multiple times. In all cases, I had to format the root partition and reinstall Linux. All 3 had btrfs for / in common. I moved back to EXT4 and it never happened again since then.
j3097736 10y ago
I purposefully ran btrfs on a malfunctioning drive for over a year (kernel 3.12, only metadata dup), much more reliable than ext4 which would lock up the entire filesystem on read/write failure and often go read-only, with btrfs the only visible signs of malfunction were dmesg and the scrub log. Also been using it as / since 2012 with no issues.
syshum 10y ago
btrfs ran before it could walk. It is ok for a single disk FS. It is no ZFS though which is its largest problem, people keep marketing it as "linux awnser to ZFS" No, it is not. Maybe some day, but today it is no ZFS. I love zfs.. Further the ZFS utilities are far easier to use and understand. zfs and zpool commands well documented, and intuitive. btrfs utilities are are not, IMO I am fine with using btrfs as a replacment for ext on my OS drive, but for my large data arrays of multiple disks it s ZoL all the way
__david__ 10y ago
How is the speed now? I never had data issues but I definitely had speed issues. Btrfs was really slow compared to ext4/xfs back when I tried it. And I mean orders of magnitude slow. I had an application that did a lot disk access and switching off of btrfs brought the runtime down from a week to just hours. I want to like btrfs, but after that I just can't trust it for high disk load situations.
- untoreh 10y ago
  with cow is still slower than zfs. It still uses lzo for compression whereas zfs uses lz4, which is 10x faster for decompression.
mfukar 10y ago
What kind of workloads are you using it for?

by zanny 10y ago

_ecqc 10y ago

Are you running BTRFS with its built in RAID? That's been the biggest blocker for me. There have been numerous RAID bugs that have caused data-loss and I believe at least one of them is still unpatched.

kzrdude 10y ago
I think Chris Mason is open with that raid support is not really stable yet, right? Status page says so: https://btrfs.wiki.kernel.org/index.php/Status https://btrfs.wiki.kernel.org/index.php/Status
- aarmenaa 10y ago
  As far as I know they consider RAID0, 1, and 10 to be stable. Last time I used it rebuilds were substantially slower than ZFS or mdraid. Rebuild performance seems to one of a few issues that BTRFS has had trouble solving. RAID 5 and 6 were declared stable last year, only to have that retracted when some fatal flaw was discovered that would apparently cause data loss if you needed to rebuild.
  - mshook 10y ago
    It's mostly true but the issues as far as I know are: - RAID 1 with more than 2 disks is not what you think it is: the data will be mirrored but only once, no matter how many disks you have (meaning if you have a mirror with 3 disks, you only have 2 copies of your data). Because in BTRFS lingo, RAID 1 means '2 copies of the data' https://btrfs.wiki.kernel.org/index.php/FAQ#What_are_the_differences_among_MD-RAID_.2F_device_mapper_.2F_btrfs_raid.3F https://btrfs.wiki.kernel.org/index.php/FAQ#What_are_the_dif... which is not what people expect from RAID 1 with more than 2 disks - RAID 1 always needs 2 disks to be working, if not you can't mount the thing... Well you can, but only once... https://btrfs.wiki.kernel.org/index.php/Gotchas#raid1_volumes_only_mountable_once_RW_if_degraded https://btrfs.wiki.kernel.org/index.php/Gotchas#raid1_volume... - RAID 10 inherits these special RAID 1 cases as a result Most of that stuff is described here: https://btrfs.wiki.kernel.org/index.php/Status https://btrfs.wiki.kernel.org/index.php/Status So as they say on the status page: "mostly ok"
lomnakkus 10y ago
My main issue isn't actually a single thing that's wrong -- it's the completely and utterly haphazard way many of the features in btrfs have been "designed"[1]. Some of these problem they've had seem, to me at least, to be a fundamental lack of a coherent design. That does not bode well for stability, even 10(?) years after its first version. bcachefs seems to have a much more coherent design. [1] "Oh, yeah, I don't know how to handle this code path yet, let's stick a BUG_ON in there! I'm sure we'll figure something out later."

jdc 10y ago

I lost data on my workstation running brtfs in 2016, but would give it another shot if I knew I could expect reliability from it. Makes me wonder if anyone's tried something like Jepsen for filesystems.

kev009 10y ago
https://github.com/kdave/xfstests https://github.com/kdave/xfstests is the old standard. Folks have also been using AFL to fuzz. In ZFS, zdb is often used for similar purposes.
- koverstreet 10y ago
  xfstests is still the standard, on Linux - https://git.kernel.org/pub/scm/fs/xfs/xfstests-dev.git https://git.kernel.org/pub/scm/fs/xfs/xfstests-dev.git I use it pretty much every day.

rrauenza 10y ago

I've been using btrfs since about 2014 on CentOS7. I only use it with mirroring+compression. No snapshots. I mostly went with it for data scrubbing and compression. My experience has been mixed, but haven't had any data loss. There was a bug for a while regarding free space so occasionally the system would seem to be full but wasn't ....and it was a real pain to correct it. I now have a cron job that does a monthly btrfs balance along with a mount -oremount,clear_cache. I also run the latest kernels from http://elrepo.org http://elrepo.org instead of the CentOS7 kernels so that I get the latest patches.

Zardoz84 10y ago
Why the remount,clear_cache ? I'm doing a monthly balance before I got problems on a few servers.
- rrauenza 10y ago
  It's a little black magic -- didn't have time to completely research (this is a home NAS system.) I believe there was a bug in the clear space cache. This could cause the system to think it didn't have free blocks... you'd have to mount another device to create more space in order to rebalance and fix. Eventually I saw a bug fix report about a corruption in the cache... I never investigated to see if my current kernel has the fix.
gigatexal 10y ago
ZFS gets you all of that and raid and snapshots and has been tested time and time again. Why run BRTFS then?
- tylerjd 10y ago
  ZFS is not GPL? I dunno, I like feeling safe when I do kernel upgrades knowing even if for whatever reason my ZFS module doesn't compile/work under the new kernel, I won't be left without a root FS. I have been running BTRFS for 5 years with very little issues, and enjoy not having to compile a new dkms module with every kernel.

noja 10y ago

Me too. I love it, but only with raid-1 or no raid. btrfs as a filesystem will be dead within the next 12 months unless they fix raid-5 and raid-6.

Zardoz84 10y ago
I'm using BTRFS on many VMs running Ubuntu servers, and I find that expanding the virtual hard-disk, on Proxmox, without stooping the VM it's far trivial.
- arde 10y ago
  I think you missed a key word there so the meaning is lost. Is it "far from trivial" or "far more trivial"? You also surely meant stopping, not stooping.
jl6 10y ago
I think RAID 5/6 will be dead before btrfs.
- noja 10y ago
  How are you doing redundancy then? Raid-1?
  - jl6 10y ago
    Me personally? Yes, RAID-1.

mrmondo 10y ago

Likewise, I've run it on hundreds of machines over the past three years without issue. I do continue to use EXT4 for database hosts as it far outperforms BTRFS with PostgreSQL from what I've seen.

owaislone 10y ago

I had a bad experience with in around 2014-15 on Ubuntu. 3 different laptops in my house suddenly stopped working (didn't boot at all) within the time-span of a year, 1 laptop faced the problem multiple times. In all cases, I had to format the root partition and reinstall Linux. All 3 had btrfs for / in common. I moved back to EXT4 and it never happened again since then.

j3097736 10y ago

I purposefully ran btrfs on a malfunctioning drive for over a year (kernel 3.12, only metadata dup), much more reliable than ext4 which would lock up the entire filesystem on read/write failure and often go read-only, with btrfs the only visible signs of malfunction were dmesg and the scrub log. Also been using it as / since 2012 with no issues.

syshum 10y ago

btrfs ran before it could walk. It is ok for a single disk FS. It is no ZFS though which is its largest problem, people keep marketing it as "linux awnser to ZFS" No, it is not. Maybe some day, but today it is no ZFS. I love zfs.. Further the ZFS utilities are far easier to use and understand. zfs and zpool commands well documented, and intuitive. btrfs utilities are are not, IMO I am fine with using btrfs as a replacment for ext on my OS drive, but for my large data arrays of multiple disks it s ZoL all the way

__david__ 10y ago

How is the speed now? I never had data issues but I definitely had speed issues. Btrfs was really slow compared to ext4/xfs back when I tried it. And I mean orders of magnitude slow. I had an application that did a lot disk access and switching off of btrfs brought the runtime down from a week to just hours. I want to like btrfs, but after that I just can't trust it for high disk load situations.

untoreh 10y ago
with cow is still slower than zfs. It still uses lzo for compression whereas zfs uses lz4, which is 10x faster for decompression.

mfukar 10y ago

What kind of workloads are you using it for?