7 ms·

It would be interesting to see comparison between ZFS and Btrfs, ReiserFS in terms of features and development effort. Any idea?

by giis 11y ago

mrweasel 11y ago
Why ReiserFS? It's pretty much a defunct filesystem at this point. It's not really in the same class as ZFS and Btrfs, if you want a third option maybe look at HammerFS? Although I think pretty much everyone is waiting for Hammer2. There was a talk "Why OpenBSD sucks" by Henning Brauer, where he complains that ZFS is a "kitchen sink" approach to a filesystem. He doesn't really explain further, but it would be interesting to hear someone elaborate that criticism.
- glass- 11y ago
  Also see, Ted Unangst's post about ZFS where he lists some criticisms[0] which somewhat expands on Brauer's criticism, and he talks about when he was interviewed on BSD Now[1]. Keep in mind that their criticisms are mostly about it not being a good choice for OpenBSD, although some of them apply across the board. [0] http://www.tedunangst.com/flak/post/ZFS-on-OpenBSD http://www.tedunangst.com/flak/post/ZFS-on-OpenBSD [1] http://www.bsdnow.tv/episodes/2014_02_05-time_signatures http://www.bsdnow.tv/episodes/2014_02_05-time_signatures
- giis 11y ago
  Though ReiserFS is not actively developed, If I'm not wrong some Btrfs concepts ((like tail-packing?) derived from ReiserFS. Thanks for info about Henning Brauer talk, will check it out.
jlouis 11y ago
Far more interesting is the question: what error in the filesystem led to a situation where you lost data in production? It is very easy to crank out claims of features while not having really implemented said feature in a correct way. This leads to bugs which are hard to find and almost impossible to eradicate.
nisa 11y ago
RaiserFS is dead as far as I've read. btrfs is a shitshow - sorry to having to resort to such words but I will refrain from using it ever again if I have a choice. If you look at the btrfs wiki there a lot of features but most of these are horrible broken or implemented in a way that's surprising. It's full of bugs and not-yet-implemented stuff - some stuff that comes to mind: * RAID is utterly broken: - http://www.spinics.net/lists/linux-btrfs/msg48561.html http://www.spinics.net/lists/linux-btrfs/msg48561.html - http://www.spinics.net/lists/linux-btrfs/msg47845.html http://www.spinics.net/lists/linux-btrfs/msg47845.html * Lot's of ENOSPC bugs (just look at the mailing list) * You need to run the latest kernel (don't try anything below 4.2) - if not: Data corruption bugs, deadlocks * Quota is utterly broken and continues to be breaking (just look at the mailinglist) * It's useless for databases or VM images (disabling datacow is a ugly hack IMHO) - e.g. see here: http://blog.pgaddict.com/posts/friends-dont-let-friends-use-btrfs-for-oltp http://blog.pgaddict.com/posts/friends-dont-let-friends-use-... ZFS is not perfect either especially on Linux it's lacking tight integration into the VM subsystem and at least a few versions ago was prone to deadlocking, too. However it's not eating data and compared to btrfs it's rock solid.
- giis 11y ago
  Interesting ReiserFS comment http://blog.pgaddict.com/posts/friends-dont-let-friends-use-btrfs-for-oltp#comment-2266022309 http://blog.pgaddict.com/posts/friends-dont-let-friends-use-...
- simoncion 11y ago
  > * RAID is utterly broken: - http://www.spinics.net/lists/linux-btrfs/msg48561.html http://www.spinics.net/lists/linux-btrfs/msg48561.html sigh From the message that you linked: "FWIW... Older btrfs userspace such as your v3.17 is "OK" for normal runtime use, assuming you don't need any newer features, as in normal runtime, it's the kernel code doing the real work and userspace for the most part simply makes the appropriate kernel calls to do that work. But, once you get into a recovery situation like the one you're in now, current userspace becomes much more mportant, as the various things you'll do to attempt recovery rely far more on userspace code directly accessing the filesystem, and it's only the newest userspace code that has the latest fixes." If you look, you see that the guy reporting the problem was using a very new kernel, but a one-year-old btrfs-tools package. It makes no sense to use tools that are substantially older than the kernel you're running. He might have had much better luck with recovery (that is, he might have been able to get a rw volume with all his data, rather than an ro volume -with all his data- out of the process) with a recent tools package. > http://www.spinics.net/lists/linux-btrfs/msg47845.html http://www.spinics.net/lists/linux-btrfs/msg47845.html Maybe I'm reading this one wrong, but it looks like the summary of the issues are: * For multi-disk arrays with disks of varying sizes, each smaller disk does not receive data until all larger disks have as much or less free space than the smaller disk in question. [0] * The read scheduling algo is currently naive, using PID evenness to determine the disk in a pair to read, rather than current IO load on the disk. Did I miss something, or misunderstand something? If I didn't miss anything, I don't see how you can get "RAID is utterly broken" from "RAID has less-than-optimal write and read scheduling". > * It's useless for databases or VM images (disabling datacow is a ugly hack IMHO) I can't agree. I use btrfs on spinning rust to store my multi-terabyte Postgres 9.4 DB. It works well enough for my low-to-medium-volume [1] workload. > * Lot's of ENOSPC bugs (just look at the mailing list) I don't run into those, and haven't for literal years. > * Quota is utterly broken... Yes, quota is a difficult problem that continues to be unsolved. > * You need to run the latest kernel... Yes. Agreed. > ...if not: Data corruption bugs ... I haven't run into any, ever on the handful of systems on which I use BTRFS. > ...if not: ... deadlocks True. There was that unlink/move deadlock in 4.1. I hit it from time to time when updating my copy of the metadata from the Gentoo Portage tree. Happily, it only blocked operation on the affected file. The remainder of the system kept running along, and a -ugh- reboot unblocked the operation. [2] Look. I'm not saying that btrfs is feature-complete, suitable for every workload, or that any given person should use it for their workload. I'm just saying that the situation is far less dire than you're making it out to be. I've been using it as the rootfs for at least one of my daily use systems for the past 5.5 years, and haven't had any trouble out of it [3] in the past 2+ years. [0] That phrasing was abysmal. Here's an example: Given an array of one each of 3GB, 2GB, 1.5GB devices. the 2GB device will get data after 1GB has been written to the 3GB device. The 1.5GB device will get data after 0.5GB of additional data has been written to each of the 3GB and 2GB devices (making a total of 1GB additional data written to the array before the 1.5GB device starts getting data). AIUI, if you're running in RAID1 mode, then the same logic applies, but with paired devices. [1] Most definitely not web-scale workload. [2] I don't know if a unmount/remount cycle would have also worked. The wedged operation happened on my rootfs, so I had no way to test this. [3] Except for the unlink/move deadlock in 4.1.

It would be interesting to see comparison between ZFS and Btrfs, ReiserFS in terms of features and development effort. Any idea?

by giis 11y ago

It would be interesting to see comparison between ZFS and Btrfs, ReiserFS in terms of features and development effort. Any idea?

mrweasel 11y ago

Why ReiserFS? It's pretty much a defunct filesystem at this point. It's not really in the same class as ZFS and Btrfs, if you want a third option maybe look at HammerFS? Although I think pretty much everyone is waiting for Hammer2. There was a talk "Why OpenBSD sucks" by Henning Brauer, where he complains that ZFS is a "kitchen sink" approach to a filesystem. He doesn't really explain further, but it would be interesting to hear someone elaborate that criticism.

glass- 11y ago
Also see, Ted Unangst's post about ZFS where he lists some criticisms[0] which somewhat expands on Brauer's criticism, and he talks about when he was interviewed on BSD Now[1]. Keep in mind that their criticisms are mostly about it not being a good choice for OpenBSD, although some of them apply across the board. [0] http://www.tedunangst.com/flak/post/ZFS-on-OpenBSD http://www.tedunangst.com/flak/post/ZFS-on-OpenBSD [1] http://www.bsdnow.tv/episodes/2014_02_05-time_signatures http://www.bsdnow.tv/episodes/2014_02_05-time_signatures
giis 11y ago
Though ReiserFS is not actively developed, If I'm not wrong some Btrfs concepts ((like tail-packing?) derived from ReiserFS. Thanks for info about Henning Brauer talk, will check it out.

jlouis 11y ago

Far more interesting is the question: what error in the filesystem led to a situation where you lost data in production? It is very easy to crank out claims of features while not having really implemented said feature in a correct way. This leads to bugs which are hard to find and almost impossible to eradicate.

nisa 11y ago

RaiserFS is dead as far as I've read. btrfs is a shitshow - sorry to having to resort to such words but I will refrain from using it ever again if I have a choice. If you look at the btrfs wiki there a lot of features but most of these are horrible broken or implemented in a way that's surprising. It's full of bugs and not-yet-implemented stuff - some stuff that comes to mind: * RAID is utterly broken: - http://www.spinics.net/lists/linux-btrfs/msg48561.html http://www.spinics.net/lists/linux-btrfs/msg48561.html - http://www.spinics.net/lists/linux-btrfs/msg47845.html http://www.spinics.net/lists/linux-btrfs/msg47845.html * Lot's of ENOSPC bugs (just look at the mailing list) * You need to run the latest kernel (don't try anything below 4.2) - if not: Data corruption bugs, deadlocks * Quota is utterly broken and continues to be breaking (just look at the mailinglist) * It's useless for databases or VM images (disabling datacow is a ugly hack IMHO) - e.g. see here: http://blog.pgaddict.com/posts/friends-dont-let-friends-use-btrfs-for-oltp http://blog.pgaddict.com/posts/friends-dont-let-friends-use-... ZFS is not perfect either especially on Linux it's lacking tight integration into the VM subsystem and at least a few versions ago was prone to deadlocking, too. However it's not eating data and compared to btrfs it's rock solid.

giis 11y ago
Interesting ReiserFS comment http://blog.pgaddict.com/posts/friends-dont-let-friends-use-btrfs-for-oltp#comment-2266022309 http://blog.pgaddict.com/posts/friends-dont-let-friends-use-...
simoncion 11y ago
> * RAID is utterly broken: - http://www.spinics.net/lists/linux-btrfs/msg48561.html http://www.spinics.net/lists/linux-btrfs/msg48561.html sigh From the message that you linked: "FWIW... Older btrfs userspace such as your v3.17 is "OK" for normal runtime use, assuming you don't need any newer features, as in normal runtime, it's the kernel code doing the real work and userspace for the most part simply makes the appropriate kernel calls to do that work. But, once you get into a recovery situation like the one you're in now, current userspace becomes much more mportant, as the various things you'll do to attempt recovery rely far more on userspace code directly accessing the filesystem, and it's only the newest userspace code that has the latest fixes." If you look, you see that the guy reporting the problem was using a very new kernel, but a one-year-old btrfs-tools package. It makes no sense to use tools that are substantially older than the kernel you're running. He might have had much better luck with recovery (that is, he might have been able to get a rw volume with all his data, rather than an ro volume -with all his data- out of the process) with a recent tools package. > http://www.spinics.net/lists/linux-btrfs/msg47845.html http://www.spinics.net/lists/linux-btrfs/msg47845.html Maybe I'm reading this one wrong, but it looks like the summary of the issues are: * For multi-disk arrays with disks of varying sizes, each smaller disk does not receive data until all larger disks have as much or less free space than the smaller disk in question. [0] * The read scheduling algo is currently naive, using PID evenness to determine the disk in a pair to read, rather than current IO load on the disk. Did I miss something, or misunderstand something? If I didn't miss anything, I don't see how you can get "RAID is utterly broken" from "RAID has less-than-optimal write and read scheduling". > * It's useless for databases or VM images (disabling datacow is a ugly hack IMHO) I can't agree. I use btrfs on spinning rust to store my multi-terabyte Postgres 9.4 DB. It works well enough for my low-to-medium-volume [1] workload. > * Lot's of ENOSPC bugs (just look at the mailing list) I don't run into those, and haven't for literal years. > * Quota is utterly broken... Yes, quota is a difficult problem that continues to be unsolved. > * You need to run the latest kernel... Yes. Agreed. > ...if not: Data corruption bugs ... I haven't run into any, ever on the handful of systems on which I use BTRFS. > ...if not: ... deadlocks True. There was that unlink/move deadlock in 4.1. I hit it from time to time when updating my copy of the metadata from the Gentoo Portage tree. Happily, it only blocked operation on the affected file. The remainder of the system kept running along, and a -ugh- reboot unblocked the operation. [2] Look. I'm not saying that btrfs is feature-complete, suitable for every workload, or that any given person should use it for their workload. I'm just saying that the situation is far less dire than you're making it out to be. I've been using it as the rootfs for at least one of my daily use systems for the past 5.5 years, and haven't had any trouble out of it [3] in the past 2+ years. [0] That phrasing was abysmal. Here's an example: Given an array of one each of 3GB, 2GB, 1.5GB devices. the 2GB device will get data after 1GB has been written to the 3GB device. The 1.5GB device will get data after 0.5GB of additional data has been written to each of the 3GB and 2GB devices (making a total of 1GB additional data written to the array before the 1.5GB device starts getting data). AIUI, if you're running in RAID1 mode, then the same logic applies, but with paired devices. [1] Most definitely not web-scale workload. [2] I don't know if a unmount/remount cycle would have also worked. The wedged operation happened on my rootfs, so I had no way to test this. [3] Except for the unlink/move deadlock in 4.1.