4 ms·

[W]orking through the git history of ZoL I have also discovered that many races and locking bugs have been fixed in ZoL and never made it back to Illumo

[W]orking through the git history of ZoL I have also discovered that many races and locking bugs have been fixed in ZoL and never made it back to Illumos and thus FreeBSD. This state of affairs has led to a general agreement among the stakeholders that I have spoken to that it makes sense to rebase FreeBSD's ZFS on ZoL. This port will provide FreeBSD users with multi modifier protection, project quotas, encrypted datasets, allocation classes, vectorized raidz, vectorized checksums, and various command line improvements. From content, parent reply is correct. ZoL was far ahead FreeBSD's native ZFS port, that FreeBSD devs found that is better using ZoL instead. Edit: Also related presentation by A. Jude at https://2019.eurobsdcon.org/slides/The%20Future%20of%20OpenZFS%20and%20FreeBSD%20-%20Allan%20Jude.pdf https://2019.eurobsdcon.org/slides/The%20Future%20of%20OpenZ...

hnlmorg 6y ago
> From content, parent reply is correct. ZoL was far ahead FreeBSD's native ZFS port, that FreeBSD devs found that is better using ZoL instead. The FreeBSD fork also had a bunch of ZFS features that hadn't yet made it to ZoL as well as their own bug fixes too. I know this first hand from porting volumes between ZoL and FreeBSD. FreeBSD didn't switch their code base and dump everything they had in their own fork. They switched upstream so that everyone worked on the same common code base and merged what was practical and kept any of their own OS dependent code separate. Just as I said in my previous post and just as the presentation you linked to also confirms. And for what it's worth, it's the OS dependant stuff that makes ZFS's experience better on FreeBSD than it is on Linux. I also don't understand why so many people want to assume a negative spin on the story. Projects fork and change upstreams all the time when it's identified that there's a commonality they can share resources on. This isn't any different.
- forgotpwd16 6y ago
  I don't see it as negative relying to an upstream with larger developer base, and as you said this allows better collaboration. But this dependence makes it hard for me to see how, though once it was, this is FreeBSD's killer feature. How is ZFS, now based on ZFS-on-Linux, more native on FreeBSD than Linux?
  - tw04 6y ago
    Today that's easy to answer: Try setting up ZFS boot with a mirrored boot drive and encryption on FreeBSD, then try it on Redhat, or SLES, or Ubuntu Server. One of the experiences is very much "native" - the other is possible, but a GIANT pain to accomplish. At some point I'm sure, Ubuntu at least, will make installing to ZFS on a server OS more straightforward. I've got several systems doing it today, but I can tell you it's not for the faint-hearted and it took a TON of trial and error to get it to do what I wanted consistently. I also cringe every time I apply an update because I know it's not something that is getting any significant testing by Canonical or the community at large.
    - hnlmorg 6y ago
      My production Linux+ZFS server is still running a LVM mirror with ext4 on root because of exactly the fear you mentioned. All the FreeBSD systems have had mirrored ZFS root since v10.x (released 20 January 2014 according to Wikipedia). 7+ years later and I'm still not confident enough to run the same set up in production using Linux.
      - nix23 6y ago
        Do i smell Proxmox here? Fantastic system!!
        hnlmorg 6y ago
        Not here but I have used Proxmox before (v1 -- back when containerisation in Linux required a custom kernel) and it was pretty nice.
  - hnlmorg 6y ago
    I'd already answered that -- both in my OP and in the comment you've just replied to. It's the OS specific code. Using ZFS on FreeBSD is as natural as using ext4 on Linux. Sure, ZoL exists and some distro's ship pre-compiled binaries for ZFS, but if your server drops into an emergency shell or even fails to boot entirely, if you need to boot from a live image (CD/USB) or your boot volume gets cocked up... fixing all of these requires extra hurdles on Linux if root is ZFS. Whereas that isn't the case on FreeBSD. There's other OS specific features that FreeBSD does better with regards to ZFS such as ACLs too. But like with trashed volumes, they're edge cases. FreeBSD also has better bundled tooling for working with ZFS. By this I don't mean `zfs` and `zpool` but little shell scripts et al that are FreeBSD specific. Then there is the documentation. If you're running Linux then all the documentation regarding server management is about standard server management (ie with an ext4 or xfs root). But FreeBSD documentation will talk about how to do management with the assumption you're running ZFS. The FreeBSD docs (which are excellent, by the way) has pages and pages on ZFS outside of the stuff published by ZoL (which is pretty poor it has to be said) and OpenZFS. And I say the last points as someone who's had to manually compile kernel modules on Linux to get the latest patch to support a feature that made it to FreeBSD first. I do have servers running ZFS + Linux and they work really well. So I'm not taking anything away from that architecture. But when things go wrong or you're trying to do something a little more specialised outside of basic file storage, ZFS has always felt like an awesome hack on top of Linux rather than an integral part of it's design. Unlike with FreeBSD where the same people managing the kernel and userland are the same people pushing ZFS as the default file system. If Torvalds, Pottering or others held ZoL in the same esteem then I'm sure the edge cases I've described would go away (Linux certainly has the larger developer community).

hnlmorg 6y ago

> From content, parent reply is correct. ZoL was far ahead FreeBSD's native ZFS port, that FreeBSD devs found that is better using ZoL instead. The FreeBSD fork also had a bunch of ZFS features that hadn't yet made it to ZoL as well as their own bug fixes too. I know this first hand from porting volumes between ZoL and FreeBSD. FreeBSD didn't switch their code base and dump everything they had in their own fork. They switched upstream so that everyone worked on the same common code base and merged what was practical and kept any of their own OS dependent code separate. Just as I said in my previous post and just as the presentation you linked to also confirms. And for what it's worth, it's the OS dependant stuff that makes ZFS's experience better on FreeBSD than it is on Linux. I also don't understand why so many people want to assume a negative spin on the story. Projects fork and change upstreams all the time when it's identified that there's a commonality they can share resources on. This isn't any different.

forgotpwd16 6y ago
I don't see it as negative relying to an upstream with larger developer base, and as you said this allows better collaboration. But this dependence makes it hard for me to see how, though once it was, this is FreeBSD's killer feature. How is ZFS, now based on ZFS-on-Linux, more native on FreeBSD than Linux?
- tw04 6y ago
  Today that's easy to answer: Try setting up ZFS boot with a mirrored boot drive and encryption on FreeBSD, then try it on Redhat, or SLES, or Ubuntu Server. One of the experiences is very much "native" - the other is possible, but a GIANT pain to accomplish. At some point I'm sure, Ubuntu at least, will make installing to ZFS on a server OS more straightforward. I've got several systems doing it today, but I can tell you it's not for the faint-hearted and it took a TON of trial and error to get it to do what I wanted consistently. I also cringe every time I apply an update because I know it's not something that is getting any significant testing by Canonical or the community at large.
  - hnlmorg 6y ago
    My production Linux+ZFS server is still running a LVM mirror with ext4 on root because of exactly the fear you mentioned. All the FreeBSD systems have had mirrored ZFS root since v10.x (released 20 January 2014 according to Wikipedia). 7+ years later and I'm still not confident enough to run the same set up in production using Linux.
    - nix23 6y ago
      Do i smell Proxmox here? Fantastic system!!
      - hnlmorg 6y ago
        Not here but I have used Proxmox before (v1 -- back when containerisation in Linux required a custom kernel) and it was pretty nice.
- hnlmorg 6y ago
  I'd already answered that -- both in my OP and in the comment you've just replied to. It's the OS specific code. Using ZFS on FreeBSD is as natural as using ext4 on Linux. Sure, ZoL exists and some distro's ship pre-compiled binaries for ZFS, but if your server drops into an emergency shell or even fails to boot entirely, if you need to boot from a live image (CD/USB) or your boot volume gets cocked up... fixing all of these requires extra hurdles on Linux if root is ZFS. Whereas that isn't the case on FreeBSD. There's other OS specific features that FreeBSD does better with regards to ZFS such as ACLs too. But like with trashed volumes, they're edge cases. FreeBSD also has better bundled tooling for working with ZFS. By this I don't mean `zfs` and `zpool` but little shell scripts et al that are FreeBSD specific. Then there is the documentation. If you're running Linux then all the documentation regarding server management is about standard server management (ie with an ext4 or xfs root). But FreeBSD documentation will talk about how to do management with the assumption you're running ZFS. The FreeBSD docs (which are excellent, by the way) has pages and pages on ZFS outside of the stuff published by ZoL (which is pretty poor it has to be said) and OpenZFS. And I say the last points as someone who's had to manually compile kernel modules on Linux to get the latest patch to support a feature that made it to FreeBSD first. I do have servers running ZFS + Linux and they work really well. So I'm not taking anything away from that architecture. But when things go wrong or you're trying to do something a little more specialised outside of basic file storage, ZFS has always felt like an awesome hack on top of Linux rather than an integral part of it's design. Unlike with FreeBSD where the same people managing the kernel and userland are the same people pushing ZFS as the default file system. If Torvalds, Pottering or others held ZoL in the same esteem then I'm sure the edge cases I've described would go away (Linux certainly has the larger developer community).