4 ms·

tcaputi works with me at Datto. We use ZFS as our main file system for some 200+ PB of storage. It is a truly amazing file system, and this change, once merged,

by wspeirs 10y ago

tcaputi works with me at Datto. We use ZFS as our main file system for some 200+ PB of storage. It is a truly amazing file system, and this change, once merged, will add another layer of security for our customers. Great stuff!

okket 10y ago
At this scale, why ZoL and not a more 'native' ZFS with FreeBSD (or even Illumos)? Which also perform better and have more features? Is it the lack of drivers? Better availability of other 3rd party software for Linux? The package manager? Or just "more brains in Linux"/"our CTO grew up with Linux"?
- abrookewood 10y ago
  I thought that ZoL and ZFS+FreeBSD were largely feature comparable?
  - justincormack 10y ago
    Other than orders of magnitude more testing and experience and usage on FreeBSD. Much code is the same, but the glue may have bugs.
    - pantalaimon 10y ago
      Well Linux receives orders of magnitude more testing and usage than FreeBSD, so I don't see what your argument is
      - fyolnish 10y ago
        His argument refers to the current state of this specific component.
        pantalaimon 10y ago
        ZFS on Linux has been around for 3 years now and is integrated in Ubuntu 16.04 - I'd argue that by now it has received more usage than the BSD version.
        Annatar 10y ago
        ZFS on Linux will always lag in fixes and features because OpenZFS is developed on and for illumos first and foremost.
        mioelnir 10y ago
        You can argue that the earth is shaped like a torus. I can argue that the moon is violet. That is not the point, and has nothing to do with the merit of the argument. Setting this aside, all usage hours are not equal. How do you compare an hour of enduser desktop usage on Ubuntu 23.57.something to a usage hour of a commercial NAS storage solution internally based on FreeBSD and ZFS? Do you think the enduser performed comparable product testing cycles? I've seen ZOL systems where the zpool claimed it was online, but contained no vdevs. `zfs list` had no datasets, but datasets where mounted and trying to read from them got your process stuck in-kernel. It just lost/forgot its devices. Up until one of the latest releases on every boot you could roll the dice by which name your pool would import the devices. Behaving differently on identical machines and setups. Personally, I am still not trusting that problem to not reappear again. ZOL is bolted on. With a large nailgun. Simple as that. At times, it feels about as integrated as pjd's original patches distributed on the FreeBSD mailinglists. And since this division is not technical, but legal, based on the license choice made 30 something years ago for Linux with regards to where the code could be exported to and what could be imported into, this situation will not resolve itself.
  - Annatar 10y ago
    No; run zpool upgrade -v; zfs upgrade -v on illumos and on Linux and you will discover the truth. (upgrade -v lists the versions and features instead of performing an actual upgrade.)
- gnoway 10y ago
  Is there a meaningful performance difference between ZFS in FreeBSD or Illumos and ZoL? Edit: this is a legitimate question. I know previously Linux-related ZFS efforts used FUSE, but ZoL is native. I assume performance should be roughly equivalent between Linux and FreeBSD, certainly.
  - Annatar 10y ago
    Better observability with Dtrace, mdb, iostat, and vmstat on illumos-based systems for sure. Also simple logic dictatates that if something is made on something for that certain something, it's going to run best on that something. Linux has ZFS, but it's grafted on and the illumos POSIX layer is emulated in that sense. Further, Linux's version of OpenZFS will always lag behind fixes and features in the illumos-based systems; even FreeBSD usually contains newer vesions of OpenZFS sooner than ZFS on Linux does. Linux is just "the last hole on the porting flute". Also, experience tells me that illumos and FreeBSD based systems will always perform faster than Linux with regads to ZFS but I'd have to publish a full line of benchmarks and I bet even those would get vehemently disputed because Linux is all the rage right now, so that's a lost cause: you have to try it for yourself and make up your own mind.

by wspeirs 10y ago

okket 10y ago

At this scale, why ZoL and not a more 'native' ZFS with FreeBSD (or even Illumos)? Which also perform better and have more features? Is it the lack of drivers? Better availability of other 3rd party software for Linux? The package manager? Or just "more brains in Linux"/"our CTO grew up with Linux"?

abrookewood 10y ago
I thought that ZoL and ZFS+FreeBSD were largely feature comparable?
- justincormack 10y ago
  Other than orders of magnitude more testing and experience and usage on FreeBSD. Much code is the same, but the glue may have bugs.
  - pantalaimon 10y ago
    Well Linux receives orders of magnitude more testing and usage than FreeBSD, so I don't see what your argument is
    - fyolnish 10y ago
      His argument refers to the current state of this specific component.
      - pantalaimon 10y ago
        ZFS on Linux has been around for 3 years now and is integrated in Ubuntu 16.04 - I'd argue that by now it has received more usage than the BSD version.
        Annatar 10y ago
        ZFS on Linux will always lag in fixes and features because OpenZFS is developed on and for illumos first and foremost.
        mioelnir 10y ago
        You can argue that the earth is shaped like a torus. I can argue that the moon is violet. That is not the point, and has nothing to do with the merit of the argument. Setting this aside, all usage hours are not equal. How do you compare an hour of enduser desktop usage on Ubuntu 23.57.something to a usage hour of a commercial NAS storage solution internally based on FreeBSD and ZFS? Do you think the enduser performed comparable product testing cycles? I've seen ZOL systems where the zpool claimed it was online, but contained no vdevs. `zfs list` had no datasets, but datasets where mounted and trying to read from them got your process stuck in-kernel. It just lost/forgot its devices. Up until one of the latest releases on every boot you could roll the dice by which name your pool would import the devices. Behaving differently on identical machines and setups. Personally, I am still not trusting that problem to not reappear again. ZOL is bolted on. With a large nailgun. Simple as that. At times, it feels about as integrated as pjd's original patches distributed on the FreeBSD mailinglists. And since this division is not technical, but legal, based on the license choice made 30 something years ago for Linux with regards to where the code could be exported to and what could be imported into, this situation will not resolve itself.
- Annatar 10y ago
  No; run zpool upgrade -v; zfs upgrade -v on illumos and on Linux and you will discover the truth. (upgrade -v lists the versions and features instead of performing an actual upgrade.)
gnoway 10y ago
Is there a meaningful performance difference between ZFS in FreeBSD or Illumos and ZoL? Edit: this is a legitimate question. I know previously Linux-related ZFS efforts used FUSE, but ZoL is native. I assume performance should be roughly equivalent between Linux and FreeBSD, certainly.
- Annatar 10y ago
  Better observability with Dtrace, mdb, iostat, and vmstat on illumos-based systems for sure. Also simple logic dictatates that if something is made on something for that certain something, it's going to run best on that something. Linux has ZFS, but it's grafted on and the illumos POSIX layer is emulated in that sense. Further, Linux's version of OpenZFS will always lag behind fixes and features in the illumos-based systems; even FreeBSD usually contains newer vesions of OpenZFS sooner than ZFS on Linux does. Linux is just "the last hole on the porting flute". Also, experience tells me that illumos and FreeBSD based systems will always perform faster than Linux with regads to ZFS but I'd have to publish a full line of benchmarks and I bet even those would get vehemently disputed because Linux is all the rage right now, so that's a lost cause: you have to try it for yourself and make up your own mind.