7 ms·

Native ZFS for Linux

64 points by spahl 16y ago

wnoise 16y ago
Sort of. From the FAQ: 1.3 How do I mount the file system? You can’t… at least not today. While we have ported the majority of the ZFS code to the Linux kernel that does not yet include the ZFS posix layer. The only interface currently available from user space is the ZVOL virtual block device.
- dataguy 16y ago
  ... ZPL, so the Z POSIX Layer, is the really awesome thing. Virtual block device access is neat, though. I wonder when the first one starts building a high performance data base project based on virtual blocks in zfs for linux. Could be interesting.
  - ssmoot 16y ago
    ZVOLs are block devices. So you could install another fs on top of one. Or more usefully export it as an iSCSI target.
    - ithkuil 16y ago
      sorry for the stupid question, but what features I get with ZVOLS exported as iSCSI/AoE versus exporting a LVM device?
- jrmxrf 16y ago
  so what is the big news? I've been using zfs-fuse for 2 years already
- Freaky 16y ago
  As I recall, the FreeBSD ZFS port had ZVOL integrated into GEOM (FreeBSD block device layer) about a week after the porting effort was started. ZPL took significantly longer.
binomial 16y ago
What I'd love is for the L2ARC system built on top of ZFS to be ported as well. That's a killer feature for web applications. Basically, it can extend your cache to include an SSD, but manage that all internally, so you just use your database as usual on top of ZFS and let it handle all the caching and moving less used data from SSD to disk and vice versa, and of course the usual filesystem caching in RAM as well. Though you could get similar functionality by using membase at the expense of being limited to get/set operations (it's a K/V store).
- ssmoot 16y ago
  If it's as small as a single server, you probably don't need an SSD to cache access. If your db server is partitioned out, just put the whole fs on SSD. You can't partition the L2ARC, so you really don't want to mix web assets with database volumes. The assets will push your db out of the cache unless it's so big that you're massively overpowered, or traffic so low it didn't matter in the first place. That's some hard won knowledge there. :-) (You can choose to cache assets meta-data only, but that tuning has major downsides itself and while it may protect your db, it's also likely to render your caches very under-utilized. Basically it isn't a silver bullet and it's still important to think of volumes and work-loads in terms of what "spindles" they're on.
- kmavm 16y ago
  See Facebook's FlashCache if you get tired of waiting: http://www.facebook.com/note.php?note_id=388112370932 http://www.facebook.com/note.php?note_id=388112370932
  - wmf 16y ago
    Or bcache: http://bcache.evilpiepirate.org/FrontPage http://bcache.evilpiepirate.org/FrontPage
    - binomial 16y ago
      Very nice, thanks for both.
steve19 16y ago
It has been successfully ported to FreeBSD and now there is a functional Linux ZFS kernel port too So this is a port and therefor CDDL licesned and no able to be merged into the mainline Linux Kernel, right? Is ZFS still a hot commodity with BTRfs just around the corner?
- patrickgzill 16y ago
  I would say yes, given my experience with ZFS of a number of years in production.
  - mhansen 16y ago
    Please explain?
    - bensummers 16y ago
      Would you like to be one of the first to discover a data-loss bug in brtfs? If not, you probably want the tried and trusted ZFS.
      - spahl 16y ago
        I wouldn't really call this ZFS port "tried and trusted". :-D
        bensummers 16y ago
        I wouldn't use it on Linux! Obviously Solaris would be the 'best' place to use it. But even after a port, the underlying FS will have many more years of production use than brtfs. You might find bugs in the interface between the kernel and ZFS, but hopefully they wouldn't mess with any of the core fs code.
        sigil 16y ago
        Fully agree there. FreeBSD has had ZFS for 3+ years, and I still see some serious issues once in a while. Beware all new filesystems.
nivertech 16y ago
Can you implement shared filesystem on multiple EC2 istances with this ZFS implementation? Which Linux distro is best for it?
- spahl 16y ago
  ZFS is not a distributed filesystem. What you can do is export it over nfs for example. It also has replication features with send/receive but I don't think this implementation supports it. Edit: I notice that a lot of people think ZFS is a cluster/distributed filesystem. I don't understand where they get this idea.
  - nivertech 16y ago
    Shared filesystem is not the same as cluster/distributed filesystem. It's mere a management convenience and letting legacy code work without changes. There is a problem on Amazon EC2, that you can mount EBS (Elastic Block Storage) only to one EC2 instance at a time. I was told by several people (including some from Sun), that the only way to mount the same EBS instance to multiple EC2 instances is to use OpenSolaris and ZFS. I don't think this solution involves NFS. Anyway now when ZFS almost ported to Linux, it would be great to check this issue again (requires ZPL).
    - carson 16y ago
      There is no way of attaching EBS volumes to multiple instances, see the EBS API overview: http://docs.amazonwebservices.com/AWSEC2/2010-08-31/DeveloperGuide/ebs-api-overview.html http://docs.amazonwebservices.com/AWSEC2/2010-08-31/Develope... What might work if you wanted to do this is to mount the EBS volume to an instance and then expose the volume using a network block device to multiple instances. Seems convoluted. Being able to attach EBS to multiple instances may be something Amazon will add if there is enough interest.
    - ssmoot 16y ago
      So I'd guess EBS is nothing more than ISCSI targets then. ZFS won't do anything to make those usable on multiple machines at once. ZFS does make it easy to mirror multiple LUNs for redundancy though, so if you have a timeout or permanent disconnect from the device you remain operational. In addition detaching the device and moving it to another machine is trivial with zfs export/import commands.
c00p3r 16y ago
If it isn't backed by RedHat it is useless. Redhat have his amazing triple develop/test/bugfix process Fedora-development->Fedora->RHEL. But even Fedora is quite stable.
- c00p3r 16y ago
  to stupid down-voters: try to count what amount of testing a standalone package gets compared to the package which is a part of Fedora Project? In term of number of installations? community involved? Fedora's team and infrastructure?
  - wmf 16y ago
    Doesn't Ubuntu have more users than Fedora? You really sound like you're trolling. Maybe you should have said "if it isn't backed by a major distro it's useless".
    - c00p3r 16y ago
      More than Fedora + CentOS + RHEL? Are web-hosting or shared-hosting providers using Ubuntu? What do you thing Amazon or RackSpace are using?
      - moe 16y ago
        http://www.google.com/insights/search/#q=%20Centos%2C%20RedHat%2C%20Fedora%2C%20RHEL%2C%20Ubuntu&cmpt=q http://www.google.com/insights/search/#q=%20Centos%2C%20RedH... RPM based distros generally have a tough standing since Ubuntu began to eclipse everything else in terms of popularity.
        c00p3r 16y ago
        OK. I'll try to explain. This search results showing that there are a lot of Linux home users who're using Ubuntu because it is most popular Desktop distro. Its popularity in a large extend due to money Canonical spent on promotion. (Remember free shipping of CD's and other actions) But if you're considering a server segment, there are traditionally CentOS, RHEL or Fedora. Or even that stupid Unbreakable Linux. Why it is so? (Because it got that way!) First, because most of really important key developers, like mr. Ulrich Drepper, works for RedHat or with Redhat. Redhat also actively supported by IBM, that's why RHEL is a such huge success. Second, since RHEL4, when they made that stupid decision to maintain all the patches by themselves (RHEL4's kernel source rpm had up to 300 patches in it, but I'm not quite sure you know what it is all about.) Now, they work with a mainstream kernel developers, so all their code got a most extensive testing possible. This is what Google's Linux engineers still cannot understand - the best testing and bug hunting is in primary linux source tree. So, what about Ubuntu? Compared to Fedora it is outdated, very conservative, desktop-oriented distro for an average mediocre user. I don't want to go deeper and try to compare source packages of key components, such as kernel, glibc and related, but I'm pretty sure there are different set of patches. One can compare common packages like perl, python, erlang, gcc, php by themselves. Conclusion? OK, 3-5 years ago using Ubuntu or Debian as a server instead of Fedora-derived distros was a sign of non-professionalism, if not a fanboyism or even ignorance. What changed today? Almost nothing. Ubuntu is following the same conservative policies. There are some so-called server editions of Ubuntu, but it is rather a marketing movement. And, you might be surprised, but key kernel developers are using Fedora. =)
leif 16y ago
So...when can I use this to share data between a BSD and a Linux machine?

64 points by spahl 16y ago

Sort of. From the FAQ: 1.3 How do I mount the file system? You can’t… at least not today. While we have ported the majority of the ZFS code to the Linux kernel that does not yet include the ZFS posix layer. The only interface currently available from user space is the ZVOL virtual block device.

dataguy 16y ago
... ZPL, so the Z POSIX Layer, is the really awesome thing. Virtual block device access is neat, though. I wonder when the first one starts building a high performance data base project based on virtual blocks in zfs for linux. Could be interesting.
- ssmoot 16y ago
  ZVOLs are block devices. So you could install another fs on top of one. Or more usefully export it as an iSCSI target.
  - ithkuil 16y ago
    sorry for the stupid question, but what features I get with ZVOLS exported as iSCSI/AoE versus exporting a LVM device?
jrmxrf 16y ago
so what is the big news? I've been using zfs-fuse for 2 years already
Freaky 16y ago
As I recall, the FreeBSD ZFS port had ZVOL integrated into GEOM (FreeBSD block device layer) about a week after the porting effort was started. ZPL took significantly longer.

binomial 16y ago

What I'd love is for the L2ARC system built on top of ZFS to be ported as well. That's a killer feature for web applications. Basically, it can extend your cache to include an SSD, but manage that all internally, so you just use your database as usual on top of ZFS and let it handle all the caching and moving less used data from SSD to disk and vice versa, and of course the usual filesystem caching in RAM as well. Though you could get similar functionality by using membase at the expense of being limited to get/set operations (it's a K/V store).

ssmoot 16y ago
If it's as small as a single server, you probably don't need an SSD to cache access. If your db server is partitioned out, just put the whole fs on SSD. You can't partition the L2ARC, so you really don't want to mix web assets with database volumes. The assets will push your db out of the cache unless it's so big that you're massively overpowered, or traffic so low it didn't matter in the first place. That's some hard won knowledge there. :-) (You can choose to cache assets meta-data only, but that tuning has major downsides itself and while it may protect your db, it's also likely to render your caches very under-utilized. Basically it isn't a silver bullet and it's still important to think of volumes and work-loads in terms of what "spindles" they're on.
kmavm 16y ago
See Facebook's FlashCache if you get tired of waiting: http://www.facebook.com/note.php?note_id=388112370932 http://www.facebook.com/note.php?note_id=388112370932
- wmf 16y ago
  Or bcache: http://bcache.evilpiepirate.org/FrontPage http://bcache.evilpiepirate.org/FrontPage
  - binomial 16y ago
    Very nice, thanks for both.

steve19 16y ago

It has been successfully ported to FreeBSD and now there is a functional Linux ZFS kernel port too So this is a port and therefor CDDL licesned and no able to be merged into the mainline Linux Kernel, right? Is ZFS still a hot commodity with BTRfs just around the corner?

patrickgzill 16y ago
I would say yes, given my experience with ZFS of a number of years in production.
- mhansen 16y ago
  Please explain?
  - bensummers 16y ago
    Would you like to be one of the first to discover a data-loss bug in brtfs? If not, you probably want the tried and trusted ZFS.
    - spahl 16y ago
      I wouldn't really call this ZFS port "tried and trusted". :-D
      - bensummers 16y ago
        I wouldn't use it on Linux! Obviously Solaris would be the 'best' place to use it. But even after a port, the underlying FS will have many more years of production use than brtfs. You might find bugs in the interface between the kernel and ZFS, but hopefully they wouldn't mess with any of the core fs code.
      - sigil 16y ago
        Fully agree there. FreeBSD has had ZFS for 3+ years, and I still see some serious issues once in a while. Beware all new filesystems.

nivertech 16y ago

Can you implement shared filesystem on multiple EC2 istances with this ZFS implementation? Which Linux distro is best for it?

spahl 16y ago
ZFS is not a distributed filesystem. What you can do is export it over nfs for example. It also has replication features with send/receive but I don't think this implementation supports it. Edit: I notice that a lot of people think ZFS is a cluster/distributed filesystem. I don't understand where they get this idea.
- nivertech 16y ago
  Shared filesystem is not the same as cluster/distributed filesystem. It's mere a management convenience and letting legacy code work without changes. There is a problem on Amazon EC2, that you can mount EBS (Elastic Block Storage) only to one EC2 instance at a time. I was told by several people (including some from Sun), that the only way to mount the same EBS instance to multiple EC2 instances is to use OpenSolaris and ZFS. I don't think this solution involves NFS. Anyway now when ZFS almost ported to Linux, it would be great to check this issue again (requires ZPL).
  - carson 16y ago
    There is no way of attaching EBS volumes to multiple instances, see the EBS API overview: http://docs.amazonwebservices.com/AWSEC2/2010-08-31/DeveloperGuide/ebs-api-overview.html http://docs.amazonwebservices.com/AWSEC2/2010-08-31/Develope... What might work if you wanted to do this is to mount the EBS volume to an instance and then expose the volume using a network block device to multiple instances. Seems convoluted. Being able to attach EBS to multiple instances may be something Amazon will add if there is enough interest.
  - ssmoot 16y ago
    So I'd guess EBS is nothing more than ISCSI targets then. ZFS won't do anything to make those usable on multiple machines at once. ZFS does make it easy to mirror multiple LUNs for redundancy though, so if you have a timeout or permanent disconnect from the device you remain operational. In addition detaching the device and moving it to another machine is trivial with zfs export/import commands.

c00p3r 16y ago

If it isn't backed by RedHat it is useless. Redhat have his amazing triple develop/test/bugfix process Fedora-development->Fedora->RHEL. But even Fedora is quite stable.

c00p3r 16y ago
to stupid down-voters: try to count what amount of testing a standalone package gets compared to the package which is a part of Fedora Project? In term of number of installations? community involved? Fedora's team and infrastructure?
- wmf 16y ago
  Doesn't Ubuntu have more users than Fedora? You really sound like you're trolling. Maybe you should have said "if it isn't backed by a major distro it's useless".
  - c00p3r 16y ago
    More than Fedora + CentOS + RHEL? Are web-hosting or shared-hosting providers using Ubuntu? What do you thing Amazon or RackSpace are using?
    - moe 16y ago
      http://www.google.com/insights/search/#q=%20Centos%2C%20RedHat%2C%20Fedora%2C%20RHEL%2C%20Ubuntu&cmpt=q http://www.google.com/insights/search/#q=%20Centos%2C%20RedH... RPM based distros generally have a tough standing since Ubuntu began to eclipse everything else in terms of popularity.
      - c00p3r 16y ago
        OK. I'll try to explain. This search results showing that there are a lot of Linux home users who're using Ubuntu because it is most popular Desktop distro. Its popularity in a large extend due to money Canonical spent on promotion. (Remember free shipping of CD's and other actions) But if you're considering a server segment, there are traditionally CentOS, RHEL or Fedora. Or even that stupid Unbreakable Linux. Why it is so? (Because it got that way!) First, because most of really important key developers, like mr. Ulrich Drepper, works for RedHat or with Redhat. Redhat also actively supported by IBM, that's why RHEL is a such huge success. Second, since RHEL4, when they made that stupid decision to maintain all the patches by themselves (RHEL4's kernel source rpm had up to 300 patches in it, but I'm not quite sure you know what it is all about.) Now, they work with a mainstream kernel developers, so all their code got a most extensive testing possible. This is what Google's Linux engineers still cannot understand - the best testing and bug hunting is in primary linux source tree. So, what about Ubuntu? Compared to Fedora it is outdated, very conservative, desktop-oriented distro for an average mediocre user. I don't want to go deeper and try to compare source packages of key components, such as kernel, glibc and related, but I'm pretty sure there are different set of patches. One can compare common packages like perl, python, erlang, gcc, php by themselves. Conclusion? OK, 3-5 years ago using Ubuntu or Debian as a server instead of Fedora-derived distros was a sign of non-professionalism, if not a fanboyism or even ignorance. What changed today? Almost nothing. Ubuntu is following the same conservative policies. There are some so-called server editions of Ubuntu, but it is rather a marketing movement. And, you might be surprised, but key kernel developers are using Fedora. =)

leif 16y ago

So...when can I use this to share data between a BSD and a Linux machine?