10 ms·

ZFS Root Filesystem on AWS

174 points by lscotte 10y ago

devoply 10y ago
I have messed with ZFS on Linux on Ubuntu and I have to say that I would not yet trust it in production. It's not as bullet proof as it needs to be and still under heavy dev. Not even at version 1.0 yet.
- petre 10y ago
  It's bullet proof on Solaris and FreeBSD.
  - user5994461 10y ago
    Which doesn't say anything about its state on Linux.
    - FooBarWidget 10y ago
      You didn't get the hint. He's saying you should be using Solaris or FreeBSD instead of Linux.
    - ianai 10y ago
      The implementation might be lacking but the underlying FS should be more reliable. I'd still argue that ZFS should be deployed on FreeBSD or Solaris. There are plenty of ways to fire up a Linux environment from there.
- iso-8859-1 10y ago
  Do you trust btrfs? Suse has been having it as the default since 2014...
  - deleted 10y ago
    [deleted]
  - jlgaddis 10y ago
    > The parity RAID code has multiple serious data-loss bugs in it. It should not be used for anything other than testing purposes. [0] [0]: https://btrfs.wiki.kernel.org/index.php/RAID56 https://btrfs.wiki.kernel.org/index.php/RAID56
    - penagwin 10y ago
      Important to note that is only referring to Raid 5 and 6
- phil21 10y ago
  Depends on what you're worried about. Operationally speaking I agree, it's not plug and play. But it's at a point where it safely stores your data correctly. Perhaps some init scripts fail on boot to import your pool/etc. but the data is there. We do run it production, but we also have in-house tooling built around it.
- dmm 10y ago
  Do you have any specific reasons not to trust ZoL? ZFS-on-Linux devs say it's ready for production[1]. Lawrence Livermore laboratory stores petabytes of data using ZoL[2]. If we're sharing anecdotes, ZoL has served me fantastically for several years. [1] https://clusterhq.com/2014/09/11/state-zfs-on-linux/ https://clusterhq.com/2014/09/11/state-zfs-on-linux/ [2] http://computation.llnl.gov/newsroom/livermores-zfs-linux-port-hit-it-industry http://computation.llnl.gov/newsroom/livermores-zfs-linux-po...
  - user5994461 10y ago
    I thought "ZoL" was a pun with ZFS and LOL to tell how not ready it is for production ^^
    - ryao 10y ago
      ZoL is an acronym for ZFSonLinux.
  - devoply 10y ago
    My reasons are as follows: 1) Seen users complaining about data loss on issues on github. 2) Had the init script fail on upgrade and had to fix it by hand when upgrading Ubuntu. Probably a one time issue. Need a bit more reliability from a file system.
  - otterley 10y ago
    We have encountered a reproducible panic and deadlocks when a containerized process gets terminated by the kernel for exceeding its memory limit: https://github.com/zfsonlinux/zfs/issues/5535 https://github.com/zfsonlinux/zfs/issues/5535 We're strongly considering using something else until this gets addressed. The problem is, we don't know what, because every other CoW implementation also has issues. * dm-thinp: Slow, wastes disk space * OverlayFS: No SELinux support * aufs: Not in mainline or CentOS kernel; rename(2) not implemented correctly; slow writes
    - brendangregg 10y ago
      The issue you link to was opened a day ago. If that were me, I'd see how quickly it was fixed before strongly considering something else.
- lscotte 10y ago
  It's maturing rapidly and has proven to be very stable so far. We're not using it by default everywhere, at least not yet, and building out an AMI that uses ZFS for the rootfs is still a bit of a research project - but we have been using it to do RAID0 striping of ephemeral drives for a year or two on a number of workloads.
- jlgaddis 10y ago
  My newly built (Ubuntu 16.04 LTS) workstation is using ZFS exclusively. I'm keeping my fingers crossed.
  - shiftoutbox 10y ago
    My workstation uses FreeBSD 11 with zfs on root and it just works . I don't have to worry about the filesystem.
- deleted 10y ago
  [deleted]
- tnorgaard 10y ago
  We have been running ZFS on Linux in production since April 2015 on over 1500 instances in AWS EC2 with Ubuntu 14.04 and 16.04. Only one kernel panic observed so far, on a Jenkins/CI instance, but that was due to Jenkins doing magic on ZFS mounts, believing it was a Solaris ZFS mount. In our opinion, when we made the switch, it was much more important to trust the integrity of the data, than any possible kernel panic.
- brendangregg 10y ago
  We've actually been running it in production at Netflix for a few microservices for over a year (as Scott said, for a few workloads, but a long way from everywhere). I don't think we've made any kind of announcement, but "ZFS" has shown up in a number of Netflix presentations and slide decks: eg, for Titus (container management). ZFS has worked well on Linux for us. I keep meaning to blog about it, but there's been so many things to share (BPF has kept me more busy). Glad Scott found the time to share the root ZFS stuff.
  - voncopec 10y ago
    If I had to choose between a filesystem with silent and/or visible data corruption up to pretty much eating itself and having to restore an entire server, versus a filesystem for which you can trust but could have a kernel deadlock/panic..I would choose the latter, and in-fact did. I have seen a few servers with ext4/mdraid over the last five years have serious corruption but have had to reset a ZoL server maybe twice.
    - ianhowson 10y ago
      Story time. I transitioned an md RAID1 from spinning disks to SSDs last week. After I removed the last spinning disk, one of the SSDs started returning garbage. 1/3 reads are returning garbage and ext4 freaks out, of course. It's too late and the array is shot. I restore from backup. This would have been a non-event with ZFS. I've got a few production ZoL arrays running and the only problems I've had have been around memory consumption and responsiveness under load. Data integrity has been perfect.
      - voncopec 10y ago
        I've seen the same type of thing with respect to memory and load.
- sigstoat 10y ago
  i've been using zfs on ubuntu since ~2010 for a small set of machines, reading/writing 24/7 with different loads. it's worked great through quite a few drive replacements, and various other hardware failures. i'm perfectly willing to believe there may be some rare situations where zfs on linux will cause you a problem. but i bet they're rare enough it'll have saved you a few times before it bites you.
- jeduardo 10y ago
  Well, we (and by this I mean myself and my fantastic team) have been running it since 2015 as the main filesystem for a double-digit number of KVM hosts running a triple-digit number of virtual machines executing an interesting mix of workloads, ranging from lightweight (file servers for light sharing, web application hosts) to heavy I/O bound ones (databases, build farms) with fantastic results so far. All this on Debian Stable. The setup process was a bit painful given some interesting delays when using some HW storage controllers that caused udev to not make some HDD devices available under /dev before the ZFS scripts kicked in and we have been bitten a couple times by changes (or bugs) in the boot scripts, however the gains provided by ZFS in terms of data integrity, backup, and virtual machine provisioning workflow were definitely worth it.
deleted 10y ago
[deleted]
briankwest 10y ago
I thought you shouldn't be running ZFS inside a virtualized environment?
- zorked 10y ago
  Why?
  - briankwest 10y ago
    Looks like its not an issue, Was just reading the warnings a year or so ago specifically relating to FreeNAS and how it would corrupt your data.
  - jlgaddis 10y ago
    Ideally, ZFS has exclusive control over the storage. When it's virtualized, it doesn't and there may be various HBAs, raid controllers, etc., in between ZFS and the actual disks. These can (do) "get in the way" and you (can) lose one of the biggest features of ZFS (data integrity and error correction).
    - simcop2387 10y ago
      For a little more detail about this, it's somewhat common for these controllers to lie about when thinks hit the disk. They'll cache it and then write either out but if they lose power or a write gets corrupted zfs will have no way of discovering it until it's too late.
      - bsuh 10y ago
        Are other file systems better off with recovery when interacting with virtual disks and host power failure?
        ryao 10y ago
        The short answer is no.
jsjohnst 10y ago
Anybody tried ZFS on Centos or other RHEL like distros?
- jlgaddis 10y ago
  cf. https://github.com/zfsonlinux/zfs/wiki/RHEL-%26-CentOS https://github.com/zfsonlinux/zfs/wiki/RHEL-%26-CentOS
  - jsjohnst 10y ago
    Thanks! I'd read that before, was more looking for anecdata from someone who'd tried it, especially in production.
- EvanAnderson 10y ago
  Ed White has been doing it in production on RHEL/CentOS for a number of years now. He's got a great write-up on a "roll your own" clustered ZFS-based SAN replacement here: https://github.com/ewwhite/zfs-ha/wiki https://github.com/ewwhite/zfs-ha/wiki You can probably hit-up Ed via Github or Server Fault if you want to talk to him about it directly.
- vlahmot 10y ago
  We run Kafka (and I believe Cassandra) on top of Centos/ZFS in AWS and haven't had any issues so far. (About a TB of data moving through every 3 days)
- coredog64 10y ago
  As a follow up question to yours, how about Amazon Linux?
dekhn 10y ago
I've been waiting for Ubuntu to finish getting the native installed to do simple ZFS-root installs for a while now. This document is basically the same process: bootstrap linux on a supported FS, then use user space tools to make a ZFS fs on a new block device, copy linux there, adjust the boot system's pointers, and reboot.
- weitzj 10y ago
  Try out Proxmox (Debian). This works quite well for root ZFS
  - dekhn 10y ago
    i don't want a derivative- I just want pure plain debian or Ubuntu.
- josteink 10y ago
  I've used this guide and it works fine: https://github.com/zfsonlinux/zfs/wiki/Ubuntu-16.04-Root-on-ZFS https://github.com/zfsonlinux/zfs/wiki/Ubuntu-16.04-Root-on-... It however advises to use blockid, not device-id, for mounts. Any idea if this doesn't apply to AMIs?
  - dekhn 10y ago
    My point is I want the installer itself to lay down the original disk FS as ZFS in the installer.
    - josteink 10y ago
      Considering how the current installer supports btrfs, it really doesn't seem like it should take too much effort. Someone however, will have to put in that effort. And maybe people using ZFS want proper volume/subvolume management as good as support for traditional partitions or LVM volumes? If so, it will probably take a while longer to land.
      - 10y ago
machbio 10y ago
for less production instructions on ubuntu 16.04 - here is a simple guide - http://www.howtogeek.com/272220/how-to-install-and-use-zfs-on-ubuntu-and-why-youd-want-to/ http://www.howtogeek.com/272220/how-to-install-and-use-zfs-o...
malisper 10y ago
I'm currently managing a Postgres cluster with a petabyte of data running on ZFS on Linux on AWS. Most of the issues we've come across are around us not knowing ZFS. The first main issue was the arc_shrink_shift default being poor for machines with a large ARC. Our machines have Arc at several 100GB, so the default arc_shrink_shift was flushing several GBs to disk at a time. This was causing our machines to become unresponsive for several seconds at a time pretty frequently. The other main issue we encountered was when we tried to delete lots of data at a time. We aren't sure why, but when we tried to delete a lot of data (~200GB from each machine which each contain several TB of data), our databases become unresponsive for an hour. Other then these issues, ZFS has worked incredibly well. The builtin compression has saved us lots of $$$. It's just the unknown unknowns that have been getting us.
- lomnakkus 10y ago
  Agreed, ZFS has its caveats, but feature-wise and stability-wise ZFS is to -- a large degree -- what BTRFS should have been. The licensing is incredibly unfortunate, though. (I don't care about the reasoning for the license, it's just bad that it isn't GPL-compatible so that it could be compatible with the most prolific kernel in the world.) Anyway, back to BTRFS-vs-ZFS. It seems abundantly clear that a filesystem is (no longer) a thing where you can just "throw an early idea out there" and hope that others will pick up the slack and fix all the bugs. There's just too much design (not code) that goes into these things that it's not just about code any more. My (small) bet right now as to the "next gen" FS on Linux is on bcachefs[1, 2]. It sounds much sounder from a design perspective than BFS, plus it's built on the already-proven bcache, etc. etc. (Read the page for details.) [1] https://www.patreon.com/bcachefs https://www.patreon.com/bcachefs [1] https://bcache.evilpiepirate.org/Bcachefs/ https://bcache.evilpiepirate.org/Bcachefs/
  - jen20 10y ago
    According to Canonical, it _is_ GPL compatible. Either way, that shouldn't get in the way of the best file system in existence being used with the kernel of last resort.
caseyf7 10y ago
Is it ok to run ZFS without ECC RAM?
- kondor6c 10y ago
  Yes, recommended but not required. Even then the underlying hardware has ECC RAM. Good question though.
- X86BSD 10y ago
  Depends who you ask. Can you? Sure. Is it smart to possibly introduce errors into the data path in an otherwise fully checksummed path? Not so much.
- ianhowson 10y ago
  If your memory is error-free already, you'll be fine. If it's imperfect, ZFS will occasionally calculate a checksum wrong and write this to all drives in the array. At some point in the future, like when you read the file, the checksum will fail on all drives and the whole file will be marked corrupt. This gets annoying fast. Your memory is not error-free, but it might be close enough.
- aidenn0 10y ago
  Yes. It always has. The "ZFS needs ECC RAM" meme comes from the fact that on many systems the (non-ecc) RAM is the weakest point in the data integrity path if you are running ZFS. For an analogy, consider a world where car engines have a non-trivial chance to instantly explode when involved in a crash. Then someone comes out with an engine that doesn't explode. People say "you should wear seatbelts if you use this non-exploding engine," but your car has no seatbelts; clearly you are still safer with the non-exploding engine, but all of the sudden, seatbelts are more likely to save your life than before.
  - anoother 10y ago
    In think the concern is more that if you encounter a bit-flip checksumming a block on write, then ZFS will mark the data corrupt on read; and thus, make good data unreadable. A non-checksumming FS would not be vulnerable to this particular issue. On the other hand, would undetected corruption through bit rot be a worse problem? Almost certainly. And considering how vanishingly unlikely such a scenario is, I do agree with your sentiment. That said, I'm not sure if my understanding of the issue is complete and would welcome an explanation of the failure scenarios that [very] occasional RAM bit-flips expose ZFS to.
cmurf 10y ago
This is quite a bit easier to do with Btrfs since there are installers that support it; but also with two neat features lacking in ZFS. 1. Reflink copies. Basically this is a file level snapshot. The metadata is unique to the new file, but it shares (initially) extents with the original. 2. Seed device. The volume is set read-only, mounted, add a 2nd device, remount rw, delete the seed. This causes the seed to be replicated to the 2nd device, but with a new volume UUID. Use case might be to do an installation minimally configured so as to be generic, and then use it as a seed for quickly creating multiple unique instances. Another use case: don't delete the 1st volume. Each VM has two devices: the read only seed (shared), a read write 2nd device (unique). The rw device (sprout) internally references the read-only seed, so you need only mount by the UUID of the sprout. Seed-sprout is something like an overlay, or a volume-wide snapshot.
kim0 10y ago
Can someone explain why the author is aligning to 4096 sectors (2M) boundary. While most tools (gdisk et al) default to 2048 sectors?
- lscotte 10y ago
  I can (I'm the author). The partitions are aligned to 2048 sectors (4096 is evenly divisible by 2048). But also note the first usable sector is 2048, not 0 - so the first partition, although we tell sgdisk to start at 0, actually is from sector 2048-4095. I don't know the exact reason why the first usable sector is 2048 - I believe it has to do with legacy support and MBR compatibility - but I'm not sure of the details.
  - dordoka 10y ago
    I was intrigued by the first usable sector exact reason and found out this link that might be an interesting read http://jdebp.eu./FGA/disc-partition-alignment.html http://jdebp.eu./FGA/disc-partition-alignment.html
daurnimator 10y ago
> We also allow overlay mount on /var. This is an obscure but important bit - when the system initially boots, it will log to /var/log before the /var ZFS filesystem is mounted. You shouldn't need to in a systemd based distro: journald logs to /run until /var comes up. (and then flushes across) See https://www.freedesktop.org/software/systemd/man/journald.conf.html#Storage= https://www.freedesktop.org/software/systemd/man/journald.co...
- lscotte 10y ago
  Shouldn't, perhaps, but we definitely have to (and are running with systemd)! I have not made any effort to understand what is logging early - there's not much that should happen prior to ZFS mounting everything.
  - daurnimator 10y ago
    Did you (accidentally) create the directory /var/log/journal in the rootfs? Make sure your /var is actually empty in the rootfs image.
    - lscotte 10y ago
      Yep, it's empty when the target was created/installed. It's something happening early at boot time, but I haven't diagnosed exactly what's going on. And now that I think about it, I've seen the same thing on local ZFS Debian installs as well - worth digging into when I have some time, likely a bug somewhere.
voncopec 10y ago
I have an Ubuntu image for a thumb drive that I made that does both mbr/efi boot with root on ZoL. I've used it to install ~10 computers now by booting, partitioning, attaching the internal disk to rpool and then detaching the thumbdrive and then installing the bootloader without ever rebooting to have a fully working install after that. It is pretty slick.
ShakataGaNai 10y ago
Anyone have thoughts on how to turn this into something more reproducible, like Packer? Love the idea but would hate to rebuild AMI's by hand every X time for Y regions.
- lscotte 10y ago
  I definitely have thoughts about automating the process :-) - we certainly plan on getting the whole process into an automated, repeatable build.
  - jen20 10y ago
    (Disclaimer - I work at HashiCorp) I'm working on a Packer builder type that will support this use case as we speak. The existing AWS `chroot` builder would likely be sufficient, but requires running from within AWS.
    - lscotte 10y ago
      I'll be curious to hear what you come up with. You should be able to create the image file locally and then shove it into S3, and use AWS just for dd'ing it to EBS.
    - ShakataGaNai 10y ago
      Seems like a perfectly logical pre-req. Easy enough to automate with say a Lambda function to spin up a pre-configured host instance to build the AMI's. So... very exciting, looking forward to hearing more about when it "hits the streets"!
  - ShakataGaNai 10y ago
    ::claps madly with delight::

174 points by lscotte 10y ago

I have messed with ZFS on Linux on Ubuntu and I have to say that I would not yet trust it in production. It's not as bullet proof as it needs to be and still under heavy dev. Not even at version 1.0 yet.

petre 10y ago
It's bullet proof on Solaris and FreeBSD.
- user5994461 10y ago
  Which doesn't say anything about its state on Linux.
  - FooBarWidget 10y ago
    You didn't get the hint. He's saying you should be using Solaris or FreeBSD instead of Linux.
  - ianai 10y ago
    The implementation might be lacking but the underlying FS should be more reliable. I'd still argue that ZFS should be deployed on FreeBSD or Solaris. There are plenty of ways to fire up a Linux environment from there.
iso-8859-1 10y ago
Do you trust btrfs? Suse has been having it as the default since 2014...
- deleted 10y ago
  [deleted]
- jlgaddis 10y ago
  > The parity RAID code has multiple serious data-loss bugs in it. It should not be used for anything other than testing purposes. [0] [0]: https://btrfs.wiki.kernel.org/index.php/RAID56 https://btrfs.wiki.kernel.org/index.php/RAID56
  - penagwin 10y ago
    Important to note that is only referring to Raid 5 and 6
phil21 10y ago
Depends on what you're worried about. Operationally speaking I agree, it's not plug and play. But it's at a point where it safely stores your data correctly. Perhaps some init scripts fail on boot to import your pool/etc. but the data is there. We do run it production, but we also have in-house tooling built around it.
dmm 10y ago
Do you have any specific reasons not to trust ZoL? ZFS-on-Linux devs say it's ready for production[1]. Lawrence Livermore laboratory stores petabytes of data using ZoL[2]. If we're sharing anecdotes, ZoL has served me fantastically for several years. [1] https://clusterhq.com/2014/09/11/state-zfs-on-linux/ https://clusterhq.com/2014/09/11/state-zfs-on-linux/ [2] http://computation.llnl.gov/newsroom/livermores-zfs-linux-port-hit-it-industry http://computation.llnl.gov/newsroom/livermores-zfs-linux-po...
- user5994461 10y ago
  I thought "ZoL" was a pun with ZFS and LOL to tell how not ready it is for production ^^
  - ryao 10y ago
    ZoL is an acronym for ZFSonLinux.
- devoply 10y ago
  My reasons are as follows: 1) Seen users complaining about data loss on issues on github. 2) Had the init script fail on upgrade and had to fix it by hand when upgrading Ubuntu. Probably a one time issue. Need a bit more reliability from a file system.
- otterley 10y ago
  We have encountered a reproducible panic and deadlocks when a containerized process gets terminated by the kernel for exceeding its memory limit: https://github.com/zfsonlinux/zfs/issues/5535 https://github.com/zfsonlinux/zfs/issues/5535 We're strongly considering using something else until this gets addressed. The problem is, we don't know what, because every other CoW implementation also has issues. * dm-thinp: Slow, wastes disk space * OverlayFS: No SELinux support * aufs: Not in mainline or CentOS kernel; rename(2) not implemented correctly; slow writes
  - brendangregg 10y ago
    The issue you link to was opened a day ago. If that were me, I'd see how quickly it was fixed before strongly considering something else.
lscotte 10y ago
It's maturing rapidly and has proven to be very stable so far. We're not using it by default everywhere, at least not yet, and building out an AMI that uses ZFS for the rootfs is still a bit of a research project - but we have been using it to do RAID0 striping of ephemeral drives for a year or two on a number of workloads.
jlgaddis 10y ago
My newly built (Ubuntu 16.04 LTS) workstation is using ZFS exclusively. I'm keeping my fingers crossed.
- shiftoutbox 10y ago
  My workstation uses FreeBSD 11 with zfs on root and it just works . I don't have to worry about the filesystem.
deleted 10y ago
[deleted]
tnorgaard 10y ago
We have been running ZFS on Linux in production since April 2015 on over 1500 instances in AWS EC2 with Ubuntu 14.04 and 16.04. Only one kernel panic observed so far, on a Jenkins/CI instance, but that was due to Jenkins doing magic on ZFS mounts, believing it was a Solaris ZFS mount. In our opinion, when we made the switch, it was much more important to trust the integrity of the data, than any possible kernel panic.
brendangregg 10y ago
We've actually been running it in production at Netflix for a few microservices for over a year (as Scott said, for a few workloads, but a long way from everywhere). I don't think we've made any kind of announcement, but "ZFS" has shown up in a number of Netflix presentations and slide decks: eg, for Titus (container management). ZFS has worked well on Linux for us. I keep meaning to blog about it, but there's been so many things to share (BPF has kept me more busy). Glad Scott found the time to share the root ZFS stuff.
- voncopec 10y ago
  If I had to choose between a filesystem with silent and/or visible data corruption up to pretty much eating itself and having to restore an entire server, versus a filesystem for which you can trust but could have a kernel deadlock/panic..I would choose the latter, and in-fact did. I have seen a few servers with ext4/mdraid over the last five years have serious corruption but have had to reset a ZoL server maybe twice.
  - ianhowson 10y ago
    Story time. I transitioned an md RAID1 from spinning disks to SSDs last week. After I removed the last spinning disk, one of the SSDs started returning garbage. 1/3 reads are returning garbage and ext4 freaks out, of course. It's too late and the array is shot. I restore from backup. This would have been a non-event with ZFS. I've got a few production ZoL arrays running and the only problems I've had have been around memory consumption and responsiveness under load. Data integrity has been perfect.
    - voncopec 10y ago
      I've seen the same type of thing with respect to memory and load.
sigstoat 10y ago
i've been using zfs on ubuntu since ~2010 for a small set of machines, reading/writing 24/7 with different loads. it's worked great through quite a few drive replacements, and various other hardware failures. i'm perfectly willing to believe there may be some rare situations where zfs on linux will cause you a problem. but i bet they're rare enough it'll have saved you a few times before it bites you.
jeduardo 10y ago
Well, we (and by this I mean myself and my fantastic team) have been running it since 2015 as the main filesystem for a double-digit number of KVM hosts running a triple-digit number of virtual machines executing an interesting mix of workloads, ranging from lightweight (file servers for light sharing, web application hosts) to heavy I/O bound ones (databases, build farms) with fantastic results so far. All this on Debian Stable. The setup process was a bit painful given some interesting delays when using some HW storage controllers that caused udev to not make some HDD devices available under /dev before the ZFS scripts kicked in and we have been bitten a couple times by changes (or bugs) in the boot scripts, however the gains provided by ZFS in terms of data integrity, backup, and virtual machine provisioning workflow were definitely worth it.

deleted 10y ago

[deleted]

briankwest 10y ago

I thought you shouldn't be running ZFS inside a virtualized environment?

zorked 10y ago
Why?
- briankwest 10y ago
  Looks like its not an issue, Was just reading the warnings a year or so ago specifically relating to FreeNAS and how it would corrupt your data.
- jlgaddis 10y ago
  Ideally, ZFS has exclusive control over the storage. When it's virtualized, it doesn't and there may be various HBAs, raid controllers, etc., in between ZFS and the actual disks. These can (do) "get in the way" and you (can) lose one of the biggest features of ZFS (data integrity and error correction).
  - simcop2387 10y ago
    For a little more detail about this, it's somewhat common for these controllers to lie about when thinks hit the disk. They'll cache it and then write either out but if they lose power or a write gets corrupted zfs will have no way of discovering it until it's too late.
    - bsuh 10y ago
      Are other file systems better off with recovery when interacting with virtual disks and host power failure?
      - ryao 10y ago
        The short answer is no.

jsjohnst 10y ago

Anybody tried ZFS on Centos or other RHEL like distros?

jlgaddis 10y ago
cf. https://github.com/zfsonlinux/zfs/wiki/RHEL-%26-CentOS https://github.com/zfsonlinux/zfs/wiki/RHEL-%26-CentOS
- jsjohnst 10y ago
  Thanks! I'd read that before, was more looking for anecdata from someone who'd tried it, especially in production.
EvanAnderson 10y ago
Ed White has been doing it in production on RHEL/CentOS for a number of years now. He's got a great write-up on a "roll your own" clustered ZFS-based SAN replacement here: https://github.com/ewwhite/zfs-ha/wiki https://github.com/ewwhite/zfs-ha/wiki You can probably hit-up Ed via Github or Server Fault if you want to talk to him about it directly.
vlahmot 10y ago
We run Kafka (and I believe Cassandra) on top of Centos/ZFS in AWS and haven't had any issues so far. (About a TB of data moving through every 3 days)
coredog64 10y ago
As a follow up question to yours, how about Amazon Linux?

dekhn 10y ago

I've been waiting for Ubuntu to finish getting the native installed to do simple ZFS-root installs for a while now. This document is basically the same process: bootstrap linux on a supported FS, then use user space tools to make a ZFS fs on a new block device, copy linux there, adjust the boot system's pointers, and reboot.

weitzj 10y ago
Try out Proxmox (Debian). This works quite well for root ZFS
- dekhn 10y ago
  i don't want a derivative- I just want pure plain debian or Ubuntu.
josteink 10y ago
I've used this guide and it works fine: https://github.com/zfsonlinux/zfs/wiki/Ubuntu-16.04-Root-on-ZFS https://github.com/zfsonlinux/zfs/wiki/Ubuntu-16.04-Root-on-... It however advises to use blockid, not device-id, for mounts. Any idea if this doesn't apply to AMIs?
- dekhn 10y ago
  My point is I want the installer itself to lay down the original disk FS as ZFS in the installer.
  - josteink 10y ago
    Considering how the current installer supports btrfs, it really doesn't seem like it should take too much effort. Someone however, will have to put in that effort. And maybe people using ZFS want proper volume/subvolume management as good as support for traditional partitions or LVM volumes? If so, it will probably take a while longer to land.
    - 10y ago

machbio 10y ago

for less production instructions on ubuntu 16.04 - here is a simple guide - http://www.howtogeek.com/272220/how-to-install-and-use-zfs-on-ubuntu-and-why-youd-want-to/ http://www.howtogeek.com/272220/how-to-install-and-use-zfs-o...

malisper 10y ago

I'm currently managing a Postgres cluster with a petabyte of data running on ZFS on Linux on AWS. Most of the issues we've come across are around us not knowing ZFS. The first main issue was the arc_shrink_shift default being poor for machines with a large ARC. Our machines have Arc at several 100GB, so the default arc_shrink_shift was flushing several GBs to disk at a time. This was causing our machines to become unresponsive for several seconds at a time pretty frequently. The other main issue we encountered was when we tried to delete lots of data at a time. We aren't sure why, but when we tried to delete a lot of data (~200GB from each machine which each contain several TB of data), our databases become unresponsive for an hour. Other then these issues, ZFS has worked incredibly well. The builtin compression has saved us lots of $$$. It's just the unknown unknowns that have been getting us.

lomnakkus 10y ago
Agreed, ZFS has its caveats, but feature-wise and stability-wise ZFS is to -- a large degree -- what BTRFS should have been. The licensing is incredibly unfortunate, though. (I don't care about the reasoning for the license, it's just bad that it isn't GPL-compatible so that it could be compatible with the most prolific kernel in the world.) Anyway, back to BTRFS-vs-ZFS. It seems abundantly clear that a filesystem is (no longer) a thing where you can just "throw an early idea out there" and hope that others will pick up the slack and fix all the bugs. There's just too much design (not code) that goes into these things that it's not just about code any more. My (small) bet right now as to the "next gen" FS on Linux is on bcachefs[1, 2]. It sounds much sounder from a design perspective than BFS, plus it's built on the already-proven bcache, etc. etc. (Read the page for details.) [1] https://www.patreon.com/bcachefs https://www.patreon.com/bcachefs [1] https://bcache.evilpiepirate.org/Bcachefs/ https://bcache.evilpiepirate.org/Bcachefs/
- jen20 10y ago
  According to Canonical, it _is_ GPL compatible. Either way, that shouldn't get in the way of the best file system in existence being used with the kernel of last resort.

caseyf7 10y ago

Is it ok to run ZFS without ECC RAM?

kondor6c 10y ago
Yes, recommended but not required. Even then the underlying hardware has ECC RAM. Good question though.
X86BSD 10y ago
Depends who you ask. Can you? Sure. Is it smart to possibly introduce errors into the data path in an otherwise fully checksummed path? Not so much.
ianhowson 10y ago
If your memory is error-free already, you'll be fine. If it's imperfect, ZFS will occasionally calculate a checksum wrong and write this to all drives in the array. At some point in the future, like when you read the file, the checksum will fail on all drives and the whole file will be marked corrupt. This gets annoying fast. Your memory is not error-free, but it might be close enough.
aidenn0 10y ago
Yes. It always has. The "ZFS needs ECC RAM" meme comes from the fact that on many systems the (non-ecc) RAM is the weakest point in the data integrity path if you are running ZFS. For an analogy, consider a world where car engines have a non-trivial chance to instantly explode when involved in a crash. Then someone comes out with an engine that doesn't explode. People say "you should wear seatbelts if you use this non-exploding engine," but your car has no seatbelts; clearly you are still safer with the non-exploding engine, but all of the sudden, seatbelts are more likely to save your life than before.
- anoother 10y ago
  In think the concern is more that if you encounter a bit-flip checksumming a block on write, then ZFS will mark the data corrupt on read; and thus, make good data unreadable. A non-checksumming FS would not be vulnerable to this particular issue. On the other hand, would undetected corruption through bit rot be a worse problem? Almost certainly. And considering how vanishingly unlikely such a scenario is, I do agree with your sentiment. That said, I'm not sure if my understanding of the issue is complete and would welcome an explanation of the failure scenarios that [very] occasional RAM bit-flips expose ZFS to.

cmurf 10y ago

This is quite a bit easier to do with Btrfs since there are installers that support it; but also with two neat features lacking in ZFS. 1. Reflink copies. Basically this is a file level snapshot. The metadata is unique to the new file, but it shares (initially) extents with the original. 2. Seed device. The volume is set read-only, mounted, add a 2nd device, remount rw, delete the seed. This causes the seed to be replicated to the 2nd device, but with a new volume UUID. Use case might be to do an installation minimally configured so as to be generic, and then use it as a seed for quickly creating multiple unique instances. Another use case: don't delete the 1st volume. Each VM has two devices: the read only seed (shared), a read write 2nd device (unique). The rw device (sprout) internally references the read-only seed, so you need only mount by the UUID of the sprout. Seed-sprout is something like an overlay, or a volume-wide snapshot.

kim0 10y ago

Can someone explain why the author is aligning to 4096 sectors (2M) boundary. While most tools (gdisk et al) default to 2048 sectors?

lscotte 10y ago
I can (I'm the author). The partitions are aligned to 2048 sectors (4096 is evenly divisible by 2048). But also note the first usable sector is 2048, not 0 - so the first partition, although we tell sgdisk to start at 0, actually is from sector 2048-4095. I don't know the exact reason why the first usable sector is 2048 - I believe it has to do with legacy support and MBR compatibility - but I'm not sure of the details.
- dordoka 10y ago
  I was intrigued by the first usable sector exact reason and found out this link that might be an interesting read http://jdebp.eu./FGA/disc-partition-alignment.html http://jdebp.eu./FGA/disc-partition-alignment.html

daurnimator 10y ago

> We also allow overlay mount on /var. This is an obscure but important bit - when the system initially boots, it will log to /var/log before the /var ZFS filesystem is mounted. You shouldn't need to in a systemd based distro: journald logs to /run until /var comes up. (and then flushes across) See https://www.freedesktop.org/software/systemd/man/journald.conf.html#Storage= https://www.freedesktop.org/software/systemd/man/journald.co...

lscotte 10y ago
Shouldn't, perhaps, but we definitely have to (and are running with systemd)! I have not made any effort to understand what is logging early - there's not much that should happen prior to ZFS mounting everything.
- daurnimator 10y ago
  Did you (accidentally) create the directory /var/log/journal in the rootfs? Make sure your /var is actually empty in the rootfs image.
  - lscotte 10y ago
    Yep, it's empty when the target was created/installed. It's something happening early at boot time, but I haven't diagnosed exactly what's going on. And now that I think about it, I've seen the same thing on local ZFS Debian installs as well - worth digging into when I have some time, likely a bug somewhere.

voncopec 10y ago

I have an Ubuntu image for a thumb drive that I made that does both mbr/efi boot with root on ZoL. I've used it to install ~10 computers now by booting, partitioning, attaching the internal disk to rpool and then detaching the thumbdrive and then installing the bootloader without ever rebooting to have a fully working install after that. It is pretty slick.

ShakataGaNai 10y ago

Anyone have thoughts on how to turn this into something more reproducible, like Packer? Love the idea but would hate to rebuild AMI's by hand every X time for Y regions.

lscotte 10y ago
I definitely have thoughts about automating the process :-) - we certainly plan on getting the whole process into an automated, repeatable build.
- jen20 10y ago
  (Disclaimer - I work at HashiCorp) I'm working on a Packer builder type that will support this use case as we speak. The existing AWS `chroot` builder would likely be sufficient, but requires running from within AWS.
  - lscotte 10y ago
    I'll be curious to hear what you come up with. You should be able to create the image file locally and then shove it into S3, and use AWS just for dd'ing it to EBS.
  - ShakataGaNai 10y ago
    Seems like a perfectly logical pre-req. Easy enough to automate with say a Lambda function to spin up a pre-configured host instance to build the AMI's. So... very exciting, looking forward to hearing more about when it "hits the streets"!
- ShakataGaNai 10y ago
  ::claps madly with delight::