12 ms·
In Praise of ZFS on Linux's ZED 'ZFS Event Daemon'
- yjftsjthsd-h 6y agoThat's neat, certainly, but I'm struggling to think what to actually use this for? The article mentions a couple cases, of which "taking action if devices fail" seems the more concrete; AFAIK that would make it probably straightforward to replace a drive with a spare? Anybody want to share any other concrete uses?
- motakraxxer 6y agoMail sysadmins about the drive failures for example.
- yjftsjthsd-h 6y ago... thank you, that was the example that made me realize I can/should go use this at work because it will solve a problem we have:) And a good point in general.
- onei 6y agoAre mail alerts not already a thing on your hardware? I know HP(E) stuff does it, not sure about Dell or other vendors.
- simonjgreen 6y agoDrive controller noticing a failure vs ZFS throwing a disk are very distinct things. You want alerting at both levels.
- CaliforniaKarl 6y agoI think you are referring to one of two HPE components: Either the iLO, or the iLO combined with a Smart Array controller. (Dell has equivalent products: For this discussion, iDRAC is the equivalent of iLO, and PERC is the equivalent to Smart Array.) With those products, the RAID controller (Smart Array or PERC) will be connected to internal and/or external drives, will handle RAID in hardware (ideally with a battery backup write cache), and (through the iLO or iDRAC) generate alerts when a drive fails (or is close to failure). In the context of ZFS, you don't have that. Your drive controller is either on-motherboard, or a PCIe card like (for example) the (Broadcom) LSI SAS 9300-8e. Those cards do have a RAID option (MegaRAID), but they are often used without. The rest of the ZFS storage setup is pretty similar to the setup you are used to: Internal drives will have a SAS expander (if needed) on the motherboard, or will use a SAS expander card (for example, an HPE part #870549-B21 SAS expander card). External drives will be in a JBOD that has one or two expanders, and which is connected back to the server using SAS cables. One ZFS difference is that if you have many JBODs, instead of daisy-chaining arrays, you might choose to use a SAS switch (for example, an A54812-SW-01). With all that I described with ZFS, I haven't mentioned how RAID is handled: ZFS handles RAID in software. RAID Z-1 is equivalent to RAID 5, Z-2 to 6, and Z-3 to 7. RAID also supports RAID 1 (two-drive mirrors), as well as a RAID 10 equivalent (striping across mirrored pairs of drives). Since RAID is handled in software, and with the physical equipment I described, it is left to the OS to handle almost all monitoring an alerting. The one exception is that JBODs and rack-mount SAS switches (like the Astek) often have an Ethernet connection for monitoring and (basic) hardware control. But even that can often be handled within the OS, using SCSI Enclosure Services (SES, where the enclosure/switch itself is a device the OS can see and query).
- laumars 6y agoMail/Slack sysadmins seems like an ideal use case. Tempted to set this up on my own storage pools today.
- deelowe 6y agoMail on certain events (scrub start/finish, warnings, errors, failures), automatically replace failed drives, and all sorts of other things. ZED, for example, can be used to create systemd mounts at boot.
- Geezus_42 6y agoOn drive failure automatically swap failed drive for hot spare and resilver.
- ncrmro 6y agoJust installed Ubuntu on a zfs with native zfs encryption. Working real nice
- acd 6y agoInsightful article. Zfs is the best file system. Ability to know that you do not have silent file corruption. Running without raid controllers. Feature to take snapshots super fast without waiting and that take little extra space. You can use Cache SSD with ZFS for read acceleration of physical disk. Transparent file compression. Good command line interface. Now you can take good actions with ZED for example sending a notification to an alert system like Slack, ticket system when disk fails or start disk scrubbing/rebuild. If you are on Linux I can highly recommend ZFS and Minio for S3 like storage. ZFS for local storage.
- arianvanp 6y agoI don't understand where minio suddenly comes from? Minio is barely documented. I had to ask in Slack to interpret what it meant when minio said "your cluster is 5 red and 7 yellow" as the colours aren't event documented. Every minio cluster I hosted had dataloss. Each one has lead to a reported issue on their GitHub that hasn't been closed to date. Nothing about recovery is documented. Documentation is slim in general. Really confused why you're naming it next to ZFS which is sublimely documented and withstood the test of time. I'd advise something battletested through time like Ceph for object storage instead.
- wfn 6y agoCould you expand more re: your minio experience? We're about to enter production with minio as our S3 interface-providing file storage system, and have found documentation to be sufficient. However, I'm slightly concerned that we haven't tested it enough, and that doc and support may prove to be lacking (like in your case) when we hit edge cases and failure scenarios.
- ddorian43 6y agoI'm checking it out too. No LTS version. Bugs will be fixed in a new version, which may include new bugs. Some upgrades need full cluster restart. If you ask too many question you may get "it's better start a subscription".
- 6y ago
- jclulow 6y agoDespite what the article says about illumos not having something analogous, we do actually have something and have had it for more than a decade: https://illumos.org/man/1M/syseventadm https://illumos.org/man/1M/syseventadm It allows programs to be run in response to sysevents, some of which are generated by ZFS and some of which are generated by other parts of the system (e.g., device hotplug).
- trasz 6y agoSounds similar to FreeBSD's devd(8) (https://www.freebsd.org/cgi/man.cgi?devd https://www.freebsd.org/cgi/man.cgi?devd).
- otterley 6y agoWhere are the events that ZFS emits documented?
- spullara 6y agoTyping your question into Google reveals that they aren't properly documented: http://manpages.ubuntu.com/manpages/bionic/man5/zfs-events.5.html http://manpages.ubuntu.com/manpages/bionic/man5/zfs-events.5...
- gnufx 6y agoThat seems the right approach. Why would you want anything ZFS-specific -- other than events not being documented, which presumably means you dig them out of ZED?
- StavrosK 6y agoHas anyone noticed really slow deletes on ZFS? I have a home NAS with RAIDZ and deleting a 1 GB file takes about 30 seconds. I asked on IRC repeatedly but the disks just aren't very busy during the delete (or at all) and nobody managed to figure out why this is happening. It's been that way for years throughout various ZFS versions, and it's driving me crazy.
- mkj 6y agoTry temporarily setting sync=disabled ? If that speeds it maybe a fast ssd log device would fix it.
- StavrosK 6y agoHmm, interesting, I'll try that, thanks!
- ubercow13 6y agoIs it slow if you delete a file locally on the NAS instead of via a network file system?
- StavrosK 6y agoYes, "rm <file>" locally is what's slow :/
- briffle 6y agoZFS deletes can be really slowed down by having dedupe enabled on the pool.
- StavrosK 6y agoIt's off, but I'll triple check, thank you.
- Demiurge 6y agoI have been using Ubuntu + LXD + RaidZ for as long as it was available. Recently (a year ago?) I first had noticed some kind of unclear performance issue with a FLEX container. Heavy file streaming/download would hose and make entire server unusable. I had no idea what to do, and I pretty much randomly converted the LXD storage to BTRFS, and everything was fine. Last few months I ran into same problem with a bunch of PostgreSQL containers doings lots of writes, and same solution helped. Not sure if it's a ZFS problem or LXD, but I will not use ZFS in high load scenario again.
- Mekantis 6y agoI'd love to switch to ZFS, but the RAM requirements are absurd. I don't have a separate storage server, and I'm not really to sacrifice 10GB of RAM (1GB/TB of storage if I'm to believe what I find through Google) on my home desktop just for it when the vast majority of my data could probably handle a rotted bit or two. I did briefly try ZFS on my laptop a year or so ago, and it ate up half of my RAM permanently. Since it was already fairly limited, that wasn't a sacrifice I wasn't willing to make either when I have plenty of backups anyway.
- vetinari 6y agoThat RAM is needed only when you run deduplication (you have to store the checksums of blocks that you deduplicate somewhere). If you don't, the RAM requirements are similar to other filesystems. On your home desktop, you don't have to run dedup. You will get still the bitrot protection.
- Mekantis 6y agoWhy wouldn't I want deduplication though? "You can use ZFS just fine, just turn off one of its most useful features." Really? And I'm being downvoted for it. Thanks, guys. You realize my home desktop is doubling as my storage, right? Which goes back to RAM requirements being an issue. I hate the cargoculting on this fucking site.
- vetinari 6y ago> Why wouldn't I want deduplication though? On a desktop? Chances are, that you don't have many users saving the same or slightly modified version of a file on the same storage. For a single person, it doesn't make much sense. > "You can use ZFS just fine, just turn off one of its most useful features." ZFS has many useful features. They come with a price though, because there's no free computation (see also laws of thermodynamics). It is then a matter of deciding, which features you want or need, and are willing to pay the price for. You obviously are not willing the pay the price for dedup (lvmvdo asks for similar price, so it is not ZFS-specific), so why are you complaining that you cannot use it? ZFS still has many more useful features. You also have another option: add RAM to your desktop. It is cheap. Then you will be able to use that one feature. > I hate the cargoculting on this fucking site. Sigh. I'm actually btrfs fan, all my data are on a btrfs volume (at work, we do use ZFS though, so I do have the experience). But that doesn't mean I won't point out something that the other club does well.
- kim0 6y agoAny way to get commercial support for ZoL ?
- ksec 6y agoQnap will hopefully have an ZFS based Home NAS out soon. Are there any simple to use tools that automatically compare same files from different source and tell if they are different. I have multiple copies of Data but not knowing which file is corrupted and working through it is a pain in the bottom.