12 ms·

ZFS is mysteriously eating my CPU

332 points by mfrw 5y ago

waynesonfire 5y ago
Nice job!
jeffbee 5y ago
Memory reclaim is where all the kernel bugs are.
- amelius 5y ago
  Is this an argument for using Rust?
  - mhh__ 5y ago
    No amount of avoiding memory safety bugs will save you from writing fundamentally incorrect code. Rust stops you making dumb mistakes, and (as a high-level language, this part isn't Unique to rust in any way) lets you create abstractions to stop other people making less dumb mistakes, but will do absolutely nothing if your model of the process is wrong. The only way to really be sure of that is extensive testing at multiple levels, and ideally some form of verification like a TLA+ model.
  - marcan_42 5y ago
    No. Memory reclaim != memory leaks != memory safety. This is about filesystem caches.
  - bbarnett 5y ago
    For the love of 3$&$>=*=$, not rust again!
- marcan_42 5y ago
  Reminds me of how, after years of massive systemwide latency spikes, I finally discovered that XFS was to blame. It was blocking on I/O writes in the reclaim path... so even random processes that wanted to allocate some RAM but don't do any disk I/O ended up blocked. This was on a machine with tons of free RAM (the reclaim was for clean cache).
2OEH8eoCRo0 5y ago
Impossible. I've heard from everyone that ZFS saved their marriage and cures cancer!
- quickthrower2 5y ago
  Now ZFS is eating CPUs and causing a chip shortage
- yjftsjthsd-h 5y ago
  Even good software has bugs; ZFS is amazing most of the time, but that's hardly a guarantee of zero issues ever - but having bugs occasionally also doesn't make it not an amazing tool most of the time.
  - amelius 5y ago
    > but having bugs occasionally also doesn't make it not an amazing tool most of the time. For most software, yes. For a filesystem, not so true.
    - yjftsjthsd-h 5y ago
      Every major filesystem has had bugs. Also, notice that this bug never endangered data, it was just wasting CPU.
    - genewitch 5y ago
      Dan Luu would like to have a word with you about how magnificently atrocious filesystems are. Files are hard.
- seg_lol 5y ago
  If they were using ZFS, they probably wouldn't have golfed their memory allocations and had spare (don't the Go folks call this 'ballast') memory, so they wouldn't have trigger the routine? The bug was that they weren't using ZFS. Not that ZFS had a bug.
  - 5y ago
ComputerGuru 5y ago
Actual issue here: https://github.com/openzfs/zfs/issues/6531 https://github.com/openzfs/zfs/issues/6531 It seems the whole thing took place in 2017 and was fixed then (first by not calling the reclaim if used ARC memory was zero, then the root cause also fixed by using a PRNG instead of the default CSPRNG in 0.7).
- seg_lol 5y ago
  Ahhh, sweet sweet reuse [1]. If code is poetry, most code is noetry, software reuse should be renamed to software refuse. Refuse it straight to the /bin No code. https://ocw.mit.edu/courses/aeronautics-and-astronautics/16-355j-software-engineering-concepts-fall-2005/lecture-notes/cnotes6.pdf https://ocw.mit.edu/courses/aeronautics-and-astronautics/16-...
  - mlk 5y ago
    > stale bot closed this on 23 Nov 2020 as is tradition
- alberth 5y ago
  Where do you see the actual issue was fixed in that ticket?
  - ComputerGuru 5y ago
    You’re right, the hack/workaround to not reclaim ARC if it’s below a size was written but not merged [0]. But the PRNG thing (actual CPU usage bug) is real and in 0.7+ so far as I can tell. Ideally both would have been committed. [0]: https://github.com/openzfs/zfs/pull/6544 https://github.com/openzfs/zfs/pull/6544
    - arthur2e5 5y ago
      Might be a good idea to go to the PR and mention that. The stale bot has done its awful work… again.
deleted 5y ago
[deleted]
yjftsjthsd-h 5y ago
Well that was a wild ride - I personally lost it at "We aren't using ZFS." :) I'm surprised that the kernel module was even loaded, though; if they weren't using it, and the only team using ZFS was a minor experimental group, why was it even installed, let alone loaded, let alone burning CPU on a cryptographically randomized noop?
- pizza234 5y ago
  > I'm surprised that the kernel module was even loaded, though On Ubuntu, some kernel packages install also the zfs modules package. I think (but not 100% sure) that if they're on that O/S, with those packages installed, then the zfs module will be loaded on startup.
  - chasil 5y ago
    Netflix leans towards FreeBSD. Is ZFS removable there without recompiling? https://papers.freebsd.org/2019/fosdem/looney-netflix_and_freebsd/ https://papers.freebsd.org/2019/fosdem/looney-netflix_and_fr...
    - NamTaf 5y ago
      Disclaimer: despite using it at home, my BSD-fu is atrocious so I could well be wrong, and I’m not in front of a computer to try. However, I believe that ZFS is its own kernel driver and so, assuming it’s not being used then: # kldunload zfs.ko should do the trick. You would of course have to change the install setup to use UFS rather than a zroot. I suspect it’s probably loaded by default regardless of OS drive file system, though.
    - bmoyles 5y ago
      For OpenConnect, not necessarily everywhere else.
fencepost 5y ago
From the top of the article (added?) "I summarized this case study at Kernel Recipes in 2017." The bottom of the article notes that his book Systems Performance 2nd Edition is 45% off until 9/13. Edit: looks like the comments about "didn't I see this a few years ago?" have been deleted.
- brendangregg 5y ago
  Not added, but maybe I'll add something extra to make it more clear. I have a backlog of interesting case studies I'm trying to find time to share.
  - bostonsre 5y ago
    Not sure if you're comfortable sharing it, but if so, how many copies have you been able to sell? Also, any idea how something like safari online impacts those numbers? Thanks again for the book, it's an amazing read that I recommend to anyone that is interested in the area.
    - brendangregg 5y ago
      Systems Performance 1st edition sold over 10k copies for the English edition, and that was half Solaris when it was falling in market share. 2nd edition is more Linux focused, and shows how eBPF fits into the toolset, so I'd assume it'll sell more than 10k. (The publisher has been changing their author's portal and I haven't been able to login for months now, so I don't know the current numbers for 2nd Edition.) Various companies make it required or recommended reading for new engineers, which helps a lot (thanks!). How much pirated PDFs hurt these books I'm not sure, maybe a little, maybe a lot. With my BPF book, a "rough cut" was published on Safari and then that did the rounds as the PDF (and still does) even though it was unfinished and buggy. That's what annoyed me the most about it, was that people were reading a broken version and may be thinking it's the final version. Thanks for getting and recommending it!
- nicolas_t 5y ago
  Where is it 45% off? I somehow can:t find that?
elktea 5y ago
Love the look of all the tools netflix have at their disposal
- H8crilA 5y ago
  A quick to use remote profiler does wonders to an org. By fast to use I mean it takes me 10 seconds to kick off a remote profiling scrape. Someone entrepreneurial please make a purchase-able service out of that, I'll be feeling lost once I quit my corp (not Netflix). Stuff is 100x more useful than remote debugging which ultimately almost nobody uses in practice.
  - tdullien 5y ago
    Shameless plug: We just launched Prodfiler last week -- it's continuous in-production profiling based on eBPF, handles C/C++/Go/Rust/JVM/Python/Perl/Ruby (with NodeJS and .net in the pipeline), and it does not require symbols on the machine, nor recompiles with framepointer, nor special JVM flags. https://prodfiler.com https://prodfiler.com :-)
  - bostonsre 5y ago
    Datadog has continuous profiling that is pretty useful. I'm pretty sure they follow Brendan religiously.. I hope they release something like flamescope next.
- Cthulhu_ 5y ago
  And that he described "Normally I'd SSH into a machine", but you shouldn't even be able to do something like that if you've got infrastructure-as-code; the machines are ephemeral, they probably don't have any of the tools that you need to do an analysis, they shouldn't even have SSH or open ports unless there's some tool interacting with it, but in practice, they should be disposable on a whim if there is an update of sorts. Had a colleague who asked for SSH access to production machines to debug an issue. Ops team asked what he wanted to do, guy just wanted to look at which env vars were set. Ops team told him how to do his job - log the configuration - instead of give him access, because they had a mandate to ensure five or seven nines uptime. Can't risk it. I had a lot more respect for the ops people there than I had for my fellow SWE's.
raverbashing 5y ago
I usually find people running ZFS to run into way too many more problems than what other people run using Ext4. (Edit: I'm surprised to see how people run into issues with Btrfs) It might be good for some specific purposes but I'd reckon the majority of people using it do not fall into those. Using it "just because" is probably more trouble than worth. > ZFS really wasn't in use, ever! But at the same time, it was eating over 30% of CPU capacity! Whaaat?? Interesting bug, sounds like the best option is to not even have the module loaded then. (Not blaming the developers here, necessarily, but yeah, it's a bug)
- pseudalopex 5y ago
  Btrfs gave me the most trouble by far. And most people say the same in my experience. Replicating half of ZFS's features with ext4 is a whole lot of trouble too.
  - usefulcat 5y ago
    We have a machine that has a large ZFS storage pool, and one smaller btrfs volume on a pair of SSDs. If I delete a large directory hierarchy on the btrfs volume and then run sync, it locks up the entire machine for 5-10 minutes. WTF. Nothing like that ever happens with ZFS or ext4.
- azalemeth 5y ago
  I've had a lot to learn with zfs, but it's never just flat out lost my data, unlike btrfs (in about 2017)
  - teekert 5y ago
    Raid5?
    - dkh 5y ago
      Not sure if you are suggesting RAID5 as a relevant problem or solution, but it's arguably the former! :) Regardless, ZFS has configurations equivalent to most "traditional" RAID options anyway (raidz1 is their RAID5) but with the added benefits of all the other great stuff ZFS gives you, like checksums, compression, deduplication, snapshots, etc.
underscore_ku 5y ago
om nom nom
deleted 5y ago
[deleted]

332 points by mfrw 5y ago

waynesonfire 5y ago

Nice job!

jeffbee 5y ago

Memory reclaim is where all the kernel bugs are.

amelius 5y ago
Is this an argument for using Rust?
- mhh__ 5y ago
  No amount of avoiding memory safety bugs will save you from writing fundamentally incorrect code. Rust stops you making dumb mistakes, and (as a high-level language, this part isn't Unique to rust in any way) lets you create abstractions to stop other people making less dumb mistakes, but will do absolutely nothing if your model of the process is wrong. The only way to really be sure of that is extensive testing at multiple levels, and ideally some form of verification like a TLA+ model.
- marcan_42 5y ago
  No. Memory reclaim != memory leaks != memory safety. This is about filesystem caches.
- bbarnett 5y ago
  For the love of 3$&$>=*=$, not rust again!
marcan_42 5y ago
Reminds me of how, after years of massive systemwide latency spikes, I finally discovered that XFS was to blame. It was blocking on I/O writes in the reclaim path... so even random processes that wanted to allocate some RAM but don't do any disk I/O ended up blocked. This was on a machine with tons of free RAM (the reclaim was for clean cache).

2OEH8eoCRo0 5y ago

Impossible. I've heard from everyone that ZFS saved their marriage and cures cancer!

quickthrower2 5y ago
Now ZFS is eating CPUs and causing a chip shortage
yjftsjthsd-h 5y ago
Even good software has bugs; ZFS is amazing most of the time, but that's hardly a guarantee of zero issues ever - but having bugs occasionally also doesn't make it not an amazing tool most of the time.
- amelius 5y ago
  > but having bugs occasionally also doesn't make it not an amazing tool most of the time. For most software, yes. For a filesystem, not so true.
  - yjftsjthsd-h 5y ago
    Every major filesystem has had bugs. Also, notice that this bug never endangered data, it was just wasting CPU.
  - genewitch 5y ago
    Dan Luu would like to have a word with you about how magnificently atrocious filesystems are. Files are hard.
seg_lol 5y ago
If they were using ZFS, they probably wouldn't have golfed their memory allocations and had spare (don't the Go folks call this 'ballast') memory, so they wouldn't have trigger the routine? The bug was that they weren't using ZFS. Not that ZFS had a bug.
- 5y ago

ComputerGuru 5y ago

Actual issue here: https://github.com/openzfs/zfs/issues/6531 https://github.com/openzfs/zfs/issues/6531 It seems the whole thing took place in 2017 and was fixed then (first by not calling the reclaim if used ARC memory was zero, then the root cause also fixed by using a PRNG instead of the default CSPRNG in 0.7).

seg_lol 5y ago
Ahhh, sweet sweet reuse [1]. If code is poetry, most code is noetry, software reuse should be renamed to software refuse. Refuse it straight to the /bin No code. https://ocw.mit.edu/courses/aeronautics-and-astronautics/16-355j-software-engineering-concepts-fall-2005/lecture-notes/cnotes6.pdf https://ocw.mit.edu/courses/aeronautics-and-astronautics/16-...
- mlk 5y ago
  > stale bot closed this on 23 Nov 2020 as is tradition
alberth 5y ago
Where do you see the actual issue was fixed in that ticket?
- ComputerGuru 5y ago
  You’re right, the hack/workaround to not reclaim ARC if it’s below a size was written but not merged [0]. But the PRNG thing (actual CPU usage bug) is real and in 0.7+ so far as I can tell. Ideally both would have been committed. [0]: https://github.com/openzfs/zfs/pull/6544 https://github.com/openzfs/zfs/pull/6544
  - arthur2e5 5y ago
    Might be a good idea to go to the PR and mention that. The stale bot has done its awful work… again.

deleted 5y ago

[deleted]

yjftsjthsd-h 5y ago

Well that was a wild ride - I personally lost it at "We aren't using ZFS." :) I'm surprised that the kernel module was even loaded, though; if they weren't using it, and the only team using ZFS was a minor experimental group, why was it even installed, let alone loaded, let alone burning CPU on a cryptographically randomized noop?

pizza234 5y ago
> I'm surprised that the kernel module was even loaded, though On Ubuntu, some kernel packages install also the zfs modules package. I think (but not 100% sure) that if they're on that O/S, with those packages installed, then the zfs module will be loaded on startup.
- chasil 5y ago
  Netflix leans towards FreeBSD. Is ZFS removable there without recompiling? https://papers.freebsd.org/2019/fosdem/looney-netflix_and_freebsd/ https://papers.freebsd.org/2019/fosdem/looney-netflix_and_fr...
  - NamTaf 5y ago
    Disclaimer: despite using it at home, my BSD-fu is atrocious so I could well be wrong, and I’m not in front of a computer to try. However, I believe that ZFS is its own kernel driver and so, assuming it’s not being used then: # kldunload zfs.ko should do the trick. You would of course have to change the install setup to use UFS rather than a zroot. I suspect it’s probably loaded by default regardless of OS drive file system, though.
  - bmoyles 5y ago
    For OpenConnect, not necessarily everywhere else.

fencepost 5y ago

From the top of the article (added?) "I summarized this case study at Kernel Recipes in 2017." The bottom of the article notes that his book Systems Performance 2nd Edition is 45% off until 9/13. Edit: looks like the comments about "didn't I see this a few years ago?" have been deleted.

brendangregg 5y ago
Not added, but maybe I'll add something extra to make it more clear. I have a backlog of interesting case studies I'm trying to find time to share.
- bostonsre 5y ago
  Not sure if you're comfortable sharing it, but if so, how many copies have you been able to sell? Also, any idea how something like safari online impacts those numbers? Thanks again for the book, it's an amazing read that I recommend to anyone that is interested in the area.
  - brendangregg 5y ago
    Systems Performance 1st edition sold over 10k copies for the English edition, and that was half Solaris when it was falling in market share. 2nd edition is more Linux focused, and shows how eBPF fits into the toolset, so I'd assume it'll sell more than 10k. (The publisher has been changing their author's portal and I haven't been able to login for months now, so I don't know the current numbers for 2nd Edition.) Various companies make it required or recommended reading for new engineers, which helps a lot (thanks!). How much pirated PDFs hurt these books I'm not sure, maybe a little, maybe a lot. With my BPF book, a "rough cut" was published on Safari and then that did the rounds as the PDF (and still does) even though it was unfinished and buggy. That's what annoyed me the most about it, was that people were reading a broken version and may be thinking it's the final version. Thanks for getting and recommending it!
nicolas_t 5y ago
Where is it 45% off? I somehow can:t find that?

elktea 5y ago

Love the look of all the tools netflix have at their disposal

H8crilA 5y ago
A quick to use remote profiler does wonders to an org. By fast to use I mean it takes me 10 seconds to kick off a remote profiling scrape. Someone entrepreneurial please make a purchase-able service out of that, I'll be feeling lost once I quit my corp (not Netflix). Stuff is 100x more useful than remote debugging which ultimately almost nobody uses in practice.
- tdullien 5y ago
  Shameless plug: We just launched Prodfiler last week -- it's continuous in-production profiling based on eBPF, handles C/C++/Go/Rust/JVM/Python/Perl/Ruby (with NodeJS and .net in the pipeline), and it does not require symbols on the machine, nor recompiles with framepointer, nor special JVM flags. https://prodfiler.com https://prodfiler.com :-)
- bostonsre 5y ago
  Datadog has continuous profiling that is pretty useful. I'm pretty sure they follow Brendan religiously.. I hope they release something like flamescope next.
Cthulhu_ 5y ago
And that he described "Normally I'd SSH into a machine", but you shouldn't even be able to do something like that if you've got infrastructure-as-code; the machines are ephemeral, they probably don't have any of the tools that you need to do an analysis, they shouldn't even have SSH or open ports unless there's some tool interacting with it, but in practice, they should be disposable on a whim if there is an update of sorts. Had a colleague who asked for SSH access to production machines to debug an issue. Ops team asked what he wanted to do, guy just wanted to look at which env vars were set. Ops team told him how to do his job - log the configuration - instead of give him access, because they had a mandate to ensure five or seven nines uptime. Can't risk it. I had a lot more respect for the ops people there than I had for my fellow SWE's.

raverbashing 5y ago

I usually find people running ZFS to run into way too many more problems than what other people run using Ext4. (Edit: I'm surprised to see how people run into issues with Btrfs) It might be good for some specific purposes but I'd reckon the majority of people using it do not fall into those. Using it "just because" is probably more trouble than worth. > ZFS really wasn't in use, ever! But at the same time, it was eating over 30% of CPU capacity! Whaaat?? Interesting bug, sounds like the best option is to not even have the module loaded then. (Not blaming the developers here, necessarily, but yeah, it's a bug)

pseudalopex 5y ago
Btrfs gave me the most trouble by far. And most people say the same in my experience. Replicating half of ZFS's features with ext4 is a whole lot of trouble too.
- usefulcat 5y ago
  We have a machine that has a large ZFS storage pool, and one smaller btrfs volume on a pair of SSDs. If I delete a large directory hierarchy on the btrfs volume and then run sync, it locks up the entire machine for 5-10 minutes. WTF. Nothing like that ever happens with ZFS or ext4.
azalemeth 5y ago
I've had a lot to learn with zfs, but it's never just flat out lost my data, unlike btrfs (in about 2017)
- teekert 5y ago
  Raid5?
  - dkh 5y ago
    Not sure if you are suggesting RAID5 as a relevant problem or solution, but it's arguably the former! :) Regardless, ZFS has configurations equivalent to most "traditional" RAID options anyway (raidz1 is their RAID5) but with the added benefits of all the other great stuff ZFS gives you, like checksums, compression, deduplication, snapshots, etc.

underscore_ku 5y ago

om nom nom

deleted 5y ago

[deleted]