4 ms·

The article says that in data center workloads, 15-20% of CPU cycles are spent in the kernel. Contrast that with Brendan Gregg's statement (for Netflix running

The article says that in data center workloads, 15-20% of CPU cycles are spent in the kernel. Contrast that with Brendan Gregg's statement (for Netflix running on EC2): "Most microservices we have run hot in user-level applications (think 99% user time), not the kernel, so it's difficult to find large gains from the OS or kernel." [1] I wonder why the OP and Brendan had such different observations. [1]: https://news.ycombinator.com/item?id=13081465 https://news.ycombinator.com/item?id=13081465

dogruck 9y ago
Great question. I'd guess that Google's workload differs from Netflix. What's your guess?
AndrewDavis 9y ago
I wonder how this would compare to Netflix's CDN machines. Where Netflix moved TLS encryption into the kernel.
- brendangregg 9y ago
  CDNs are kernel heavy. :)
rtpg 9y ago
Doesn't Brendan Gregg work on things like Illumos/SmartOS? I think it's probably using lightweight OSs like that which help Meanwhile most people just kinda take out of the box Ubuntu
- brendangregg 9y ago
  I've primarily worked on Linux for 3+ years.
- buster 9y ago
  In any case, I wouldn't consider illumos much more lightweight as Linux or bsd.. It's Solaris at its heart.. (not to disrespect Solaris, it is/was a great OS)
brendangregg 9y ago
Right. Most cloud systems we have are user heavy, there's not much kernel, usually <3%. They cited a Google paper with 15-20% kernel, which is a lot. Although I've heard from other sources that Google is kernel heavy, so this further confirms that. Different workloads. It'd be interesting if someone like sysdig would share (if they could) the usr/sys breakdown for their entire customer base (I seem to recall they were sharing other such statistics). It's a bit hard to see how much an exokernel or unikernel would help given, say, a workload that was 5% kernel. You might say it would only improve performance by less than 5%. But that's just considering the cycles saved. Such kernels should also be smaller, and consume the CPU hardware caches less, leaving more room for the application to execute on those precious resources. That would improve the performance gain further than just the reduction in cycles alone.
- pjmlp 9y ago
  Security and boot times are also relevant, no? Unikernels are much easier to manage than containers.
  - brendangregg 9y ago
    Sure, both are relevant. Cutting boot time only helps so far: we also have to tune application warmup to be faster as well. This is a little dated (we've tuned it down since then), but here's a microservice with a 10 minute application warmup time: http://www.brendangregg.com/blog/2016-09-28/java-warmup.html http://www.brendangregg.com/blog/2016-09-28/java-warmup.html
    - yaantc 9y ago
      Could reduced MMU overhead be significant on a server? I work in the embedded space, so very different with small CPUs. Using an MMU compared to a simple MPU can waste 25 to 30% of a CPU cycles easily. I wouldn't expect such overhead on a big server CPU with much bigger TLB (but also bigger working sets?), still a unikernel could use a very simple mapping with in some case a single large page, and see very little MMU management overhead. I have no idea on the possible gain on a server, but I'm curious. Any hindsight welcome.

dogruck 9y ago

Great question. I'd guess that Google's workload differs from Netflix. What's your guess?

AndrewDavis 9y ago

I wonder how this would compare to Netflix's CDN machines. Where Netflix moved TLS encryption into the kernel.

brendangregg 9y ago
CDNs are kernel heavy. :)

rtpg 9y ago

Doesn't Brendan Gregg work on things like Illumos/SmartOS? I think it's probably using lightweight OSs like that which help Meanwhile most people just kinda take out of the box Ubuntu

brendangregg 9y ago
I've primarily worked on Linux for 3+ years.
buster 9y ago
In any case, I wouldn't consider illumos much more lightweight as Linux or bsd.. It's Solaris at its heart.. (not to disrespect Solaris, it is/was a great OS)

brendangregg 9y ago

Right. Most cloud systems we have are user heavy, there's not much kernel, usually <3%. They cited a Google paper with 15-20% kernel, which is a lot. Although I've heard from other sources that Google is kernel heavy, so this further confirms that. Different workloads. It'd be interesting if someone like sysdig would share (if they could) the usr/sys breakdown for their entire customer base (I seem to recall they were sharing other such statistics). It's a bit hard to see how much an exokernel or unikernel would help given, say, a workload that was 5% kernel. You might say it would only improve performance by less than 5%. But that's just considering the cycles saved. Such kernels should also be smaller, and consume the CPU hardware caches less, leaving more room for the application to execute on those precious resources. That would improve the performance gain further than just the reduction in cycles alone.

pjmlp 9y ago
Security and boot times are also relevant, no? Unikernels are much easier to manage than containers.
- brendangregg 9y ago
  Sure, both are relevant. Cutting boot time only helps so far: we also have to tune application warmup to be faster as well. This is a little dated (we've tuned it down since then), but here's a microservice with a 10 minute application warmup time: http://www.brendangregg.com/blog/2016-09-28/java-warmup.html http://www.brendangregg.com/blog/2016-09-28/java-warmup.html
  - yaantc 9y ago
    Could reduced MMU overhead be significant on a server? I work in the embedded space, so very different with small CPUs. Using an MMU compared to a simple MPU can waste 25 to 30% of a CPU cycles easily. I wouldn't expect such overhead on a big server CPU with much bigger TLB (but also bigger working sets?), still a unikernel could use a very simple mapping with in some case a single large page, and see very little MMU management overhead. I have no idea on the possible gain on a server, but I'm curious. Any hindsight welcome.