8 ms·

What Does an Idle CPU Do?

563 points by brunorsini 12y ago

Correctille 12y ago
It makes coffee for the GPU.
AceJohnny2 12y ago
Gustavo Duarte's (not to be confused with Androider Matias Duarte) blog posts are awesome. I still highly recommend his 2009 in-depth review of how the Linux kernel manages memory: http://duartes.org/gustavo/blog/post/how-the-kernel-manages-your-memory/ http://duartes.org/gustavo/blog/post/how-the-kernel-manages-...
- pwelch 12y ago
  Wish I could upvote you twice. His posts are amazing.
mindslight 12y ago
Spread spectrum modulate its emissions to backhaul cached crypto keys? (/me ducks) edit: oops, the downvoters are correct; modern cpus have dedicated hardware for that.
amluto 12y ago
Tiny little correction: On x86 multi-processor machines, the CPU can be woken from idle by an interrupt or by a write to a designated address in main memory from another CPU. Linux uses* the latter technique to very efficiently wake up a process on one CPU when it receives data from another CPU. This is a big speedup on some workloads. * This didn't work very well until Linux 3.16.
DogeDogeDoge 12y ago
nop nop nop nop
leonatan 12y ago
When I was young, I never understood why there was an idle process in Windows, eating my CPU... The days without Internet.
typedweb 12y ago
I seem to recall the idle 'task' on Windows zero's empty memory pages, so there might be less of what's happening in the article on those machines.
DonGateley 12y ago
In the days of OS/360 and OS/370 (IBM's mainframe OS in the 60's and '70s) no instructions at all executed in the wait state. There was no OS level idle process. Instead, the internal CPU microcode itself ran a little loop without dispatching any machine instructions until the tentacles that reached out from that loop were tickled by an external interrupting event. That loop had to be very short for latency reasons and very prickly to test for all the conditions that could take the processor out of it to resume dispatching instructions or do critical internal things. I know this because I wrote that 6 (initially 8) micro-instruction loop for the IBM System/370 Model 155-157 in the late '60s. I was told back then that since that model became the most widely sold machine ever I had written the code most executed in the history of computing. :-) The status latch in the machine that indicated that loop was running was fed to a light on the console and, yep, it was nearly always lit other than when booting the OS.
- beachstartup 12y ago
  8 to 6... a 25% optimization. what'd you take out?
  - DonGateley 12y ago
    I got the hardware guys to merge some of the conditions that caused a loop exit into condition groups that were tested by other micro-instructions until a couple went away. Getting out quick was important, sorting out why was less so.
- wglb 12y ago
  Awesome. Were some of the smaller machines (135 or smaller) non-microcode? E.g. RTL or TTL hardwired?
  - DonGateley 12y ago
    Everything in that generation of machines was micro-coded except for the Model 195 which was then IBM's super scientific computer.
s_kanev 12y ago
Yeah, cpu idleness is pretty fascinating and, as it turns out, quite important. Shameless plug, I just published a paper [1], showing that if you mess up how deeply you go to sleep in cpuidle, you can loose up to ~15% latency in datacenter services like websearch. [1] http://www.skanev.org/papers/iiswc14ep.pdf http://www.skanev.org/papers/iiswc14ep.pdf
- amluto 12y ago
  This reminds me: if you're stuck on a Linux kernel before 3.9 and you're using Sandy Bridge hardware or newer, turn off C1E in BIOS. The "C1E" setting controls a misfeature in which the CPU will look at the OS's request for a light sleep and say "Ha! You must have asked for light sleep by accident! I'm going to go all the way to sleep instead!" This causes surprisingly severe performance problems (we've seen milliseconds of unexplained random hiccups). Linux 3.9 and higher is immune: it just forcibly disables this feature.
  - makomk 12y ago
    Hah. I guess that's a major benchmark win - decreases power usage in common cases, and few CPU benchmarks test wake-from-sleep latency either directly or indirectly.
amelius 12y ago
But what does a CPU running an idle virtual machine do?
- zheng 12y ago
  Not sure why this is being downvoted, it's actually a good question. If all of your VM's are idle, you would want your host CPU to idle as well, although my guess is this doesn't happen in things like Xen, VMWare, etc.
  - vegardx 12y ago
    There has actually been problems with that, there was a bug with Citrix XenServer [0] where the CPU would, if left on idle for a long time, step down and turn off more or less all cores, which resulted in a locked up and unresponsive host. You can imagine what that did to the virtual machine supposedly running on that host. [0] http://support.citrix.com/article/CTX127395 http://support.citrix.com/article/CTX127395
- amluto 12y ago
  Off the top of my head: the VM executes either HLT or MWAIT, and those instructions are generally programmed to cause a VM exit, so the hypervisor can take action. The hypervisor will deschedule the VM and possibly go idle itself. TBH, I have no idea how MONITOR works in a VM. It might be quite messy. Edit: Here's a really nice explanation: http://www.contrib.andrew.cmu.edu/~somlo/OSXKVM/mwait.html http://www.contrib.andrew.cmu.edu/~somlo/OSXKVM/mwait.html
  - hderms 12y ago
    Does it reinitiate VM execution by waiting for the hypervisor to be scheduled again which schedules out to its constituent vms?
- ciupicri 12y ago
  By the way, FreeDOS and MS-DOS use 100% of the CPU when running on a VirtualBox virtual machine [1]. [1]: http://www.freedos.org/wiki/index.php/VirtualBox_-_Chapter_7 http://www.freedos.org/wiki/index.php/VirtualBox_-_Chapter_7
sholanozie 12y ago
This is a really awesome article - thanks for sharing!
beat 12y ago
What does an idle CPU do? Oh, lifts weights, writes a little poetry, works on its side project startup, surfs for cat videos...
Codhisattva 12y ago
Counts sheep.
abuzzooz 12y ago
I believe that these days unused CPU cores are physically shut down by turning off the voltage supply that's feeding them, in order to save power. This is a commonly used technique in chip design called "power gating". It is one of the most important techniques for prolonging battery life in mobile devices.
bjornsing 12y ago
I was hoping for an explanation as to why process 0 needs to be "multitasking cooperatively", as in calling needs_reschedule()... Anybody?
- ww520 12y ago
  It's for avoiding to halt the CPU when there are other pending tasks ready to run. When the idle task is the only task running, it halts the CPU to save power. When the scheduling timer interrupt fires, it wakes up the CPU and calls the scheduler to schedule tasks. If there are other tasks ready to run, the idle task is swapped out to let another task to run but it is still in the ready state. Depending on the scheduling algorithm, the idle task might get picked to run again before other ready tasks. In that case, it should not halt the CPU but rather just busy-loops the rest of the quantum. At that time, the timer interrupt fires and schedules another ready task to run right the way without needing to wake up the CPU.
kakakiki 12y ago
Excellent blog! I am loving all the other posts he has written!
CmonDev 12y ago
So there is a distributed mega-super-computer being idle on this planet.
sbierwagen 12y ago
Back In The Day, before Windows implemented calling the HLT instruction in the idle loop, overclockers would run a userland program that did the same thing, called cpuidle. http://screenshots.en.sftcdn.net/en/scrn/5000/5197/cpu-idle-6.jpg http://screenshots.en.sftcdn.net/en/scrn/5000/5197/cpu-idle-... http://www.cpuidle.de/news.php http://www.cpuidle.de/news.php
- coryking 12y ago
  There was actually several of them used by overclockers. I was always a fan of "waterfall". It made a huge difference in the CPU temperature! http://estu.nit.ac.jp/~e982457/other/cpuidle/idle.htm http://estu.nit.ac.jp/~e982457/other/cpuidle/idle.htm
esmi 12y ago
"The fundamental axiom here is that at any given moment, exactly one task is active on a CPU." Maybe I missed it, and I did check the earlier post and I understand this is trying to be a simplification, but what was the argument for why I should take the above axiom? Was there one? Or I am just being too literal and all the author is really trying to say is the current task pointer must always be valid, in Linux. Which is probably true enough (I don't know Linux well) but a much less powerful statement. From the post contents I suspect the author is using OS as a synonym for Linux.

563 points by brunorsini 12y ago

It makes coffee for the GPU.

Gustavo Duarte's (not to be confused with Androider Matias Duarte) blog posts are awesome. I still highly recommend his 2009 in-depth review of how the Linux kernel manages memory: http://duartes.org/gustavo/blog/post/how-the-kernel-manages-your-memory/ http://duartes.org/gustavo/blog/post/how-the-kernel-manages-...

pwelch 12y ago
Wish I could upvote you twice. His posts are amazing.

mindslight 12y ago

Spread spectrum modulate its emissions to backhaul cached crypto keys? (/me ducks) edit: oops, the downvoters are correct; modern cpus have dedicated hardware for that.

amluto 12y ago

Tiny little correction: On x86 multi-processor machines, the CPU can be woken from idle by an interrupt or by a write to a designated address in main memory from another CPU. Linux uses* the latter technique to very efficiently wake up a process on one CPU when it receives data from another CPU. This is a big speedup on some workloads. * This didn't work very well until Linux 3.16.

DogeDogeDoge 12y ago

nop nop nop nop

leonatan 12y ago

When I was young, I never understood why there was an idle process in Windows, eating my CPU... The days without Internet.

typedweb 12y ago

I seem to recall the idle 'task' on Windows zero's empty memory pages, so there might be less of what's happening in the article on those machines.

DonGateley 12y ago

In the days of OS/360 and OS/370 (IBM's mainframe OS in the 60's and '70s) no instructions at all executed in the wait state. There was no OS level idle process. Instead, the internal CPU microcode itself ran a little loop without dispatching any machine instructions until the tentacles that reached out from that loop were tickled by an external interrupting event. That loop had to be very short for latency reasons and very prickly to test for all the conditions that could take the processor out of it to resume dispatching instructions or do critical internal things. I know this because I wrote that 6 (initially 8) micro-instruction loop for the IBM System/370 Model 155-157 in the late '60s. I was told back then that since that model became the most widely sold machine ever I had written the code most executed in the history of computing. :-) The status latch in the machine that indicated that loop was running was fed to a light on the console and, yep, it was nearly always lit other than when booting the OS.

beachstartup 12y ago
8 to 6... a 25% optimization. what'd you take out?
- DonGateley 12y ago
  I got the hardware guys to merge some of the conditions that caused a loop exit into condition groups that were tested by other micro-instructions until a couple went away. Getting out quick was important, sorting out why was less so.
wglb 12y ago
Awesome. Were some of the smaller machines (135 or smaller) non-microcode? E.g. RTL or TTL hardwired?
- DonGateley 12y ago
  Everything in that generation of machines was micro-coded except for the Model 195 which was then IBM's super scientific computer.

s_kanev 12y ago

Yeah, cpu idleness is pretty fascinating and, as it turns out, quite important. Shameless plug, I just published a paper [1], showing that if you mess up how deeply you go to sleep in cpuidle, you can loose up to ~15% latency in datacenter services like websearch. [1] http://www.skanev.org/papers/iiswc14ep.pdf http://www.skanev.org/papers/iiswc14ep.pdf

amluto 12y ago
This reminds me: if you're stuck on a Linux kernel before 3.9 and you're using Sandy Bridge hardware or newer, turn off C1E in BIOS. The "C1E" setting controls a misfeature in which the CPU will look at the OS's request for a light sleep and say "Ha! You must have asked for light sleep by accident! I'm going to go all the way to sleep instead!" This causes surprisingly severe performance problems (we've seen milliseconds of unexplained random hiccups). Linux 3.9 and higher is immune: it just forcibly disables this feature.
- makomk 12y ago
  Hah. I guess that's a major benchmark win - decreases power usage in common cases, and few CPU benchmarks test wake-from-sleep latency either directly or indirectly.

amelius 12y ago

But what does a CPU running an idle virtual machine do?

zheng 12y ago
Not sure why this is being downvoted, it's actually a good question. If all of your VM's are idle, you would want your host CPU to idle as well, although my guess is this doesn't happen in things like Xen, VMWare, etc.
- vegardx 12y ago
  There has actually been problems with that, there was a bug with Citrix XenServer [0] where the CPU would, if left on idle for a long time, step down and turn off more or less all cores, which resulted in a locked up and unresponsive host. You can imagine what that did to the virtual machine supposedly running on that host. [0] http://support.citrix.com/article/CTX127395 http://support.citrix.com/article/CTX127395
amluto 12y ago
Off the top of my head: the VM executes either HLT or MWAIT, and those instructions are generally programmed to cause a VM exit, so the hypervisor can take action. The hypervisor will deschedule the VM and possibly go idle itself. TBH, I have no idea how MONITOR works in a VM. It might be quite messy. Edit: Here's a really nice explanation: http://www.contrib.andrew.cmu.edu/~somlo/OSXKVM/mwait.html http://www.contrib.andrew.cmu.edu/~somlo/OSXKVM/mwait.html
- hderms 12y ago
  Does it reinitiate VM execution by waiting for the hypervisor to be scheduled again which schedules out to its constituent vms?
ciupicri 12y ago
By the way, FreeDOS and MS-DOS use 100% of the CPU when running on a VirtualBox virtual machine [1]. [1]: http://www.freedos.org/wiki/index.php/VirtualBox_-_Chapter_7 http://www.freedos.org/wiki/index.php/VirtualBox_-_Chapter_7

sholanozie 12y ago

This is a really awesome article - thanks for sharing!

beat 12y ago

What does an idle CPU do? Oh, lifts weights, writes a little poetry, works on its side project startup, surfs for cat videos...

Codhisattva 12y ago

Counts sheep.

abuzzooz 12y ago

I believe that these days unused CPU cores are physically shut down by turning off the voltage supply that's feeding them, in order to save power. This is a commonly used technique in chip design called "power gating". It is one of the most important techniques for prolonging battery life in mobile devices.

bjornsing 12y ago

I was hoping for an explanation as to why process 0 needs to be "multitasking cooperatively", as in calling needs_reschedule()... Anybody?

ww520 12y ago
It's for avoiding to halt the CPU when there are other pending tasks ready to run. When the idle task is the only task running, it halts the CPU to save power. When the scheduling timer interrupt fires, it wakes up the CPU and calls the scheduler to schedule tasks. If there are other tasks ready to run, the idle task is swapped out to let another task to run but it is still in the ready state. Depending on the scheduling algorithm, the idle task might get picked to run again before other ready tasks. In that case, it should not halt the CPU but rather just busy-loops the rest of the quantum. At that time, the timer interrupt fires and schedules another ready task to run right the way without needing to wake up the CPU.

kakakiki 12y ago

Excellent blog! I am loving all the other posts he has written!

CmonDev 12y ago

So there is a distributed mega-super-computer being idle on this planet.

sbierwagen 12y ago

Back In The Day, before Windows implemented calling the HLT instruction in the idle loop, overclockers would run a userland program that did the same thing, called cpuidle. http://screenshots.en.sftcdn.net/en/scrn/5000/5197/cpu-idle-6.jpg http://screenshots.en.sftcdn.net/en/scrn/5000/5197/cpu-idle-... http://www.cpuidle.de/news.php http://www.cpuidle.de/news.php

coryking 12y ago
There was actually several of them used by overclockers. I was always a fan of "waterfall". It made a huge difference in the CPU temperature! http://estu.nit.ac.jp/~e982457/other/cpuidle/idle.htm http://estu.nit.ac.jp/~e982457/other/cpuidle/idle.htm

esmi 12y ago

"The fundamental axiom here is that at any given moment, exactly one task is active on a CPU." Maybe I missed it, and I did check the earlier post and I understand this is trying to be a simplification, but what was the argument for why I should take the above axiom? Was there one? Or I am just being too literal and all the author is really trying to say is the current task pointer must always be valid, in Linux. Which is probably true enough (I don't know Linux well) but a much less powerful statement. From the post contents I suspect the author is using OS as a synonym for Linux.