7 ms·

Where Linux's load average comes from in the kernel

172 points by zdw 5y ago

tanelpoder 5y ago
When I was looking into this, I found that just running ps (with -L option to see all threads, not just processes/thread group leaders) with some grep/sort/uniq was the easiest way to break down where does the “too high Linux system load” come from. No need to compile C code or have root access to drill down into load. And you could drill down further by sampling some additional /proc/PID/task/TID/ files, like “syscall”, “stack” to see which (blocked) syscall is contributing to the load and where in the kernel is it stuck. Knowing what kind of process/thread level /proc files are available and reading/sampling them with a shell one-liner is a powerful entry point for performance troubleshooting and may allow you to delay writing advanced kernel tracing scripts further. For those that are interested: https://tanelpoder.com/posts/high-system-load-low-cpu-utilization-on-linux/ https://tanelpoder.com/posts/high-system-load-low-cpu-utiliz...
- anonymousDan 5y ago
  Interesting link thanks. The part I always find tricky with kernel debugging is distinguishing normal vs abnormal behaviour (e.g. how many kworker threads is too many)?
  - tanelpoder 5y ago
    Yep, one thing that helps in this context is to only look into threads in R or D state (and ignore the hundreds or sometimes thousands of kernel threads that are just sleeping or waiting for more work to do in idle (I) state). But yeah, even that doesn't clearly tell you what amount of active threads is normal for your workload/configuration. Or you can just save the thread count info (grouped by thread state) and save/graph it over time, to have a better idea of what's normal. The difference of this approach vs just graphing Linux-reported load is that you'll get a breakdown of how much of it was CPU demand vs sync I/O demand. Or you could even break the the thread states down further, by syscall and WCHAN, for example. This will give you a much more detailed idea of why the load is high, not just that it is high. (Like I mentioned in the article), I even have a tool for sampling /proc at regular intervals (1 Hz by default) and saving it into hourly CSV files, so you can easily go "back in time" and zoom into short spikes and see what was going on: $ xcapture 0xTools xcapture v1.0 by Tanel Poder [https://0x.tools] Sampling /proc... DATE TIME PID TID USERNAME ST COMMAND SYSCALL WCHAN 2020-10-17 12:01:50.583 6404 7524 mysql R (mysqld) fsync wait_on_page_bit 2020-10-17 12:01:50.583 6404 8944 mysql D (mysqld) fsync wait_on_page_bit 2020-10-17 12:01:50.583 6404 8946 mysql D (mysqld) fsync wait_on_page_bit 2020-10-17 12:01:50.583 6404 76046 mysql D (mysqld) fsync wait_on_page_bit 2020-10-17 12:01:50.583 6404 76811 mysql D (mysqld) fdatasync xfs_log_force_lsn 2020-10-17 12:01:50.583 6404 76815 mysql D (mysqld) fsync blkdev_issue_flush 2020-10-17 12:01:50.583 8803 8803 root R (md10_resync) [running] 0 DATE TIME PID TID USERNAME ST COMMAND SYSCALL WCHAN 2020-10-17 12:01:51.623 6404 7521 mysql D (mysqld) pwrite64 xfs_file_buffered_aio_write 2020-10-17 12:01:51.623 6404 7524 mysql D (mysqld) fsync xfs_log_force_lsn ...
bragr 5y ago
One of my old Linux sys admin interview questions was describing a situation with a high load number but low CPU utilization. People who said that was impossible or similar got shown the door and people who knew or could reason their way to IO wait/problem would get to proceed.
- tanelpoder 5y ago
  There's also one interesting addition that people may not be aware of: Synchronous I/O that blocks (like pread64/pwrite64) will contribute to Linux system load (threads in D state). Asynchronous I/O completion checks (libaio's io_getevents) that are willing to wait for I/O completion, will not contribute to Linux system load (threads in S state). Asynchronous I/O submissions (libaio's io_submit) either quickly submit their I/O (a small amount of time in R state) OR get stuck in io_submit() if the underlying block device I/O queue is full. When io_submit() gets stuck, then you're sleeping in D mode, thus contributing to system load again.
- walrus01 5y ago
  one possible 'correct' answer for something like this would be iowait caused by slow disk performance on a degraded raid array that was still operational. as one of many possible theoretical scenarios.
- shon 5y ago
  We used to ask the same question. TBH I haven’t read the f’ing article yet but… I’d summarize this as load=Run-que length. Number of run-eligible processes blocked by something where something is usually IO in a low CPU utilization scenario.
- EdSchouten 5y ago
  The fact that the load average also counts threads stuck on I/O happens to be Linux specific. On the BSDs it only measures CPU utilization. So it may be the case that you showed people the door, simply because their experience was based on non-Linux operating systems.
  - bragr 5y ago
deleted 5y ago
[deleted]
belter 5y ago
From the code comments... /* * kernel/sched/loadavg.c * This file contains the magic bits required to compute the global loadavg * figure. Its a silly number but people think its important. We go through * great pains to make it work on big machines and tickless kernels. */
- R0b0t1 5y ago
  The comment really undersells it. The fact that it is a regular number, despite not being strongly tied to something real, does make it useful. Even when you have a lot of IO wait causing an extremely large number you are being told something useful.
waynesonfire 5y ago
If I understand the authors use-case to get more refined top results, from the first paragraph, "Suppose, not hypothetically, that you have a machine that periodically has its load average briefly soar to relatively absurd levels for no obvious reason; the machine is normally at, say, 0.5 load average but briefly spikes to 10 or 15 a number of times a day. " $ sudo perf sched record -- sleep 5 && sudo perf sched latency seems to do the trick. I'm not even a perf engineer just did a simple google search. Though, that makes for pretty crummy blog content.
- tanelpoder 5y ago
  Perf sched shows you scheduling latency (time spent waiting on CPU runqueue in Runnable state), but not "demand" for (some of) the system resources like the load figure tries to estimate. I think the author didn't realize that the unit of load is just "number of threads" doing _something_ (wanting to be on CPU, running on CPU or waiting for I/O in D state on Linux, just CPU stuff on other Unixes). Load average is just "average number of threads" doing something over last 1,5,15 minutes. So if just the single-number averaged over multiple minutes is not good enough for drilling down into your load spikes, then you just go look into the data source (not necessarily source code) yourself. Just use ps or /proc filesystem to list the _number of threads_ that are currently either in R or D state. That's your system load at the current moment. If you want some summary/average over 10 seconds, run the same commant 100x in a row (and sleep a bit in the between) and count all threads in R & D state (and then divide by 100 to normalize it to an average). It's basically sampling-based profiling of Linux thread states.
wbh1 5y ago
Similar article from Brendan Gregg a few years back: https://www.brendangregg.com/blog/2017-08-08/linux-load-averages.html https://www.brendangregg.com/blog/2017-08-08/linux-load-aver...
tie_ 5y ago
Install atop and configure it to sample every second. I can't count the number of times this has helped solve a mysterious behavior. Atop is king.
- winrid 5y ago
  Be careful running atop with the -R flag on machines that create lots of short lived threads or do lots of IO, like many databases.
- geocrasher 5y ago
  Yes, atop is great for this! There's also "sar" (system activity report) which can do similar things. Both are quite helpful.
- jeffbee 5y ago
  It's great but it's very expensive so be aware that if you are down to the level of caring about things in the single-digit percentages of your budget, something like atop once per second could easily consume that amount of resources. A lot of these stats are really not cheap at all. E.g. you may find a substantial difference in the amount of CPU time needed to produce /proc/pid/stat as opposed to /proc/pid/statm. And some of the system-wide stats are even worse.
nousermane 5y ago
Pretty sure that running "top -b >/var/log/whatever" in background would catch the culprit(s) of originally reported "spikes".
the8472 5y ago
There is a newer alternative, pressure stall information[0], split by CPU, memory and IO pressure. htop can display the rolling PSI averages and one can also see the total time during which some tasks were stalled. They're also available per cgroup. [0] https://lwn.net/Articles/759781/ https://lwn.net/Articles/759781/
- mardifoufs 5y ago
  Now that makes me wonder how does Windows deal with this? Say when you open the task manager, what does the cpu usage % actually mean? I'd guess it's not measuring it in the same way Linux does, but I'd be surprised if it used pressure stall information?
  - SuperQue 5y ago
    Windows users just look at CPU utilization, which is what most Linux users should be doing as well. Monitoring both total system and single core utilization. Of course, 100% CPU use also isn't a "problem", it's just a fact. PSI is the thing that actually reveals problems. I'm not aware of a Windows equivalent.
    - mardifoufs 5y ago
      Okay so my question might not make a lot of sense but how does an OS or I'd guess a kernel knows what's the CPU utilization? I'm pretty sure it's reported by the CPU, but how does the CPU measure that? From my understanding, CPUs have multiple "units" that can specialize in specific operations. Is that taken into account, does it just measure the utilization of it's "general use" units, and if so how does the CPU keep track of it's own usage? I'd google it but I honestly don't know what to look for past the OS/kernel layer
      - johntb86 5y ago
        The OS is in charge of telling the CPU what to run. Literally the context switching code sets saves its old registers and reads in the registers of the program that's about to run (including the program counter that says what address to start at). So at that time it can save information about how the current time and long the previous code ran. Low-level hardware block usage generally isn't accounted for that way, but CPUs can keep track of performance counter events tracking how many of each operation are performed and how many cycles each subunit is active. The OS can periodically read that data, but it's only used for low level metrics about how efficient the code is rather than for calculating overall CPU utilization. There are separate measurements about GPU and hard disk utilization, which are either determined by when driver queues work on the device and receives a notification that the device is done, or by the device keeping track of what work it's switching between and reporting it periodically.

172 points by zdw 5y ago

When I was looking into this, I found that just running ps (with -L option to see all threads, not just processes/thread group leaders) with some grep/sort/uniq was the easiest way to break down where does the “too high Linux system load” come from. No need to compile C code or have root access to drill down into load. And you could drill down further by sampling some additional /proc/PID/task/TID/ files, like “syscall”, “stack” to see which (blocked) syscall is contributing to the load and where in the kernel is it stuck. Knowing what kind of process/thread level /proc files are available and reading/sampling them with a shell one-liner is a powerful entry point for performance troubleshooting and may allow you to delay writing advanced kernel tracing scripts further. For those that are interested: https://tanelpoder.com/posts/high-system-load-low-cpu-utilization-on-linux/ https://tanelpoder.com/posts/high-system-load-low-cpu-utiliz...

anonymousDan 5y ago
Interesting link thanks. The part I always find tricky with kernel debugging is distinguishing normal vs abnormal behaviour (e.g. how many kworker threads is too many)?
- tanelpoder 5y ago
  Yep, one thing that helps in this context is to only look into threads in R or D state (and ignore the hundreds or sometimes thousands of kernel threads that are just sleeping or waiting for more work to do in idle (I) state). But yeah, even that doesn't clearly tell you what amount of active threads is normal for your workload/configuration. Or you can just save the thread count info (grouped by thread state) and save/graph it over time, to have a better idea of what's normal. The difference of this approach vs just graphing Linux-reported load is that you'll get a breakdown of how much of it was CPU demand vs sync I/O demand. Or you could even break the the thread states down further, by syscall and WCHAN, for example. This will give you a much more detailed idea of why the load is high, not just that it is high. (Like I mentioned in the article), I even have a tool for sampling /proc at regular intervals (1 Hz by default) and saving it into hourly CSV files, so you can easily go "back in time" and zoom into short spikes and see what was going on: $ xcapture 0xTools xcapture v1.0 by Tanel Poder [https://0x.tools] Sampling /proc... DATE TIME PID TID USERNAME ST COMMAND SYSCALL WCHAN 2020-10-17 12:01:50.583 6404 7524 mysql R (mysqld) fsync wait_on_page_bit 2020-10-17 12:01:50.583 6404 8944 mysql D (mysqld) fsync wait_on_page_bit 2020-10-17 12:01:50.583 6404 8946 mysql D (mysqld) fsync wait_on_page_bit 2020-10-17 12:01:50.583 6404 76046 mysql D (mysqld) fsync wait_on_page_bit 2020-10-17 12:01:50.583 6404 76811 mysql D (mysqld) fdatasync xfs_log_force_lsn 2020-10-17 12:01:50.583 6404 76815 mysql D (mysqld) fsync blkdev_issue_flush 2020-10-17 12:01:50.583 8803 8803 root R (md10_resync) [running] 0 DATE TIME PID TID USERNAME ST COMMAND SYSCALL WCHAN 2020-10-17 12:01:51.623 6404 7521 mysql D (mysqld) pwrite64 xfs_file_buffered_aio_write 2020-10-17 12:01:51.623 6404 7524 mysql D (mysqld) fsync xfs_log_force_lsn ...

bragr 5y ago

One of my old Linux sys admin interview questions was describing a situation with a high load number but low CPU utilization. People who said that was impossible or similar got shown the door and people who knew or could reason their way to IO wait/problem would get to proceed.

tanelpoder 5y ago
There's also one interesting addition that people may not be aware of: Synchronous I/O that blocks (like pread64/pwrite64) will contribute to Linux system load (threads in D state). Asynchronous I/O completion checks (libaio's io_getevents) that are willing to wait for I/O completion, will not contribute to Linux system load (threads in S state). Asynchronous I/O submissions (libaio's io_submit) either quickly submit their I/O (a small amount of time in R state) OR get stuck in io_submit() if the underlying block device I/O queue is full. When io_submit() gets stuck, then you're sleeping in D mode, thus contributing to system load again.
walrus01 5y ago
one possible 'correct' answer for something like this would be iowait caused by slow disk performance on a degraded raid array that was still operational. as one of many possible theoretical scenarios.
shon 5y ago
We used to ask the same question. TBH I haven’t read the f’ing article yet but… I’d summarize this as load=Run-que length. Number of run-eligible processes blocked by something where something is usually IO in a low CPU utilization scenario.
EdSchouten 5y ago
The fact that the load average also counts threads stuck on I/O happens to be Linux specific. On the BSDs it only measures CPU utilization. So it may be the case that you showed people the door, simply because their experience was based on non-Linux operating systems.
- bragr 5y ago

deleted 5y ago

[deleted]

belter 5y ago

From the code comments... /* * kernel/sched/loadavg.c * This file contains the magic bits required to compute the global loadavg * figure. Its a silly number but people think its important. We go through * great pains to make it work on big machines and tickless kernels. */

R0b0t1 5y ago
The comment really undersells it. The fact that it is a regular number, despite not being strongly tied to something real, does make it useful. Even when you have a lot of IO wait causing an extremely large number you are being told something useful.

waynesonfire 5y ago

If I understand the authors use-case to get more refined top results, from the first paragraph, "Suppose, not hypothetically, that you have a machine that periodically has its load average briefly soar to relatively absurd levels for no obvious reason; the machine is normally at, say, 0.5 load average but briefly spikes to 10 or 15 a number of times a day. " $ sudo perf sched record -- sleep 5 && sudo perf sched latency seems to do the trick. I'm not even a perf engineer just did a simple google search. Though, that makes for pretty crummy blog content.

tanelpoder 5y ago
Perf sched shows you scheduling latency (time spent waiting on CPU runqueue in Runnable state), but not "demand" for (some of) the system resources like the load figure tries to estimate. I think the author didn't realize that the unit of load is just "number of threads" doing _something_ (wanting to be on CPU, running on CPU or waiting for I/O in D state on Linux, just CPU stuff on other Unixes). Load average is just "average number of threads" doing something over last 1,5,15 minutes. So if just the single-number averaged over multiple minutes is not good enough for drilling down into your load spikes, then you just go look into the data source (not necessarily source code) yourself. Just use ps or /proc filesystem to list the _number of threads_ that are currently either in R or D state. That's your system load at the current moment. If you want some summary/average over 10 seconds, run the same commant 100x in a row (and sleep a bit in the between) and count all threads in R & D state (and then divide by 100 to normalize it to an average). It's basically sampling-based profiling of Linux thread states.

wbh1 5y ago

Similar article from Brendan Gregg a few years back: https://www.brendangregg.com/blog/2017-08-08/linux-load-averages.html https://www.brendangregg.com/blog/2017-08-08/linux-load-aver...

tie_ 5y ago

Install atop and configure it to sample every second. I can't count the number of times this has helped solve a mysterious behavior. Atop is king.

winrid 5y ago
Be careful running atop with the -R flag on machines that create lots of short lived threads or do lots of IO, like many databases.
geocrasher 5y ago
Yes, atop is great for this! There's also "sar" (system activity report) which can do similar things. Both are quite helpful.
jeffbee 5y ago
It's great but it's very expensive so be aware that if you are down to the level of caring about things in the single-digit percentages of your budget, something like atop once per second could easily consume that amount of resources. A lot of these stats are really not cheap at all. E.g. you may find a substantial difference in the amount of CPU time needed to produce /proc/pid/stat as opposed to /proc/pid/statm. And some of the system-wide stats are even worse.

nousermane 5y ago

Pretty sure that running "top -b >/var/log/whatever" in background would catch the culprit(s) of originally reported "spikes".

the8472 5y ago

There is a newer alternative, pressure stall information[0], split by CPU, memory and IO pressure. htop can display the rolling PSI averages and one can also see the total time during which some tasks were stalled. They're also available per cgroup. [0] https://lwn.net/Articles/759781/ https://lwn.net/Articles/759781/

mardifoufs 5y ago
Now that makes me wonder how does Windows deal with this? Say when you open the task manager, what does the cpu usage % actually mean? I'd guess it's not measuring it in the same way Linux does, but I'd be surprised if it used pressure stall information?
- SuperQue 5y ago
  Windows users just look at CPU utilization, which is what most Linux users should be doing as well. Monitoring both total system and single core utilization. Of course, 100% CPU use also isn't a "problem", it's just a fact. PSI is the thing that actually reveals problems. I'm not aware of a Windows equivalent.
  - mardifoufs 5y ago
    Okay so my question might not make a lot of sense but how does an OS or I'd guess a kernel knows what's the CPU utilization? I'm pretty sure it's reported by the CPU, but how does the CPU measure that? From my understanding, CPUs have multiple "units" that can specialize in specific operations. Is that taken into account, does it just measure the utilization of it's "general use" units, and if so how does the CPU keep track of it's own usage? I'd google it but I honestly don't know what to look for past the OS/kernel layer
    - johntb86 5y ago
      The OS is in charge of telling the CPU what to run. Literally the context switching code sets saves its old registers and reads in the registers of the program that's about to run (including the program counter that says what address to start at). So at that time it can save information about how the current time and long the previous code ran. Low-level hardware block usage generally isn't accounted for that way, but CPUs can keep track of performance counter events tracking how many of each operation are performed and how many cycles each subunit is active. The OS can periodically read that data, but it's only used for low level metrics about how efficient the code is rather than for calculating overall CPU utilization. There are separate measurements about GPU and hard disk utilization, which are either determined by when driver queues work on the device and receives a notification that the device is done, or by the device keeping track of what work it's switching between and reporting it periodically.