5 ms·

UNIX Load Average Part 1: How It Works

49 points by helwr 17y ago

barlo 17y ago
Awesome article with great detail and explanation. I've always been curious about how load averages are calculated and what exactly they mean.
- spudlyo 17y ago
  tl;dr version: The average number of threads/processes in the run queue and/or blocked on disk i/o sampled at 1, 5, and 15 minutes.
geoffc 17y ago
My rule of thumb is that the end users of a web application will perceive that the system is "slow" if the 1 minute load average is above 4 on the application or database server.
- seiji 17y ago
  Huh? What if I have a 48 core AMD server? A fully CPU-bound load average of 48 would be great. Load average can tell you if the system is under-used, but it can't tell you how the system is over-used. I regularly have a few dual core systems spike to load averages of 50+ because of suboptimal NFS mounts. The NFS issue keeps processes waiting around with nothing to do for a while. Those processes are still counted towards the "load" even though they have no CPU activity (they are blocked on IO).
  - geoffc 17y ago
    Agreed it is not accurate for huge servers and IO bound cases but I have found it to be a good rule of thumb for a LAMP web application stack on anything from single to 8 core machines. It is a useful metric to start digging deeper into the machine.
  - spudlyo 17y ago
    The Linux kernel also checks to see if there are any tasks in a short-term sleep state called TASK_UNINTERRUPTIBLE. If there are, they are also included in the load average sample. Shown in top as being in the 'D' state.
  - FlorinAndrei 17y ago
    Always divide the load average by the number of cores. Maybe we should re-define load average, as the old definition divided by the number of cores.
pinko 17y ago
As far as I can tell, this article doesn't mention the most frequent problem people have comprehending high load averages on Linux: the inclusion of processes in uninterruptible sleep. This can result in a system with virtually no CPU load reporting a very high load average. (And, IMHO, makes the load average much less useful a metric than a pure running/run-queue-based method.)
- deleted 17y ago
  [deleted]
julio_the_squid 17y ago
This is the second or third article I've read explaining load average, and sad to say I still can't explain it. All I know is that when it's inexplicably over 3-4, you can't determine why as (no processes are using high CPU), and 1 in ten database queries are taking to 5 minutes, under normal load, that day will not be the best of your life. Well, what I've determined is that a full storage device, or overloaded i/o system for your disk, can produce very high load along with accompanying performance of doom.
- andrewvc 17y ago
  I'd agree that load average just isn't intuitive. I personally just look at CPU, Memory, and IO usage separately.
- lallysingh 17y ago
  Are you talking about the measured load (as part of the load average) or a general state of the machine as 'heavily loaded'? The short answer is: it should be smaller than the number of cpu cores you have, if the sort of tasks you're running are actually using the cpu (vs just dispatching I/O, for example). A 24-core machine with load averages in their teens is still pretty responsive. Then again, considering that the right definition should specifically say that it's the number of processes in the 'Runnable' state. It's a measurement of the CPU as a performance bottleneck. Other bottlenecks (e.g. your disk(s)) aren't covered.
  - spudlyo 17y ago
    It's a measurement of the CPU as a performance bottleneck. Other bottlenecks (e.g. your disk(s)) aren't covered. I'm afraid that's incorrect. Any process blocked on disk i/o is considered as part of the load average.
    - lallysingh 17y ago
      What? Ugh. That's depressing.
- nailer 17y ago
petercooper 17y ago
Unnecessarily rewritten headline.
- deleted 17y ago
  [deleted]
- deleted 17y ago
  [deleted]
mhd 17y ago
So, what about threads? I've seen Linux versions where they all had the same load as the base process, resulting in a huge load in a multi-threaded server application.
- spudlyo 17y ago
  In Linux, running threads that are shown in 'top' (with thread mode 'H' on) to be in state 'R' or 'D' are counted in the load average. You'll definitely see this in programs with a ton of threads, like MySQL or the Java JVM. If you're not in thread mode in top, this is why you will sometimes see a process consuming > 100% CPU usage.
Erwin 17y ago
I prefer looking at output of "sar" which shows you nicely, in 10-minute increments, how idle the system was today (and I think usually this data captured is rotated daily for a month), and gives you also a good idea of whether you have processes waiting excessively for IO. It also has a bunch of other options. On my own system, I also generally run "ps auwx" every 15 minutes, together with a scan of what queries Postgres servers are doing and dump of web request activity (read last 10k lines from access logs, find out how long ago the first request was to determine rough hits per second and ares of application they hit). That way when someone says "hey, the system was slow around this time" I can go back and find out that some cron job had a dozen processes taking up tons of memory or blocking on IO. Some of those statistics also go into some RRD-based system which makes it easier to follow e.g. number of users logged in or number of Apache children based on weekday/time of day.
- spudlyo 17y ago
  I'm a big fan of 'sar' as well. It's nice that it can also show you i/o wait for the sampled period. I also love vmstat, as you can use it to see everything that is happening with the system sampled every second if you like. The first two columns will show you the number of processes in the run queue as well as number of processes blocked on i/o.
- sophacles 17y ago
  If you don't already know about it, dstat is a good one too, and lightweight enough that it can be run on heavily loaded production systems w/out as much "top problem" going on.
- nailer 17y ago
  sar gets its info from /proc/loadavg (on Linux OSs), just like top and uptime do, which is produced using the same code the article shows.
  - Erwin 17y ago
    We must be running different versions of sar then, as "sar" by itself here (RHEL 5) shows information about the time split between user/system/waitIO/idle -- that certainly does not come from /proc/loadavg. If you run "sar -q" you could get the load average information, but that's not particularly useful, as you can't see whether the 20 load avg an hour ago was caused by heavy disk IO or a dozen CPU bound processes.
patrickgzill 17y ago
A couple notes: 1. Load averages are not directly comparable across different versions of Unix's such as FreeBSD, Linux, Solaris, etc. 2. Load average is a good way to quickly check if there is anything else to look at. However, a far better series of tools is sar plus iostat , vmstat etc. which should help you more quickly determine whether your problem is CPU, disk, or network IO .
pak 17y ago
How about an article about measuring memory usage? From what I've read, measuring the actual total memory consumption of an individual process is nontrivial on both Mac OSX and Linux, because of the way the stats are generated for things like ps, top, etc. and the way both kernels share memory between processes whenever possible.

49 points by helwr 17y ago

barlo 17y ago

Awesome article with great detail and explanation. I've always been curious about how load averages are calculated and what exactly they mean.

spudlyo 17y ago
tl;dr version: The average number of threads/processes in the run queue and/or blocked on disk i/o sampled at 1, 5, and 15 minutes.

geoffc 17y ago

My rule of thumb is that the end users of a web application will perceive that the system is "slow" if the 1 minute load average is above 4 on the application or database server.

seiji 17y ago
Huh? What if I have a 48 core AMD server? A fully CPU-bound load average of 48 would be great. Load average can tell you if the system is under-used, but it can't tell you how the system is over-used. I regularly have a few dual core systems spike to load averages of 50+ because of suboptimal NFS mounts. The NFS issue keeps processes waiting around with nothing to do for a while. Those processes are still counted towards the "load" even though they have no CPU activity (they are blocked on IO).
- geoffc 17y ago
  Agreed it is not accurate for huge servers and IO bound cases but I have found it to be a good rule of thumb for a LAMP web application stack on anything from single to 8 core machines. It is a useful metric to start digging deeper into the machine.
- spudlyo 17y ago
  The Linux kernel also checks to see if there are any tasks in a short-term sleep state called TASK_UNINTERRUPTIBLE. If there are, they are also included in the load average sample. Shown in top as being in the 'D' state.
- FlorinAndrei 17y ago
  Always divide the load average by the number of cores. Maybe we should re-define load average, as the old definition divided by the number of cores.

pinko 17y ago

As far as I can tell, this article doesn't mention the most frequent problem people have comprehending high load averages on Linux: the inclusion of processes in uninterruptible sleep. This can result in a system with virtually no CPU load reporting a very high load average. (And, IMHO, makes the load average much less useful a metric than a pure running/run-queue-based method.)

deleted 17y ago
[deleted]

julio_the_squid 17y ago

This is the second or third article I've read explaining load average, and sad to say I still can't explain it. All I know is that when it's inexplicably over 3-4, you can't determine why as (no processes are using high CPU), and 1 in ten database queries are taking to 5 minutes, under normal load, that day will not be the best of your life. Well, what I've determined is that a full storage device, or overloaded i/o system for your disk, can produce very high load along with accompanying performance of doom.

andrewvc 17y ago
I'd agree that load average just isn't intuitive. I personally just look at CPU, Memory, and IO usage separately.
lallysingh 17y ago
Are you talking about the measured load (as part of the load average) or a general state of the machine as 'heavily loaded'? The short answer is: it should be smaller than the number of cpu cores you have, if the sort of tasks you're running are actually using the cpu (vs just dispatching I/O, for example). A 24-core machine with load averages in their teens is still pretty responsive. Then again, considering that the right definition should specifically say that it's the number of processes in the 'Runnable' state. It's a measurement of the CPU as a performance bottleneck. Other bottlenecks (e.g. your disk(s)) aren't covered.
- spudlyo 17y ago
  It's a measurement of the CPU as a performance bottleneck. Other bottlenecks (e.g. your disk(s)) aren't covered. I'm afraid that's incorrect. Any process blocked on disk i/o is considered as part of the load average.
  - lallysingh 17y ago
    What? Ugh. That's depressing.
nailer 17y ago

petercooper 17y ago

Unnecessarily rewritten headline.

deleted 17y ago
[deleted]
deleted 17y ago
[deleted]

mhd 17y ago

So, what about threads? I've seen Linux versions where they all had the same load as the base process, resulting in a huge load in a multi-threaded server application.

spudlyo 17y ago
In Linux, running threads that are shown in 'top' (with thread mode 'H' on) to be in state 'R' or 'D' are counted in the load average. You'll definitely see this in programs with a ton of threads, like MySQL or the Java JVM. If you're not in thread mode in top, this is why you will sometimes see a process consuming > 100% CPU usage.

Erwin 17y ago

I prefer looking at output of "sar" which shows you nicely, in 10-minute increments, how idle the system was today (and I think usually this data captured is rotated daily for a month), and gives you also a good idea of whether you have processes waiting excessively for IO. It also has a bunch of other options. On my own system, I also generally run "ps auwx" every 15 minutes, together with a scan of what queries Postgres servers are doing and dump of web request activity (read last 10k lines from access logs, find out how long ago the first request was to determine rough hits per second and ares of application they hit). That way when someone says "hey, the system was slow around this time" I can go back and find out that some cron job had a dozen processes taking up tons of memory or blocking on IO. Some of those statistics also go into some RRD-based system which makes it easier to follow e.g. number of users logged in or number of Apache children based on weekday/time of day.

spudlyo 17y ago
I'm a big fan of 'sar' as well. It's nice that it can also show you i/o wait for the sampled period. I also love vmstat, as you can use it to see everything that is happening with the system sampled every second if you like. The first two columns will show you the number of processes in the run queue as well as number of processes blocked on i/o.
sophacles 17y ago
If you don't already know about it, dstat is a good one too, and lightweight enough that it can be run on heavily loaded production systems w/out as much "top problem" going on.
nailer 17y ago
sar gets its info from /proc/loadavg (on Linux OSs), just like top and uptime do, which is produced using the same code the article shows.
- Erwin 17y ago
  We must be running different versions of sar then, as "sar" by itself here (RHEL 5) shows information about the time split between user/system/waitIO/idle -- that certainly does not come from /proc/loadavg. If you run "sar -q" you could get the load average information, but that's not particularly useful, as you can't see whether the 20 load avg an hour ago was caused by heavy disk IO or a dozen CPU bound processes.

patrickgzill 17y ago

A couple notes: 1. Load averages are not directly comparable across different versions of Unix's such as FreeBSD, Linux, Solaris, etc. 2. Load average is a good way to quickly check if there is anything else to look at. However, a far better series of tools is sar plus iostat , vmstat etc. which should help you more quickly determine whether your problem is CPU, disk, or network IO .

pak 17y ago

How about an article about measuring memory usage? From what I've read, measuring the actual total memory consumption of an individual process is nontrivial on both Mac OSX and Linux, because of the way the stats are generated for things like ps, top, etc. and the way both kernels share memory between processes whenever possible.