5 ms·

This is the second or third article I've read explaining load average, and sad to say I still can't explain it. All I know is that when it's inexplicably over

This is the second or third article I've read explaining load average, and sad to say I still can't explain it. All I know is that when it's inexplicably over 3-4, you can't determine why as (no processes are using high CPU), and 1 in ten database queries are taking to 5 minutes, under normal load, that day will not be the best of your life. Well, what I've determined is that a full storage device, or overloaded i/o system for your disk, can produce very high load along with accompanying performance of doom.

andrewvc 17y ago
I'd agree that load average just isn't intuitive. I personally just look at CPU, Memory, and IO usage separately.
lallysingh 17y ago
Are you talking about the measured load (as part of the load average) or a general state of the machine as 'heavily loaded'? The short answer is: it should be smaller than the number of cpu cores you have, if the sort of tasks you're running are actually using the cpu (vs just dispatching I/O, for example). A 24-core machine with load averages in their teens is still pretty responsive. Then again, considering that the right definition should specifically say that it's the number of processes in the 'Runnable' state. It's a measurement of the CPU as a performance bottleneck. Other bottlenecks (e.g. your disk(s)) aren't covered.
- spudlyo 17y ago
  It's a measurement of the CPU as a performance bottleneck. Other bottlenecks (e.g. your disk(s)) aren't covered. I'm afraid that's incorrect. Any process blocked on disk i/o is considered as part of the load average.
  - lallysingh 17y ago
    What? Ugh. That's depressing.
nailer 17y ago
> All I know is that when it's inexplicably over 3-4 Depends on the box. Say you: a) have an app that spawns worker threads on demand and doesn't have a limit b) Have a modern, cheap Nehalem X5600 CPU. 2 sockets * 6 cores * 2 threads per socket mean you're only fully realizing the investment when you have a load average of 24 (assuming that each of these cores is 0% idle, which you can tell with eg top). High IO would show up as high %system time vs %user.
- nailer 17y ago
  'Threads per socket' should read 'threads per core' (long day).
sabat 17y ago
To some degree, it's voodoo. When you deal with machines of a particular OS every day, you get a feel for what the numbers mean. you can't determine why as (no processes are using high CPU) Roughly, the load avg is the length of the run queue (how many processes are waiting to be run or are being run, vs. how many processor cores are available) with some IO activity thrown in for good measure. It could be a lot of IO wait or a lot of small processes that are driving up the load avg -- not necessarily just one process trying to hog CPU time. There are a lot of good suggestions here -- many sysadms are too reliant on load avg (myself somewhat included).
btilly 17y ago
That happens when you have a problem where I/O somewhere, so lots of processes wind up blocked on I/O and are counted to load even though they aren't counting for CPU. The fact that normal queries take 5 minutes is confirmation of this. As you've figured out this can happen because of a full storage device (causing writes to fragment as it squeezes data into small empty spaces, which causes what could have been one write to be many, which adds lots of seek time) or overloaded I/O causing the pre-fetch/caching tricks to stop working, causing everything to slow down. The best solution is to never get into this situation in the first place. The second best solution is to find a way to halt I/O to the afflicted device until the system gets sorted out, figure out what was wrong, then add load and get back to a running state. The worst solution is to let things limp along until diminished load hopefully lets the disk get through its backlog and go back to normal.

andrewvc 17y ago

I'd agree that load average just isn't intuitive. I personally just look at CPU, Memory, and IO usage separately.

lallysingh 17y ago

Are you talking about the measured load (as part of the load average) or a general state of the machine as 'heavily loaded'? The short answer is: it should be smaller than the number of cpu cores you have, if the sort of tasks you're running are actually using the cpu (vs just dispatching I/O, for example). A 24-core machine with load averages in their teens is still pretty responsive. Then again, considering that the right definition should specifically say that it's the number of processes in the 'Runnable' state. It's a measurement of the CPU as a performance bottleneck. Other bottlenecks (e.g. your disk(s)) aren't covered.

spudlyo 17y ago
It's a measurement of the CPU as a performance bottleneck. Other bottlenecks (e.g. your disk(s)) aren't covered. I'm afraid that's incorrect. Any process blocked on disk i/o is considered as part of the load average.
- lallysingh 17y ago
  What? Ugh. That's depressing.

nailer 17y ago

> All I know is that when it's inexplicably over 3-4 Depends on the box. Say you: a) have an app that spawns worker threads on demand and doesn't have a limit b) Have a modern, cheap Nehalem X5600 CPU. 2 sockets * 6 cores * 2 threads per socket mean you're only fully realizing the investment when you have a load average of 24 (assuming that each of these cores is 0% idle, which you can tell with eg top). High IO would show up as high %system time vs %user.

nailer 17y ago
'Threads per socket' should read 'threads per core' (long day).

sabat 17y ago

To some degree, it's voodoo. When you deal with machines of a particular OS every day, you get a feel for what the numbers mean. you can't determine why as (no processes are using high CPU) Roughly, the load avg is the length of the run queue (how many processes are waiting to be run or are being run, vs. how many processor cores are available) with some IO activity thrown in for good measure. It could be a lot of IO wait or a lot of small processes that are driving up the load avg -- not necessarily just one process trying to hog CPU time. There are a lot of good suggestions here -- many sysadms are too reliant on load avg (myself somewhat included).

btilly 17y ago

That happens when you have a problem where I/O somewhere, so lots of processes wind up blocked on I/O and are counted to load even though they aren't counting for CPU. The fact that normal queries take 5 minutes is confirmation of this. As you've figured out this can happen because of a full storage device (causing writes to fragment as it squeezes data into small empty spaces, which causes what could have been one write to be many, which adds lots of seek time) or overloaded I/O causing the pre-fetch/caching tricks to stop working, causing everything to slow down. The best solution is to never get into this situation in the first place. The second best solution is to find a way to halt I/O to the afflicted device until the system gets sorted out, figure out what was wrong, then add load and get back to a running state. The worst solution is to let things limp along until diminished load hopefully lets the disk get through its backlog and go back to normal.