4 ms·

One of my old Linux sys admin interview questions was describing a situation with a high load number but low CPU utilization. People who said that was impossibl

by bragr 5y ago

One of my old Linux sys admin interview questions was describing a situation with a high load number but low CPU utilization. People who said that was impossible or similar got shown the door and people who knew or could reason their way to IO wait/problem would get to proceed.

tanelpoder 5y ago
There's also one interesting addition that people may not be aware of: Synchronous I/O that blocks (like pread64/pwrite64) will contribute to Linux system load (threads in D state). Asynchronous I/O completion checks (libaio's io_getevents) that are willing to wait for I/O completion, will not contribute to Linux system load (threads in S state). Asynchronous I/O submissions (libaio's io_submit) either quickly submit their I/O (a small amount of time in R state) OR get stuck in io_submit() if the underlying block device I/O queue is full. When io_submit() gets stuck, then you're sleeping in D mode, thus contributing to system load again.
walrus01 5y ago
one possible 'correct' answer for something like this would be iowait caused by slow disk performance on a degraded raid array that was still operational. as one of many possible theoretical scenarios.
shon 5y ago
We used to ask the same question. TBH I haven’t read the f’ing article yet but… I’d summarize this as load=Run-que length. Number of run-eligible processes blocked by something where something is usually IO in a low CPU utilization scenario.
EdSchouten 5y ago
The fact that the load average also counts threads stuck on I/O happens to be Linux specific. On the BSDs it only measures CPU utilization. So it may be the case that you showed people the door, simply because their experience was based on non-Linux operating systems.
- bragr 5y ago
  One does typically prefer Linux experience for a _Linux_ sys admin, but don't get me wrong that's just one of many screening question. That said, if someone doesn't have enough Linux experience to know how Linux differs from *BSD or Solaris or whatever they have time on, that seems like a valid exclusion to me. I don't tend to put any stock of claims of being able to be a quick study unless they've obviously done some interview prep on the subjects they're unfamiliar with. Best way to show you are a motivated, quick study is by being a motivated, quick study.
  - toast0 5y ago
    > Best way to show you are a motivated, quick study is by being a motivated, quick study. It's hard to get generalized knowledge being a quick study, it's a lot easier to get specific knowledge to work on an actual issue being a motivated, quick study. So if that's really what you're looking for, you've got to be OK with using a search engine and manual pages during the interview.
  - inferiorhuman 5y ago
    For context: most of my professional experience is with Linux, recreational with FreeBSD. If I'd been shown the door for failing a litmus test I'd probably be more relieved than anything. Your flippant response validates that. When I've been on the employer side of the interview table my priorities are: does this person have a rudimentary knowledge of the required subjects and what is their problem solving process like. Basically do they know enough to handle the majority of day to day things and know where to look and who to ask when something is beyond their knowledge. Arcane trivia is of little-to-no interest to me because nobody is going to memorize every single detail. There will always be surprises. And, yes, implementation details of the load average is arcane trivia – that's precisely why this article is/was on the front page of HN. With the scenario you've put forth it's absolutely possible to hit the ground running without having to know the gory details of how the Linux kernel calculates load average. Surely you're already monitoring I/O activity and CPU usage alongside load average. And if you're not (tsk tsk) a competent candidate would know where to look for current system information, and how to respond to the information (e.g. is this elevated load average worth being concerned about in the first place?). > Best way to show you are a motivated, quick study is by being a motivated, quick study. The best advice I can give is to look for successes and not failures in your candidates. Asking something tantamount to a trick question. Asking them to work through a scenario where the load average is high and the CPU utilization is low is looking for success.
bigcat123 5y ago
belter 5y ago
"The many load averages of Unix(es)" https://utcc.utoronto.ca/~cks/space/blog/unix/ManyLoadAveragesOfUnix https://utcc.utoronto.ca/~cks/space/blog/unix/ManyLoadAverag...
lamontcg 5y ago
processes stuck in a D state on stale NFS mounts.
- pram 5y ago
  Ah my old nemesis, you just had to reboot the machine to unfuck it lol
  - natmaka 5y ago
    Was 'umount -f' unable to unmount? Did you use 'soft' NFS mounting?
deleted 5y ago
[deleted]
Drybones 5y ago
Recently, I was asked an interview question about what affects Linux IO load averages with a MANGA company. When I mentioned that one of the multiple possible reasons could be IO, he acted like I said the sky was green. He did not push me through the process. He was a systems engineer too and not just some HR person reading a script.

by bragr 5y ago

tanelpoder 5y ago

There's also one interesting addition that people may not be aware of: Synchronous I/O that blocks (like pread64/pwrite64) will contribute to Linux system load (threads in D state). Asynchronous I/O completion checks (libaio's io_getevents) that are willing to wait for I/O completion, will not contribute to Linux system load (threads in S state). Asynchronous I/O submissions (libaio's io_submit) either quickly submit their I/O (a small amount of time in R state) OR get stuck in io_submit() if the underlying block device I/O queue is full. When io_submit() gets stuck, then you're sleeping in D mode, thus contributing to system load again.

walrus01 5y ago

one possible 'correct' answer for something like this would be iowait caused by slow disk performance on a degraded raid array that was still operational. as one of many possible theoretical scenarios.

shon 5y ago

We used to ask the same question. TBH I haven’t read the f’ing article yet but… I’d summarize this as load=Run-que length. Number of run-eligible processes blocked by something where something is usually IO in a low CPU utilization scenario.

EdSchouten 5y ago

The fact that the load average also counts threads stuck on I/O happens to be Linux specific. On the BSDs it only measures CPU utilization. So it may be the case that you showed people the door, simply because their experience was based on non-Linux operating systems.

bragr 5y ago
One does typically prefer Linux experience for a _Linux_ sys admin, but don't get me wrong that's just one of many screening question. That said, if someone doesn't have enough Linux experience to know how Linux differs from *BSD or Solaris or whatever they have time on, that seems like a valid exclusion to me. I don't tend to put any stock of claims of being able to be a quick study unless they've obviously done some interview prep on the subjects they're unfamiliar with. Best way to show you are a motivated, quick study is by being a motivated, quick study.
- toast0 5y ago
  > Best way to show you are a motivated, quick study is by being a motivated, quick study. It's hard to get generalized knowledge being a quick study, it's a lot easier to get specific knowledge to work on an actual issue being a motivated, quick study. So if that's really what you're looking for, you've got to be OK with using a search engine and manual pages during the interview.
- inferiorhuman 5y ago
  For context: most of my professional experience is with Linux, recreational with FreeBSD. If I'd been shown the door for failing a litmus test I'd probably be more relieved than anything. Your flippant response validates that. When I've been on the employer side of the interview table my priorities are: does this person have a rudimentary knowledge of the required subjects and what is their problem solving process like. Basically do they know enough to handle the majority of day to day things and know where to look and who to ask when something is beyond their knowledge. Arcane trivia is of little-to-no interest to me because nobody is going to memorize every single detail. There will always be surprises. And, yes, implementation details of the load average is arcane trivia – that's precisely why this article is/was on the front page of HN. With the scenario you've put forth it's absolutely possible to hit the ground running without having to know the gory details of how the Linux kernel calculates load average. Surely you're already monitoring I/O activity and CPU usage alongside load average. And if you're not (tsk tsk) a competent candidate would know where to look for current system information, and how to respond to the information (e.g. is this elevated load average worth being concerned about in the first place?). > Best way to show you are a motivated, quick study is by being a motivated, quick study. The best advice I can give is to look for successes and not failures in your candidates. Asking something tantamount to a trick question. Asking them to work through a scenario where the load average is high and the CPU utilization is low is looking for success.

bigcat123 5y ago

belter 5y ago

"The many load averages of Unix(es)" https://utcc.utoronto.ca/~cks/space/blog/unix/ManyLoadAveragesOfUnix https://utcc.utoronto.ca/~cks/space/blog/unix/ManyLoadAverag...

lamontcg 5y ago

processes stuck in a D state on stale NFS mounts.

pram 5y ago
Ah my old nemesis, you just had to reboot the machine to unfuck it lol
- natmaka 5y ago
  Was 'umount -f' unable to unmount? Did you use 'soft' NFS mounting?

deleted 5y ago

[deleted]

Drybones 5y ago

Recently, I was asked an interview question about what affects Linux IO load averages with a MANGA company. When I mentioned that one of the multiple possible reasons could be IO, he acted like I said the sky was green. He did not push me through the process. He was a systems engineer too and not just some HR person reading a script.