5 ms·
I too instinctively run `w` whenever I log into a machine, and that instinct helped me land my current job. One hour long component of a (now deprecated) SRE i
by spudlyo 9y ago
I too instinctively run `w` whenever I log into a machine, and that instinct helped me land my current job.
One hour long component of a (now deprecated) SRE interview loop was for the candidate to SSH into a series of EC2 instances and debug issues which got progressively harder as the interview wore on.
I had wasted a substantial amount of time on the first and easiest problem by really overthinking it, and not trying the simplest of debugging techniques first. By the time I got to the final, hardest problem, I had just over 5 minutes remaining. The interviewer gave me a pass to skip it, but I was having fun, and really wanted to take a crack at it.
The final problem was to try to figure out why logging into a particular machine with SSH was slow. While I sat waiting for a prompt, I had a number of thoughts. Is a reverse DNS lookup timing out? Is there a huge i/o load on the machine? Am I going to have to wire up strace to `sshd` and log in again?
When I finally get to a shell prompt, I instinctively run `w` and it just hangs. I hit ^C, `strace` it, and discover that it's blocking on:
fcntl64(5, F_SETLKW, {type=F_RDLCK, whence=SEEK_SET, start=0, len=0}) ...
I look up a bit, and discover that file descriptor 5 is /var/run/utmp. So `w` is trying to get an advisory lock on utmp and failing. Then it hits me, `sshd` is likely also trying to acquire a lock on utmp, failing, and then eventually timing out.
A little bit later, I've found and killed the the rogue program that held the lock, and SSH logins were fast again. Solving that last problem so quickly really boosted my spirits, and gave me the energy to push through the harder interviews that came later in the day.
Thanks w!
- hitekker 9y agoGood memory and good story.
- da02 9y agoHow did you find the rogue program?
- eric_the_read 9y agoI would probably use lsof(8)
- metaobject 9y agoOr maybe fuser
- ahh 9y agoThat actually seems like a great idea for a work sample test.
- matt_wulfeck 9y agoIn my opinion, a bit too much trivia to be a valuable indicator of SRE success. It would be really great if a candidate (who rated themselves well in systems debugging) could walk you through the strace output of a command like that.
- spudlyo 9y agoI really like that idea. It might be fun to be given some `strace` output (with the initial `execve` and writes to stdout/stderr redacted) and then be asked to determine which UNIX command it was, or more broadly what it was doing.