3 ms·

How would you avoid that, and still find out the disk usage?

zbentley 9y ago
You wouldn't. If the filesystem is hung, or if some common path is never yielding out of blocking-no-matter-what-calls (e.g. stat()), then the presence of the hang itself would indicate an issue. The isolation process for me would probably be something like: 1. df -h; notice that it hangs. 2. Log in to a new shell, 'strace' the old process or a new one doing the same thing, see what path it was choking on. 3. If the breakage is on an external/network filesystem, reboot the host in almost every case. Unless it was happily completing day 364/365 of some incredibly important task elsewhere, it's just not worth my time to remount a dead share and go clean up everything that was broken trying to talk to the old one. I've had database servers lose some random NFS share that the DB process wasn't using, then crash months later due to PID exhaustion because some monitoring script in cron that kept trying to talk to a somehow-corrupted mountpoint and hanging forever. Yes, timeouts and client programs should be able to handle these failures perfectly in theory. Given my experience, I have very little faith in theory matching up with reality. 4. If it's on an internal drive, check dmesg/syslog (if I can) for any smoking guns. Reboot and see if the problem goes away. If it does, unless I can find something blindingly obvious indicating that the issue was transient and unlikely to reoccur, I'm probably reprovisioning the system after a hardware diagnostic. Even if the server isn't critical and just serves a cat blog or whatever, it's not worth my time and repeated head-scratching to deal with issues like this more than once per host. 5. If I need data off of the questionable filesystem, I'll get it exclusively via a recovery environment; not worth the risk in the case of a flaky/failing drive otherwise (this applies even if the server itself was virtualized). I hope the server had some sort of LOM console set up so I can do that, otherwise someone's getting travel expenses for my trip onsite. Edits: grammar.
- insanejudge 9y ago
  Interesting (well, interesting to me) note on the nfs case, on modern linux, `umount -l` should be able to unmount pretty much anything. You'll often still be left with a pile of processes stuck in uninterruptible sleep depending on the scope of the 'random share', but at the very least it can staunch the bleeding and let you move around. TBH I get rather claustrophobic when I can't `w` with aplomb.
emmelaich 9y ago
Simple .. df -h &
sevagh 9y ago
I had a similar incident (I was able to ctrl-c out of the hanging df -h though). Luckily dmesg gave me something super clear (like `nfs host <blah> unreachable`), so I did a `umount -l` (lazy) on `/mnt/net/<blah>` and things were OK.

You wouldn't. If the filesystem is hung, or if some common path is never yielding out of blocking-no-matter-what-calls (e.g. stat()), then the presence of the hang itself would indicate an issue. The isolation process for me would probably be something like: 1. df -h; notice that it hangs. 2. Log in to a new shell, 'strace' the old process or a new one doing the same thing, see what path it was choking on. 3. If the breakage is on an external/network filesystem, reboot the host in almost every case. Unless it was happily completing day 364/365 of some incredibly important task elsewhere, it's just not worth my time to remount a dead share and go clean up everything that was broken trying to talk to the old one. I've had database servers lose some random NFS share that the DB process wasn't using, then crash months later due to PID exhaustion because some monitoring script in cron that kept trying to talk to a somehow-corrupted mountpoint and hanging forever. Yes, timeouts and client programs should be able to handle these failures perfectly in theory. Given my experience, I have very little faith in theory matching up with reality. 4. If it's on an internal drive, check dmesg/syslog (if I can) for any smoking guns. Reboot and see if the problem goes away. If it does, unless I can find something blindingly obvious indicating that the issue was transient and unlikely to reoccur, I'm probably reprovisioning the system after a hardware diagnostic. Even if the server isn't critical and just serves a cat blog or whatever, it's not worth my time and repeated head-scratching to deal with issues like this more than once per host. 5. If I need data off of the questionable filesystem, I'll get it exclusively via a recovery environment; not worth the risk in the case of a flaky/failing drive otherwise (this applies even if the server itself was virtualized). I hope the server had some sort of LOM console set up so I can do that, otherwise someone's getting travel expenses for my trip onsite. Edits: grammar.

insanejudge 9y ago
Interesting (well, interesting to me) note on the nfs case, on modern linux, `umount -l` should be able to unmount pretty much anything. You'll often still be left with a pile of processes stuck in uninterruptible sleep depending on the scope of the 'random share', but at the very least it can staunch the bleeding and let you move around. TBH I get rather claustrophobic when I can't `w` with aplomb.

emmelaich 9y ago

Simple .. df -h &

sevagh 9y ago

I had a similar incident (I was able to ctrl-c out of the hanging df -h though). Luckily dmesg gave me something super clear (like `nfs host <blah> unreachable`), so I did a `umount -l` (lazy) on `/mnt/net/<blah>` and things were OK.