5 ms·
Hm. I am impressed by the casual kernel debugging. I couldn't do that. However, why would I have to do that if running check_disk on all important file systems
by tetha 7y ago
Hm. I am impressed by the casual kernel debugging. I couldn't do that. However, why would I have to do that if running check_disk on all important file systems tells me if file systems are >80% or >90% full?
There'll be bugs I cannot debug, because I cannot casually debug my linux kernel. Yes. I still have to encounter them personally, but I'll get there. But this just seems weird, because this is a very, very basic problem to monitor for.
- rkeene2 7y agoThe great thing about Solaris is all the debugging tools, the bad thing about Solaris is you have to use all the debugging tools. As to why you might want to do this -- it wasn't obvious that the filesystem being full was related to "echo *" in /proc hanging, until after the debugging was done.
- tetha 7y agoMh, I fully agree that I don't expect '(cd /proc; echo *)' to hang because of the disk being full. However, with my current setup, the alerting checks all disks of all monitored systems every minute, and once a single of them exceeds 80%, we get alerts. That's no smart setup, that's stock nagios/iciniga/icinga2 with stock nrpe checks. This is considered a very cheap and basic setup when running servers. We've had this on our solaris systems out of the box. And practically, this is an alert with a very high true positive rate and very few false positive alerts. Most false positives we've had to deal with had been with systems with > 3-4 Tb of storage. And systems filling up more than 5% - 10% of their storage per minute tend to trigger other stock alerts as well.
- icedchai 7y agoNobody thought to run "df"? Disk being full is a common source of all sorts of weird server problems.
- Alupis 7y ago> Nobody thought to run "df"? Disk being full is a common source of all sorts of weird server problems. Yep, to the point where `df -h` has become one of the first things I run when a server starts acting funny or things stop working. Disk being full is far too common - run away logging, weird temp files, etc, or sometimes just a box that nobody maintained for years and years. The fun part starts after you've identified which partition or drive is full - now you have to identify the problem files!
- mobilemidget 7y agoFirst things you run on login? It ought to be monitored with alerting by default and inode use too.
- icedchai 7y agoTrue. Unfortunately good monitoring is an afterthought in most organizations.