6 ms·

I run df first, very often something stopped working because the disk space ran out

by dvh 9y ago

avip 9y ago
If you have a tiny fleet of instances, datadog free plan would solve that for you (not affiliated, it's a good product).
Symbiote 9y ago
That's one thing I don't need to do. If the disk gets to 80%, I get an email from the monitoring system. 90% and I get a text.
- letientai299 9y ago
  Could you please share how did you setup to be able to get such notification?
  - vk23 9y ago
    Have a look at nagios. It's a popular open source server/application monitoring tool. You can define custom alerts and monitor basicly everything.
  - FuckOffNeemo 9y ago
    Not OP, but we use Nagios XI for such alerting, it provides SMS and email monitoring for infrastructure (disk space as an example) and service monitoring. [1] Their open source NCPA (Nagios Cross Platform agent) plugin for Windows and Linux hosts is pretty great, though we were stung when we found out it didn't natively support SPARC as of yet unless you compiled your own client from the source (I did). [2] Plenty of other monitoring services equivalent to Nagios offer equivalent services. Nagios was just the flavour I suggested to my current employer because of my familiarity with the product over my last few employments. If you don't want to fork out the licensing for the supported version (XI), their open source version is free and relatively easy to deploy using Ansible if you don't mind writing in PHP. [3] Edit: words, grammar. [1] https://www.nagios.com/products/nagios-xi/ https://www.nagios.com/products/nagios-xi/ [2] https://github.com/NagiosEnterprises/ncpa https://github.com/NagiosEnterprises/ncpa [3] https://hobo.house/2016/06/24/automate-nagios-deployment-with-ansible/ https://hobo.house/2016/06/24/automate-nagios-deployment-wit...
  - more-coffee 9y ago
    I will give a simplified example of the only product I've used for this: Zabbix. Disclaimer: I'm not a sysadmin but a developer, who happened to work somewhere they used Zabbix for monitoring all kinds of network devices, databases, services, etc. Perhaps there are better solutions available, I welcome any discussion on that. Deploying Zabbix roughly consists of setting up one (or more) Zabbix Master servers, and installing a 'zabbix-agent' process on each device you want to monitor. The agent process extracts a variety of statistics about the system and makes them available to the Monitoring server, either in push ("active") or pull ("passive") fashion. The Master logs all of these statistics over time. You can then define 'Triggers' that apply logical tests to statistics, such as "in the last 5 minutes, was the free diskspace < 10GB". When this happens, it triggers an event. Then elsewhere you have defined your Notification rules that act upon generated events, to send out an email or text message. https://www.zabbix.com/features#notification https://www.zabbix.com/features#notification This was obviously really simplified, and there is so much more you can do, but I hope this at least gave you a basic picture.
  - navinsylvester 9y ago
    It's very straight forward to setup. I have used Prometheus for the same and so far the experience has been much better compared to Zabbix. # node_exporter on the client to get metrics # prometheus on server to pull the metrics from node_exporter # alertmanager for raising alerts # grafana on top of it for pretty graphs(optional)
  - softawre 9y ago
    Outside of nagios, you can also use pricier but arguably more fully featured setups like New Relic + PagerDuty.
  - mdekkers 9y ago
    Have a look at netdata: https://github.com/firehol/netdata https://github.com/firehol/netdata
- jiri 9y ago
  Anecdocte: our sysadmin set similar thing up (at 90%) at one of my boxes. but conflicting configuration of debian os also reserved 15% of capacity for OS, so system became completely unresponsive at 85% full disk, without emailing message about free space - 85% was for our purpose effectively 100% full disk. Still dont know what lesson I should learn from this story.
  - more-coffee 9y ago
    Reserved capacity as in `sudo tune2fs <volume> | grep Reserved block count`? Those should already be excluded from available diskspace, so that is kind of interesting.
    - jiri 9y ago
      I am not sure how he was getting free space for email warning, but in final state, running df showed plenty of space on disk, although writing to files signals "no disk space" error.
      - db48x 9y ago
        Reserved capacity is reserved for use by root, so when a process running as root runs df (or any similar command or syscall), then it sees that capacity as available. Kinda useless these days, but was very useful back when you might have server daemons and a bunch of normal users with shell accounts on the same machine.
        teddyh 9y ago
        Huh? I have never seen that behavior, and I can’t reproduce it as described. Did it ever work that way? Do you have any reference for this?
        db48x 9y ago
        Well, there's man tune2fs: -m reserved-blocks-percentage Set the percentage of the filesystem which may only be allocated by privileged processes. Reserving some number of filesystem blocks for use by privi‐ leged processes is done to avoid filesystem fragmentation, and to allow system daemons, such as syslogd(8), to continue to function correctly after non- privileged processes are prevented from writing to the filesystem. Normally, the default percentage of reserved blocks is 5%. -g group Set the group which can use the reserved filesystem blocks. The group parameter can be a numerical gid or a group name. If a group name is given, it is converted to a numerical gid before it is stored in the superblock.
shit_parade2 9y ago
the very definition of a failed monitoring service.
recentdarkness 9y ago
I don’t I rather run top to see some stats of the system. Mostly I am checking for slow downs and that shows me load mem consumption etc. Diskspace is rarely the case Most of the time some database connection is laggy
- dspillett 9y ago
  Or, where available, htop.
  - linedash 9y ago
    atop is best top. One of the few bits of software I evangelise. Writes to replayable binary logfile with 10 minute system-state snapshots and uses process accounting to ensure it see's every process during that time. Gives more metrics than any other top; including network, disk and all the counters that you have to check the man page to know what they refer to. Its counterpart atopsar lets you replay the data for specific stats in an easily viewable format; i.e; - atopsar -m - this shows the memory stats for todays logfile in 10 minute increments. It goes on every single server I manage without exception. With atop you can actually see why it died instead of guessing from old log entries. Screenshot of atop : https://www.atoptool.nl/images/screenshots/genericw.png https://www.atoptool.nl/images/screenshots/genericw.png Example output of atopsar: # atopsar -m *snipped* 2.6.32-896.16.1.lve1.4.51.el6.x86_64 #1 SMP Wed Jan 17 13:19:23 EST 2018 x86_64 2018/03/27 -------------------------- analysis date: 2018/03/27 -------------------------- 00:00:02 memtotal memfree buffers cached dirty slabmem swptotal swpfree _mem_ 00:10:02 14027M 928M 839M 6335M 1M 4645M 0M 0M 00:20:02 14027M 963M 842M 6393M 1M 4641M 0M 0M 00:30:02 14027M 756M 873M 6617M 1M 4638M 0M 0M 00:40:02 14027M 576M 871M 6596M 3M 4634M 0M 0M https://www.atoptool.nl/index.php https://www.atoptool.nl/index.php
    - teddyh 9y ago
      I sometimes hear about “atop”, wonder “Why don’t I have this installed?”, install it, discover that it starts (and requires) two additional daemon processes, at which point I remember, and promptly uninstall it again.
aepiepaey 9y ago
...and then df gets stuck in uninterruptible sleep (due to a file system hang), and ^C and ^Z does nothing.
- executesorder66 9y ago
  How would you avoid that, and still find out the disk usage?
  - zbentley 9y ago
    You wouldn't. If the filesystem is hung, or if some common path is never yielding out of blocking-no-matter-what-calls (e.g. stat()), then the presence of the hang itself would indicate an issue. The isolation process for me would probably be something like: 1. df -h; notice that it hangs. 2. Log in to a new shell, 'strace' the old process or a new one doing the same thing, see what path it was choking on. 3. If the breakage is on an external/network filesystem, reboot the host in almost every case. Unless it was happily completing day 364/365 of some incredibly important task elsewhere, it's just not worth my time to remount a dead share and go clean up everything that was broken trying to talk to the old one. I've had database servers lose some random NFS share that the DB process wasn't using, then crash months later due to PID exhaustion because some monitoring script in cron that kept trying to talk to a somehow-corrupted mountpoint and hanging forever. Yes, timeouts and client programs should be able to handle these failures perfectly in theory. Given my experience, I have very little faith in theory matching up with reality. 4. If it's on an internal drive, check dmesg/syslog (if I can) for any smoking guns. Reboot and see if the problem goes away. If it does, unless I can find something blindingly obvious indicating that the issue was transient and unlikely to reoccur, I'm probably reprovisioning the system after a hardware diagnostic. Even if the server isn't critical and just serves a cat blog or whatever, it's not worth my time and repeated head-scratching to deal with issues like this more than once per host. 5. If I need data off of the questionable filesystem, I'll get it exclusively via a recovery environment; not worth the risk in the case of a flaky/failing drive otherwise (this applies even if the server itself was virtualized). I hope the server had some sort of LOM console set up so I can do that, otherwise someone's getting travel expenses for my trip onsite. Edits: grammar.

I run df first, very often something stopped working because the disk space ran out

by dvh 9y ago

I run df first, very often something stopped working because the disk space ran out

avip 9y ago

If you have a tiny fleet of instances, datadog free plan would solve that for you (not affiliated, it's a good product).

Symbiote 9y ago

That's one thing I don't need to do. If the disk gets to 80%, I get an email from the monitoring system. 90% and I get a text.

letientai299 9y ago
Could you please share how did you setup to be able to get such notification?
- vk23 9y ago
  Have a look at nagios. It's a popular open source server/application monitoring tool. You can define custom alerts and monitor basicly everything.
- FuckOffNeemo 9y ago
  Not OP, but we use Nagios XI for such alerting, it provides SMS and email monitoring for infrastructure (disk space as an example) and service monitoring. [1] Their open source NCPA (Nagios Cross Platform agent) plugin for Windows and Linux hosts is pretty great, though we were stung when we found out it didn't natively support SPARC as of yet unless you compiled your own client from the source (I did). [2] Plenty of other monitoring services equivalent to Nagios offer equivalent services. Nagios was just the flavour I suggested to my current employer because of my familiarity with the product over my last few employments. If you don't want to fork out the licensing for the supported version (XI), their open source version is free and relatively easy to deploy using Ansible if you don't mind writing in PHP. [3] Edit: words, grammar. [1] https://www.nagios.com/products/nagios-xi/ https://www.nagios.com/products/nagios-xi/ [2] https://github.com/NagiosEnterprises/ncpa https://github.com/NagiosEnterprises/ncpa [3] https://hobo.house/2016/06/24/automate-nagios-deployment-with-ansible/ https://hobo.house/2016/06/24/automate-nagios-deployment-wit...
- more-coffee 9y ago
  I will give a simplified example of the only product I've used for this: Zabbix. Disclaimer: I'm not a sysadmin but a developer, who happened to work somewhere they used Zabbix for monitoring all kinds of network devices, databases, services, etc. Perhaps there are better solutions available, I welcome any discussion on that. Deploying Zabbix roughly consists of setting up one (or more) Zabbix Master servers, and installing a 'zabbix-agent' process on each device you want to monitor. The agent process extracts a variety of statistics about the system and makes them available to the Monitoring server, either in push ("active") or pull ("passive") fashion. The Master logs all of these statistics over time. You can then define 'Triggers' that apply logical tests to statistics, such as "in the last 5 minutes, was the free diskspace < 10GB". When this happens, it triggers an event. Then elsewhere you have defined your Notification rules that act upon generated events, to send out an email or text message. https://www.zabbix.com/features#notification https://www.zabbix.com/features#notification This was obviously really simplified, and there is so much more you can do, but I hope this at least gave you a basic picture.
- navinsylvester 9y ago
  It's very straight forward to setup. I have used Prometheus for the same and so far the experience has been much better compared to Zabbix. # node_exporter on the client to get metrics # prometheus on server to pull the metrics from node_exporter # alertmanager for raising alerts # grafana on top of it for pretty graphs(optional)
- softawre 9y ago
  Outside of nagios, you can also use pricier but arguably more fully featured setups like New Relic + PagerDuty.
- mdekkers 9y ago
  Have a look at netdata: https://github.com/firehol/netdata https://github.com/firehol/netdata
jiri 9y ago
Anecdocte: our sysadmin set similar thing up (at 90%) at one of my boxes. but conflicting configuration of debian os also reserved 15% of capacity for OS, so system became completely unresponsive at 85% full disk, without emailing message about free space - 85% was for our purpose effectively 100% full disk. Still dont know what lesson I should learn from this story.
- more-coffee 9y ago
  Reserved capacity as in `sudo tune2fs <volume> | grep Reserved block count`? Those should already be excluded from available diskspace, so that is kind of interesting.
  - jiri 9y ago
    I am not sure how he was getting free space for email warning, but in final state, running df showed plenty of space on disk, although writing to files signals "no disk space" error.
    - db48x 9y ago
      Reserved capacity is reserved for use by root, so when a process running as root runs df (or any similar command or syscall), then it sees that capacity as available. Kinda useless these days, but was very useful back when you might have server daemons and a bunch of normal users with shell accounts on the same machine.
      - teddyh 9y ago
        Huh? I have never seen that behavior, and I can’t reproduce it as described. Did it ever work that way? Do you have any reference for this?
        db48x 9y ago
        Well, there's man tune2fs: -m reserved-blocks-percentage Set the percentage of the filesystem which may only be allocated by privileged processes. Reserving some number of filesystem blocks for use by privi‐ leged processes is done to avoid filesystem fragmentation, and to allow system daemons, such as syslogd(8), to continue to function correctly after non- privileged processes are prevented from writing to the filesystem. Normally, the default percentage of reserved blocks is 5%. -g group Set the group which can use the reserved filesystem blocks. The group parameter can be a numerical gid or a group name. If a group name is given, it is converted to a numerical gid before it is stored in the superblock.

shit_parade2 9y ago

the very definition of a failed monitoring service.

recentdarkness 9y ago

I don’t I rather run top to see some stats of the system. Mostly I am checking for slow downs and that shows me load mem consumption etc. Diskspace is rarely the case Most of the time some database connection is laggy

dspillett 9y ago
Or, where available, htop.
- linedash 9y ago
  atop is best top. One of the few bits of software I evangelise. Writes to replayable binary logfile with 10 minute system-state snapshots and uses process accounting to ensure it see's every process during that time. Gives more metrics than any other top; including network, disk and all the counters that you have to check the man page to know what they refer to. Its counterpart atopsar lets you replay the data for specific stats in an easily viewable format; i.e; - atopsar -m - this shows the memory stats for todays logfile in 10 minute increments. It goes on every single server I manage without exception. With atop you can actually see why it died instead of guessing from old log entries. Screenshot of atop : https://www.atoptool.nl/images/screenshots/genericw.png https://www.atoptool.nl/images/screenshots/genericw.png Example output of atopsar: # atopsar -m *snipped* 2.6.32-896.16.1.lve1.4.51.el6.x86_64 #1 SMP Wed Jan 17 13:19:23 EST 2018 x86_64 2018/03/27 -------------------------- analysis date: 2018/03/27 -------------------------- 00:00:02 memtotal memfree buffers cached dirty slabmem swptotal swpfree _mem_ 00:10:02 14027M 928M 839M 6335M 1M 4645M 0M 0M 00:20:02 14027M 963M 842M 6393M 1M 4641M 0M 0M 00:30:02 14027M 756M 873M 6617M 1M 4638M 0M 0M 00:40:02 14027M 576M 871M 6596M 3M 4634M 0M 0M https://www.atoptool.nl/index.php https://www.atoptool.nl/index.php
  - teddyh 9y ago
    I sometimes hear about “atop”, wonder “Why don’t I have this installed?”, install it, discover that it starts (and requires) two additional daemon processes, at which point I remember, and promptly uninstall it again.

aepiepaey 9y ago

...and then df gets stuck in uninterruptible sleep (due to a file system hang), and ^C and ^Z does nothing.

executesorder66 9y ago
How would you avoid that, and still find out the disk usage?
- zbentley 9y ago
  You wouldn't. If the filesystem is hung, or if some common path is never yielding out of blocking-no-matter-what-calls (e.g. stat()), then the presence of the hang itself would indicate an issue. The isolation process for me would probably be something like: 1. df -h; notice that it hangs. 2. Log in to a new shell, 'strace' the old process or a new one doing the same thing, see what path it was choking on. 3. If the breakage is on an external/network filesystem, reboot the host in almost every case. Unless it was happily completing day 364/365 of some incredibly important task elsewhere, it's just not worth my time to remount a dead share and go clean up everything that was broken trying to talk to the old one. I've had database servers lose some random NFS share that the DB process wasn't using, then crash months later due to PID exhaustion because some monitoring script in cron that kept trying to talk to a somehow-corrupted mountpoint and hanging forever. Yes, timeouts and client programs should be able to handle these failures perfectly in theory. Given my experience, I have very little faith in theory matching up with reality. 4. If it's on an internal drive, check dmesg/syslog (if I can) for any smoking guns. Reboot and see if the problem goes away. If it does, unless I can find something blindingly obvious indicating that the issue was transient and unlikely to reoccur, I'm probably reprovisioning the system after a hardware diagnostic. Even if the server isn't critical and just serves a cat blog or whatever, it's not worth my time and repeated head-scratching to deal with issues like this more than once per host. 5. If I need data off of the questionable filesystem, I'll get it exclusively via a recovery environment; not worth the risk in the case of a flaky/failing drive otherwise (this applies even if the server itself was virtualized). I hope the server had some sort of LOM console set up so I can do that, otherwise someone's getting travel expenses for my trip onsite. Edits: grammar.