3 ms·

All my servers have an alarm when disk space goes above 70%. It sends an email every hour once the disk usage goes above 70%. Never had a server go down because

All my servers have an alarm when disk space goes above 70%. It sends an email every hour once the disk usage goes above 70%. Never had a server go down because of disk space issue after adopting this practise. Also one of the main reasons server disks go full is generally log files. Always remember to "logrotate" your log files and you will not have this issue that much. Yes one more thing, for all user uploaded files use external storage like NFS or S3.

ghostly_s 6y ago
Is there a package I can install to set this up?
- Moto7451 6y ago
  Icinga is a common solution for monitoring FS and other use metrics. I imagine his setup, if custom rolled, is a shell script checking df and sending an email when the usage is at or above 70
- epakai 6y ago
  I see Chris Siebenmann (https://utcc.utoronto.ca/~cks/space/blog/ https://utcc.utoronto.ca/~cks/space/blog/) blog about Prometheus (https://prometheus.io/ https://prometheus.io/) often. It seems quite capable of this and a lot more complex alerts.
bonestamp2 6y ago
> for all user uploaded files use external storage like NFS or S3 We send our log files to S3 too. I mean, we write them locally (EC2) and then push them to S3 every minute. Then we have a tool that will let us search the log files in S3 and it will parse these rotated log files and join together the relevant pieces depending on what we're looking for (or all of it for a specific time period if we don't know what we're looking for). This is great because if the server goes down and we can't access it, or the instance is gone, we can still see log files from shortly before the problem occured. We also use bugsnag, etc for real time logging and tracking where possible.
tetha 6y ago
This goes into the same vein I was going to point out. Most uncontrolled space usage comes from logs, users doing user things, or something like build servers just eating temporary and caching storage for lunch. Databases also tend to have uncontrolled space usage, but that tends to be wanted. So, if you push /var/log to it's own 20-30Gb partition, a mad logger cannot fill up /. It can kill logging, but no logging is better than fighting with a full /. Similar things with /home - let users fill up their home dirs and pout and scream about it... but / is still fine. And you can use their input to provide more storage, if they have useful workflows. Something like databases - where their primary use case is to grow - need monitoring though to add storage as necessary.
gowld 6y ago
Disk space for an active server is so cheap; why not alert at 30%?

ghostly_s 6y ago

Is there a package I can install to set this up?

Moto7451 6y ago
Icinga is a common solution for monitoring FS and other use metrics. I imagine his setup, if custom rolled, is a shell script checking df and sending an email when the usage is at or above 70
epakai 6y ago
I see Chris Siebenmann (https://utcc.utoronto.ca/~cks/space/blog/ https://utcc.utoronto.ca/~cks/space/blog/) blog about Prometheus (https://prometheus.io/ https://prometheus.io/) often. It seems quite capable of this and a lot more complex alerts.

bonestamp2 6y ago

> for all user uploaded files use external storage like NFS or S3 We send our log files to S3 too. I mean, we write them locally (EC2) and then push them to S3 every minute. Then we have a tool that will let us search the log files in S3 and it will parse these rotated log files and join together the relevant pieces depending on what we're looking for (or all of it for a specific time period if we don't know what we're looking for). This is great because if the server goes down and we can't access it, or the instance is gone, we can still see log files from shortly before the problem occured. We also use bugsnag, etc for real time logging and tracking where possible.

tetha 6y ago

This goes into the same vein I was going to point out. Most uncontrolled space usage comes from logs, users doing user things, or something like build servers just eating temporary and caching storage for lunch. Databases also tend to have uncontrolled space usage, but that tends to be wanted. So, if you push /var/log to it's own 20-30Gb partition, a mad logger cannot fill up /. It can kill logging, but no logging is better than fighting with a full /. Similar things with /home - let users fill up their home dirs and pout and scream about it... but / is still fine. And you can use their input to provide more storage, if they have useful workflows. Something like databases - where their primary use case is to grow - need monitoring though to add storage as necessary.

gowld 6y ago

Disk space for an active server is so cheap; why not alert at 30%?