2 ms·
This is the origin story for anyone who has ever set up alerts for 90% disk utilization on a machine. These are especially bad with services that generally gro
by hinkley 2y ago
This is the origin story for anyone who has ever set up alerts for 90% disk utilization on a machine.
These are especially bad with services that generally grow their logs very slowly but when something like a net split happens or a server is down they generate as much log data in an hour as they typically do all week. So you get close to full and then an incident happens, and now you have two incidents because the screaming machine goes down with a full disk right after you lose your internal DNS server or what have you.
- 3eb7988a1663 2y agoOne trick I learned is to create a big random file (must be random to ensure no funny compression filesystem tricks out smart you) called something like BIG_DUMMY_DATA_SAFE_TO_DELETE. When you find yourself in a catastrophic space situation, you can delete the file and have a less panicky recovery process as the immediate problem is gone.
- nneonneo 2y agoWorks right up until you have another incident, go to delete the file, and realize you already deleted it the first time…