4 ms·

Okay, so now you have a disk full, only become aware of it when it's full and your database throws errors. You have an easy way to fix it, just delete the space

Okay, so now you have a disk full, only become aware of it when it's full and your database throws errors. You have an easy way to fix it, just delete the spacer file. But what good does that do? You're still in the mess where your database is really unhappy. On the other hand, if your monitoring was set up well, you got a notification and had time to react to it before it was at 100%. Granted, if you have a process that just wrote a file at maximum speed, that time window is tiny, but that's not usually what happens in my experience. What happens is that something starts logging more and it slowly builds up while you're happy that your server is running so well that you don't need to pay attention. And then the alert comes and tells you that there's less than 10% space available, and you have plenty of time to investigate and avert the crisis.

vineyardmike 6y ago
>You have an easy way to fix it, just delete the spacer file. But what good does that do? You solve the issue right then and there. Step 1. realize there is a space issue and get to terminal Step 2. free space so any solution has memory Step 3. Solve by doing <??? specifics ???>
- tshaddox 6y ago
  So if your alarm sounded when there was 8 GB of free disk space (instead of 0 GB), then you could still respond in the same amount of time and you would still have an additional 8 GB worth of padding while you determined the root cause. The only difference is that you wouldn't need to actually go in and delete the spacer file (and potentially have downtime in the time it takes you to delete the spacer file). Another way to think of this is that you have the 8 GB spacer file, but when the disk fills up the spacer file is automatically deleted and your alarm goes off. Which is literally the same as having your alarm go off when free disk space reaches 8 GB.
  - zokier 6y ago
    The thing you are still missing is that no matter how you set your alarms, by the time you respond the disk can still be completely full.
    - dyingkneepad 6y ago
      The spacer solution guarantees you will always have one instance of "disk got full", even if the fill rate is slow. With an alarm there is a chance you end up solving the problem with zero instances of "disk got full" if the fill rate is not that big. That is the main difference.
  - lostcolony 6y ago
    It isn't though. Whatever rogue process is generating the garbage so quickly has likely thrown and died (potentially leaving other, useful processes, able to continue work). Not 100%, of course, but there's a solid chance that the garbage will stop being generated. Also, forcing manual intervention has a psychological effect. An alarm that goes off at 8 GB remaining? Eh, I'll get to it at some point. A "disk is full, error, error, everything is broken"? I will deal with it -right now-, especially since I know a fix. Do that, with an alarm at 16 GB (so I still get the early alert in the event I'm that good a citizen and actually prioritize getting to the bottom of it even though it's caused no issue yet), and I'm in a better position still.
- dyingkneepad 6y ago
  Let me try to explain what the other person is saying. If you have an 8gb spacer file, at some point the disk will get full and cause some errors, so you will have to log in, remove the spacer file and then deal with the problem. If you have an alarm for 8gb remaining, you will receive the alarm before any application ever notices the disk is full. You will have basically the same amount of time to solve the problem for good, but if you're able to solve it before these 8gb also ends, you won't have to deal with any "app crashed/misbehaved because it reached a point where no space was available" issues.
  - tremon 6y ago
    Your described scenario is no different than when you would have set your first warning at 16GB disk space, except that you won't have to scramble to delete the roadblock halfway between here and the crash zone.
ZoF 6y ago
No-one writing the OP's post re: 8G of wasted disk on 'all my servers' is dealing with this in meaningful enterprise production environment(although scarily perhaps). If you have an alert process great, this buffer whilst a bit retarded compared to monitoring that you trust(e.g. your above-mentioned disk warning) is still important for him. This is someone on a budget host who at the point of 'out of disk' either had his warning fail or his lack of care/monitoring let it go unnoticed and now has to resolve the issue. In that context on a host >100G base disk I am totally down with his solution. This is a real person dealing with his own linux boxes. If you've dealt with an out-of-disk system, linux or windows, you would know exactly what is meant by this.

vineyardmike 6y ago

>You have an easy way to fix it, just delete the spacer file. But what good does that do? You solve the issue right then and there. Step 1. realize there is a space issue and get to terminal Step 2. free space so any solution has memory Step 3. Solve by doing <??? specifics ???>

tshaddox 6y ago
So if your alarm sounded when there was 8 GB of free disk space (instead of 0 GB), then you could still respond in the same amount of time and you would still have an additional 8 GB worth of padding while you determined the root cause. The only difference is that you wouldn't need to actually go in and delete the spacer file (and potentially have downtime in the time it takes you to delete the spacer file). Another way to think of this is that you have the 8 GB spacer file, but when the disk fills up the spacer file is automatically deleted and your alarm goes off. Which is literally the same as having your alarm go off when free disk space reaches 8 GB.
- zokier 6y ago
  The thing you are still missing is that no matter how you set your alarms, by the time you respond the disk can still be completely full.
  - dyingkneepad 6y ago
    The spacer solution guarantees you will always have one instance of "disk got full", even if the fill rate is slow. With an alarm there is a chance you end up solving the problem with zero instances of "disk got full" if the fill rate is not that big. That is the main difference.
- lostcolony 6y ago
  It isn't though. Whatever rogue process is generating the garbage so quickly has likely thrown and died (potentially leaving other, useful processes, able to continue work). Not 100%, of course, but there's a solid chance that the garbage will stop being generated. Also, forcing manual intervention has a psychological effect. An alarm that goes off at 8 GB remaining? Eh, I'll get to it at some point. A "disk is full, error, error, everything is broken"? I will deal with it -right now-, especially since I know a fix. Do that, with an alarm at 16 GB (so I still get the early alert in the event I'm that good a citizen and actually prioritize getting to the bottom of it even though it's caused no issue yet), and I'm in a better position still.
dyingkneepad 6y ago
Let me try to explain what the other person is saying. If you have an 8gb spacer file, at some point the disk will get full and cause some errors, so you will have to log in, remove the spacer file and then deal with the problem. If you have an alarm for 8gb remaining, you will receive the alarm before any application ever notices the disk is full. You will have basically the same amount of time to solve the problem for good, but if you're able to solve it before these 8gb also ends, you won't have to deal with any "app crashed/misbehaved because it reached a point where no space was available" issues.
- tremon 6y ago
  Your described scenario is no different than when you would have set your first warning at 16GB disk space, except that you won't have to scramble to delete the roadblock halfway between here and the crash zone.

ZoF 6y ago

No-one writing the OP's post re: 8G of wasted disk on 'all my servers' is dealing with this in meaningful enterprise production environment(although scarily perhaps). If you have an alert process great, this buffer whilst a bit retarded compared to monitoring that you trust(e.g. your above-mentioned disk warning) is still important for him. This is someone on a budget host who at the point of 'out of disk' either had his warning fail or his lack of care/monitoring let it go unnoticed and now has to resolve the issue. In that context on a host >100G base disk I am totally down with his solution. This is a real person dealing with his own linux boxes. If you've dealt with an out-of-disk system, linux or windows, you would know exactly what is meant by this.