8 ms·

For everyone saying "This isn't a real solution!" I'd like to explain why I think you're wrong. 1) It's not intended to be a Real Solution(tm). It's intended t

For everyone saying "This isn't a real solution!" I'd like to explain why I think you're wrong. 1) It's not intended to be a Real Solution(tm). It's intended to buy the admin some time to solve the Real Issue. 2) Having a failsafe on standby such as this will save an admin's butt when it's 2am and PagerDuty won't shut up, and you're just awake enough to apply a temp fix and work on it in the morning. 3) Because "FIX IT NOW OR ELSE" is a thing. Okay, sure. Null the file and then fill it with 7GB. Problem solved, for now. Everybody is happy and now I can work on the Real Problem: Bob won't stop hoarding spam. That is all.

luckylion 6y ago
> 1) It's not intended to be a Real Solution(tm). It's intended to buy the admin some time to solve the Real Issue. If you don't have monitoring, will you even be aware that your disk is filling up? If you do have monitoring, why are you artificially filling up your disk so that it will be at 100% more quickly instead of just setting your monitoring up to alert you when it's at $whateverItWasSetToMinusEightGB?
- deleted 6y ago
  [deleted]
- smarx007 6y ago
  An extra failsafe? You can do both. What if your cron/netdata are not forwarding emails for some reason (eg nullmailer gets errors from Mailgun)?
  - luckylion 6y ago
    Right, but again, what good does the spacer file do if you're not aware that you're running low on disk space? That is: if your monitoring isn't working, how do you know that you need to quickly make room? And if your monitoring is working correctly, the spacer file really serves no purpose other than lowering the available disk space.
    - smarx007 6y ago
      1. When your DBMS is no longer responding to queries, your boss and your customers replace your monitoring system (unlimited free phone calls 24/7 included ;). Case in point: HN is often a better place to check than Google Cloud status page, for example. 2. Maybe you didn't get it, but "nullmailer not forwarding cron email due to mailgun problems" was a bit too specific to be an example I just made up, wasn't it? Again, the premise "if your monitoring is working correctly" is not a good one to base your reasoning upon. Especially if you have 1 VM (VPS) and not a whole k8s cluster with a devops team with rotational on-call assignments.
    - pvorb 6y ago
      Somebody will notify you. If the service is just for yourself, you don't need monitoring at all.
      - luckylion 6y ago
        Yes, yes, but they will notify you after your service is down (because that's when they notice), in part thanks to a spacer file that eats up available disk space without being of any use. A monitoring service would notify you before your service is down, users grab pitchforks and start looking for torches. I understand the benefit to be able to quickly delete some file to be able to run some command that would need space, though I find that highly theoretical. If it's your shell that requires space to start, you won't be able to run the command to remove the spacer, and once you're in the shell, I've never found it hard to clean up space; path autocompletion is the only noticeable victim usually. And at this point, the services are down anyhow, and you likely don't want to restart them before figuring out what the problem was, so I don't see the point of quickly being able to make some room. It feels like "having two flat tires at the same it is highly unlikely, so I always drive with a flat tire just to make sure I don't get an unforeseen flat tire". It's cute, but I'd look for a new job if anyone in the company suggested that unironically.
        pvorb 6y ago
        This is an additional safety net. It's like doing backups. Of course you should replace your hard drive before the other drive breaks down, but you want to have a backup in case your server burns down.
- ineedasername 6y ago
  Because even if you have monitoring, some unforseen issue rapidly eating disk space at 3:00 am may not give you the time to solve it without downtime or degraded performance unless you can immediately remove the bottleneck while you troubleshoot.
  - tshaddox 6y ago
    Then why not automate the removal of the 8 GB spacer file when the disk gets full? Or in other words, just sound your alarms when there is 8 GB of free disk space.
    - ineedasername 6y ago
      I actually suggested exactly that in another comment, thoigtnto do it in stages: 4gb with an alarm, the more alarms and the other 4gb if not resolved.
    - kevincox 6y ago
      Because if it is a broken process then it will fill up the disk again before you wake up and look at it. I think the idea is that once you are at the system you can try to find out the cause without removing the file, or worse case remove the file and act fast (you may be on a short timer at this point). So for example if you find out that process X broke and is writing a ton of logs you can disable that process, remove the file, then most of your system is operational while you can properly fix the root cause or at the very least decide how to handle the data that filled up the disk in the first place. (You can't always just delete it without thought) I think a more refined approach would be disk quotas that ensured that root (or a debugging user) always had a buffer to do the repairs. This file just serves as a system wide disk quota (but you need to remove it to take advantage of that reserved space).
- marricks 6y ago
  Monitors can fail, you can miss an email, etc etc etc There's always a big gap between what should never happen because you planned well and what does happen
- ben509 6y ago
  One argument in favor of it is the 8GB file may cause a runaway process to crash, leaving you without it continuing to chew up space and able to recover. A second argument is it's not opened by any process. One problem I've had fixing disk full errors was figuring out which process still had a file open. (For any POSIX noobs: the space occupied by a file is controlled by its inode. Deleting a file "unlinks" the inode from the directory, but an open filehandle counts as a link to that inode. Until all links to the inode are deleted, the OS won't release the space occupied by the file. Particularly with log files, you need to kill any processes that have it open to actually reclaim the disk space.)
  - ganafagol 6y ago
    lsof
    - kevincox 6y ago
      Except that you probably don't realize that a process had it open until after you deleted it.
- apocalyptic0n3 6y ago
  Besides runaway log files that aren't being properly rotated, human error can cause it too. I managed to completely eat up the disk space of one of our staging servers a few weeks ago trying to tar up a directory so I could work on it locally. Didn't realize the directory was 31GB and we only had 25GB of space. By the time the notification for 80% usage was triggered (no more than 2 minutes after we hit 80%), the entire disk was full. Luckily it was just a staging server and no real harm was done, but such a mistake could have just as easily been made on a production server. In this case, the obvious solution is to just delete the file you were creating but if you're running a more complicated process that is generating logs and many files, it may not be so easy and this 8GB empty file might be useful after you cancel the process.
tehjoker 6y ago
This reminds me of the reserve tank toggle on some motorcycles. When you run out of gas, you switch the toggle and drive directly to a gas station.
- 0_____0 6y ago
  always fun when you're barreling down the highway and the engine starts to lean out, prompting you to hurriedly locate and switch the petcock over before the engine stalls completely. suppose then that you go fill up and forget to set the petcock back to normal. 8ball says: "I see a long walk in your future."
  - geocrasher 6y ago
    I once put a new fuel pump in a Chevy pickup with two tanks on the side of the road because I was switched to the empty tank. Good times.
  - jessaustin 6y ago
    IME it doesn't take too many hikes to learn that part of the procedure for turning off the engine is "turn the fuel switch off reserve".
    - 0_____0 6y ago
      out of years of riding it's only happened to me a couple times. one time i was eastbound on the bay bridge when my bike started to sputter. i'd just reassembled the tank and had left the screw-style reserve fuel valve open, so there was no reserve fuel to be had. a very kind lady put her blinkers on behind me and followed as i coasted the last few hundred yards toward yerba buena island. i pushed my bike up the ramp and looked in the tank to assess. it's a dirtbike, so the tank has two distinct "lobes" to accomodate the top tube of the frame. I had a few ounces in the tank but they were not in the lobe with the fuel pickup, so i dumped the bike on its side to get the fuel to slosh over to where i wanted it. i got back on the highway and, going quite slowly and gently, managed to get to the gas station at west oakland bart, the engine leaning out and sputtering right as i rolled into their lot.
dilyevsky 6y ago
Setup proper monitoring and never get to the Real Issue to begin with. These sysadmin hacks are not helpful
- geocrasher 6y ago
  In a perfect world, this is true. But we don't have one of those.
  - dilyevsky 6y ago
    That’s not perfect world this is standard industry practice to have host level monitoring
    - akdma 6y ago
      THANK YOU. How are so many people in this thread content with saying “monitoring isn’t perfect, this solution is ingenious”. Ofc nothing is perfect and even when you do everything right things can still go wrong, but if you don’t have a ROBUST monitoring/alert system in place then you’re not even doing the bare minimum. They’re acting like it’s rocket science to set thresholds, and have meaningful alerts and checks in place. Not to mention if you wait until disk full you risk issues like block corruption among others and your 8GB of space doesn’t do anything. It’s why people in this industry are on call, it’s why they have monitoring on their monitoring systems. The bare minimum
      - dilyevsky 6y ago
        Yeah it’s crazy. If someone does this on their homelab server it’s probably fine but if they run it in production I really want to know because Im not buying jack from them.
    - geocrasher 6y ago
      Of course! But do you put all your trust in your monitoring, 100%? You've never had monitoring fail for any reason at all? You've never had a server fill up before you can respond to the alert? This 8gb file idea isn't to replace monitoring. It's to offer a quick stopgap solution so you can do things in a hurry and give yourself a little extra "out" when things go awry. Because believe me, they WILL go awry. And if you're not prepared for that eventuality, then I don't know what else to say.
tlibert 6y ago
Real Solutions (tm) are indeed nice, but hackers get shit done - this is an utterly shameless hack, and I do it myself.
- burnte 6y ago
  It's a tool, and should be celebrated as such. It gives you breathing rom to actually solve the problem. It's an early warning system. :)
keeperofdakeys 6y ago
I find that either a server needs more space, or has files that can be deleted. For the former you just increase the disk space, since most things are VMs these days and increasing space is easy. For the latter you can usually delete enough files to get the service back up before you start the proper cleanup. If you really need some reserve space (physical server), I'd much rather store it in a vg (or zfs/btrfs subvolume). Will you remember the file exists at 2am? What about the other admins on your team?
- cbo100 6y ago
  > Will you remember the file exists at 2am? What about the other admins on your team? Hopefully if you were doing something like this it would be part of your standard incident response runsheet/checklist.
- bastawhiz 6y ago
  > Will you remember the file exists at 2am? As someone who has been woken up at 2am for this exact issue, emphatically yes. I would much rather be back in bed than trying to Google the command to find large files on disk.
solidasparagus 6y ago
This is one of those great solutions where they got 90% of the value of the Real Solution(tm) with 5 minutes of work.
das4119 6y ago
I agree with this assessment. Of course its not a solution. Its delaying the inevitable. But depending on the rate of "filling up the disk for unknown reason" it will buy you time. So when you're running out of space, you immediately delete the junk file. Suddenly there's "No Problem" and you've reset the symptom back to hopefully well before it was an issue. Now you can run whatever you need to, do reports, do traces etc. Even add more storage if necessary. More importantly, as soon as you delete that junk file now you have space for logs. You have space and time for investigation.
cmsj 6y ago
> 1) It's not intended to be a Real Solution(tm). It's intended to buy the admin some time to solve the Real Issue. It doesn't do that though. If you don't have monitoring/alerting that can either a) give you sufficient notice that you're trending out of disk space, b) take action on its own (e.g. defensively shutting down the machine), or c) both of the above, then having your server disks fill up is bad whether you have a ballast file or not. If your database server goes to 100%, you can't trust your database anymore whether you could ssh in and delete an 8GB file or not. That's it.

luckylion 6y ago

> 1) It's not intended to be a Real Solution(tm). It's intended to buy the admin some time to solve the Real Issue. If you don't have monitoring, will you even be aware that your disk is filling up? If you do have monitoring, why are you artificially filling up your disk so that it will be at 100% more quickly instead of just setting your monitoring up to alert you when it's at $whateverItWasSetToMinusEightGB?

deleted 6y ago
[deleted]
smarx007 6y ago
An extra failsafe? You can do both. What if your cron/netdata are not forwarding emails for some reason (eg nullmailer gets errors from Mailgun)?
- luckylion 6y ago
  Right, but again, what good does the spacer file do if you're not aware that you're running low on disk space? That is: if your monitoring isn't working, how do you know that you need to quickly make room? And if your monitoring is working correctly, the spacer file really serves no purpose other than lowering the available disk space.
  - smarx007 6y ago
    1. When your DBMS is no longer responding to queries, your boss and your customers replace your monitoring system (unlimited free phone calls 24/7 included ;). Case in point: HN is often a better place to check than Google Cloud status page, for example. 2. Maybe you didn't get it, but "nullmailer not forwarding cron email due to mailgun problems" was a bit too specific to be an example I just made up, wasn't it? Again, the premise "if your monitoring is working correctly" is not a good one to base your reasoning upon. Especially if you have 1 VM (VPS) and not a whole k8s cluster with a devops team with rotational on-call assignments.
  - pvorb 6y ago
    Somebody will notify you. If the service is just for yourself, you don't need monitoring at all.
    - luckylion 6y ago
      Yes, yes, but they will notify you after your service is down (because that's when they notice), in part thanks to a spacer file that eats up available disk space without being of any use. A monitoring service would notify you before your service is down, users grab pitchforks and start looking for torches. I understand the benefit to be able to quickly delete some file to be able to run some command that would need space, though I find that highly theoretical. If it's your shell that requires space to start, you won't be able to run the command to remove the spacer, and once you're in the shell, I've never found it hard to clean up space; path autocompletion is the only noticeable victim usually. And at this point, the services are down anyhow, and you likely don't want to restart them before figuring out what the problem was, so I don't see the point of quickly being able to make some room. It feels like "having two flat tires at the same it is highly unlikely, so I always drive with a flat tire just to make sure I don't get an unforeseen flat tire". It's cute, but I'd look for a new job if anyone in the company suggested that unironically.
      - pvorb 6y ago
        This is an additional safety net. It's like doing backups. Of course you should replace your hard drive before the other drive breaks down, but you want to have a backup in case your server burns down.
ineedasername 6y ago
Because even if you have monitoring, some unforseen issue rapidly eating disk space at 3:00 am may not give you the time to solve it without downtime or degraded performance unless you can immediately remove the bottleneck while you troubleshoot.
- tshaddox 6y ago
  Then why not automate the removal of the 8 GB spacer file when the disk gets full? Or in other words, just sound your alarms when there is 8 GB of free disk space.
  - ineedasername 6y ago
    I actually suggested exactly that in another comment, thoigtnto do it in stages: 4gb with an alarm, the more alarms and the other 4gb if not resolved.
  - kevincox 6y ago
    Because if it is a broken process then it will fill up the disk again before you wake up and look at it. I think the idea is that once you are at the system you can try to find out the cause without removing the file, or worse case remove the file and act fast (you may be on a short timer at this point). So for example if you find out that process X broke and is writing a ton of logs you can disable that process, remove the file, then most of your system is operational while you can properly fix the root cause or at the very least decide how to handle the data that filled up the disk in the first place. (You can't always just delete it without thought) I think a more refined approach would be disk quotas that ensured that root (or a debugging user) always had a buffer to do the repairs. This file just serves as a system wide disk quota (but you need to remove it to take advantage of that reserved space).
marricks 6y ago
Monitors can fail, you can miss an email, etc etc etc There's always a big gap between what should never happen because you planned well and what does happen
ben509 6y ago
One argument in favor of it is the 8GB file may cause a runaway process to crash, leaving you without it continuing to chew up space and able to recover. A second argument is it's not opened by any process. One problem I've had fixing disk full errors was figuring out which process still had a file open. (For any POSIX noobs: the space occupied by a file is controlled by its inode. Deleting a file "unlinks" the inode from the directory, but an open filehandle counts as a link to that inode. Until all links to the inode are deleted, the OS won't release the space occupied by the file. Particularly with log files, you need to kill any processes that have it open to actually reclaim the disk space.)
- ganafagol 6y ago
  lsof
  - kevincox 6y ago
    Except that you probably don't realize that a process had it open until after you deleted it.
apocalyptic0n3 6y ago
Besides runaway log files that aren't being properly rotated, human error can cause it too. I managed to completely eat up the disk space of one of our staging servers a few weeks ago trying to tar up a directory so I could work on it locally. Didn't realize the directory was 31GB and we only had 25GB of space. By the time the notification for 80% usage was triggered (no more than 2 minutes after we hit 80%), the entire disk was full. Luckily it was just a staging server and no real harm was done, but such a mistake could have just as easily been made on a production server. In this case, the obvious solution is to just delete the file you were creating but if you're running a more complicated process that is generating logs and many files, it may not be so easy and this 8GB empty file might be useful after you cancel the process.

tehjoker 6y ago

This reminds me of the reserve tank toggle on some motorcycles. When you run out of gas, you switch the toggle and drive directly to a gas station.

0_____0 6y ago
always fun when you're barreling down the highway and the engine starts to lean out, prompting you to hurriedly locate and switch the petcock over before the engine stalls completely. suppose then that you go fill up and forget to set the petcock back to normal. 8ball says: "I see a long walk in your future."
- geocrasher 6y ago
  I once put a new fuel pump in a Chevy pickup with two tanks on the side of the road because I was switched to the empty tank. Good times.
- jessaustin 6y ago
  IME it doesn't take too many hikes to learn that part of the procedure for turning off the engine is "turn the fuel switch off reserve".
  - 0_____0 6y ago
    out of years of riding it's only happened to me a couple times. one time i was eastbound on the bay bridge when my bike started to sputter. i'd just reassembled the tank and had left the screw-style reserve fuel valve open, so there was no reserve fuel to be had. a very kind lady put her blinkers on behind me and followed as i coasted the last few hundred yards toward yerba buena island. i pushed my bike up the ramp and looked in the tank to assess. it's a dirtbike, so the tank has two distinct "lobes" to accomodate the top tube of the frame. I had a few ounces in the tank but they were not in the lobe with the fuel pickup, so i dumped the bike on its side to get the fuel to slosh over to where i wanted it. i got back on the highway and, going quite slowly and gently, managed to get to the gas station at west oakland bart, the engine leaning out and sputtering right as i rolled into their lot.

dilyevsky 6y ago

Setup proper monitoring and never get to the Real Issue to begin with. These sysadmin hacks are not helpful

geocrasher 6y ago
In a perfect world, this is true. But we don't have one of those.
- dilyevsky 6y ago
  That’s not perfect world this is standard industry practice to have host level monitoring
  - akdma 6y ago
    THANK YOU. How are so many people in this thread content with saying “monitoring isn’t perfect, this solution is ingenious”. Ofc nothing is perfect and even when you do everything right things can still go wrong, but if you don’t have a ROBUST monitoring/alert system in place then you’re not even doing the bare minimum. They’re acting like it’s rocket science to set thresholds, and have meaningful alerts and checks in place. Not to mention if you wait until disk full you risk issues like block corruption among others and your 8GB of space doesn’t do anything. It’s why people in this industry are on call, it’s why they have monitoring on their monitoring systems. The bare minimum
    - dilyevsky 6y ago
      Yeah it’s crazy. If someone does this on their homelab server it’s probably fine but if they run it in production I really want to know because Im not buying jack from them.
  - geocrasher 6y ago
    Of course! But do you put all your trust in your monitoring, 100%? You've never had monitoring fail for any reason at all? You've never had a server fill up before you can respond to the alert? This 8gb file idea isn't to replace monitoring. It's to offer a quick stopgap solution so you can do things in a hurry and give yourself a little extra "out" when things go awry. Because believe me, they WILL go awry. And if you're not prepared for that eventuality, then I don't know what else to say.

tlibert 6y ago

Real Solutions (tm) are indeed nice, but hackers get shit done - this is an utterly shameless hack, and I do it myself.

burnte 6y ago
It's a tool, and should be celebrated as such. It gives you breathing rom to actually solve the problem. It's an early warning system. :)

keeperofdakeys 6y ago

I find that either a server needs more space, or has files that can be deleted. For the former you just increase the disk space, since most things are VMs these days and increasing space is easy. For the latter you can usually delete enough files to get the service back up before you start the proper cleanup. If you really need some reserve space (physical server), I'd much rather store it in a vg (or zfs/btrfs subvolume). Will you remember the file exists at 2am? What about the other admins on your team?

cbo100 6y ago
> Will you remember the file exists at 2am? What about the other admins on your team? Hopefully if you were doing something like this it would be part of your standard incident response runsheet/checklist.
bastawhiz 6y ago
> Will you remember the file exists at 2am? As someone who has been woken up at 2am for this exact issue, emphatically yes. I would much rather be back in bed than trying to Google the command to find large files on disk.

solidasparagus 6y ago

This is one of those great solutions where they got 90% of the value of the Real Solution(tm) with 5 minutes of work.

das4119 6y ago

I agree with this assessment. Of course its not a solution. Its delaying the inevitable. But depending on the rate of "filling up the disk for unknown reason" it will buy you time. So when you're running out of space, you immediately delete the junk file. Suddenly there's "No Problem" and you've reset the symptom back to hopefully well before it was an issue. Now you can run whatever you need to, do reports, do traces etc. Even add more storage if necessary. More importantly, as soon as you delete that junk file now you have space for logs. You have space and time for investigation.

cmsj 6y ago

> 1) It's not intended to be a Real Solution(tm). It's intended to buy the admin some time to solve the Real Issue. It doesn't do that though. If you don't have monitoring/alerting that can either a) give you sufficient notice that you're trending out of disk space, b) take action on its own (e.g. defensively shutting down the machine), or c) both of the above, then having your server disks fill up is bad whether you have a ballast file or not. If your database server goes to 100%, you can't trust your database anymore whether you could ssh in and delete an 8GB file or not. That's it.