77 ms·

All my servers have an 8 GB empty file on disk

• loloquwowndueo 6y ago
  If you don’t have monitoring to tell you when the disk is more than X% full, then you’re at risk for more failure scenarios than just a full disk (usually trivial to buy time by deleting old logs).
  • ed25519FUUU 6y ago
    Happens all of the time even with monitoring. Somebody enables debug monitoring and it fills up in 3 minutes.
  • macintux 6y ago
    If the problem arises during a migration or other significant event, which it sounds like Marco's did, the alert will usually be triggered just in time to tell you why you're already in a world of pain.
• badcc 6y ago
  This trick has certainly saved me more times than I am willing to admit! I usually roll with: `fallocate -l 8G DELETE_IF_OUT_OF_SPACE.img`
  • innomatics 6y ago
    Nice. I liked the trick in the article, but was wondering if it might confuse the next admin who needs to figure it out. Was also thinking of customising the login greeting to mention the file (/etc/motd).
    • stjohnswarts 6y ago
      what if you named it "THIS_IS_A_BIG_FILE_IN_CASE_THE_DISK_FILLS_UP_DELETE_IN_EMERGENCIES.BIN"
• eecc 6y ago
  That’s what tune2fs is for https://www.unixtutorial.org/commands/tune2fs https://www.unixtutorial.org/commands/tune2fs
• xen2xen1 6y ago
  I will be doing this. Marvelous idea.
• rektide 6y ago
  hope you're not running -o compress=lz4 , because you are going to be in for a big surprise when you try to pull this emergency lever! you may be shocked to see you don't actually get much space back! i do wonder how many FS would actually allocate the 8GB if you, for example, opened a file, seeked to 8GB mark, and wrote a character. many file systems support "sparse files"[1]. for example on btrfs, i can run 'dd if=/dev/zero of=example.sparse count=1 seek=2000000' to make a "1GB" file that has just one byte in it. btrfs will only allocate a very small amount in this case, some meta-data to record an "extent", and a page of data. i was expecting this article to be about a rude-and-crude overprovisioning method[2], but couldn't guess how it was going to work. SSDs notably perform much much better when they have some empty space to make shuffling data around easier. leaving a couple GB for the drive to do whatever can be a colossal performance improvement, versus a full drive, where every operation has to scrounge around to find some free space. i wasn't sure how the author was going to make an empty file that could have this effect. but that's not what was going on here. [1] https://wiki.archlinux.org/index.php/sparse_file https://wiki.archlinux.org/index.php/sparse_file [2] https://superuser.com/questions/944913/over-provisioning-an-ssd-does-it-still-hold/944915 https://superuser.com/questions/944913/over-provisioning-an-...
  • sonicrocketman 6y ago
    Good point. I didn't think of that. I'm not sure how my CentOS servers handle this scenario, but it seemed to take up the full 8GB however I checked.
  • secabeen 6y ago
    > hope you're not running -o compress=lz4 , because you are going to be in for a big surprise when you try to pull this emergency lever! you may be shocked to see you don't actually get much space back! This is true. If you are replicating this, copy from /dev/urandom rather than using an empty file.
    • yjftsjthsd-h 6y ago
      It feels more elegant to do something like `touch /bigfile && chattr -c /bigfile && truncate --size 8G` to outright disable compression on that file
• 1123581321 6y ago
  This is clever. Our shaky version of this, historically, has been to run ncdu and hastily delete the first large log file we see. It’s not ideal.
  • sonicrocketman 6y ago
    (that's actually how I solved the original issue that I reference in the post, and how I got the idea for this silly solution)
  • kernelsanderz 6y ago
    ncdu saves my bacon at least every few months. I do machine learning and am always running out of space!
    • dylan604 6y ago
      sounds like your machine doesn't seem to be learning the right things.
  • ttyprintk 6y ago
    The filesystems where headroom matters are var, tmp and sometimes root. I like this strategy with logfiles because nethack.log.gz.30 was approximately as important as empty space. Keeping another 8gb on root and tmp seems extreme.
• 00deadbeef 6y ago
  Isn’t using LVM and holding some space back a better solution for this? Also I keep databases on their own partition so that nothing else can accidentally fill up the space and lead to data loss.
  • jstanley 6y ago
    Maybe, but author says: > On Linux servers it can be incredibly difficult for any process to succeed if the disk is full. You won't feel too clever if you come to grow your LVM volume into the free space and it won't work because there's no free space on the filesystem! :) (I don't actually know if this would fail or not - but the point is "rm spacer.img" is pretty much guaranteed not to fail).
    • LinuxBender 6y ago
      I've used LVM for this purpose plenty of times. lvm2 at least has not prevented me from extending a full disk. lvm + reserved blocks + a small spacer file are all decent options, even better when used together.
  • JshWright 6y ago
    rm /path/to/big.file is faster than looking up the commands to expand the LVM volume and grow the filesystem.
    • 00deadbeef 6y ago
      They’re not hard to memorise. I never look them up.
      • JshWright 6y ago
        Good for you. That's not how my memory works. If I don't use a command regularly, I don't trust myself to remember it correctly. Even if I did though, that's a multi-step process, compared to the single command needed to remove a file.
      • 6y ago
• Denvercoder9 6y ago
  > Copy commands and even deletions can fail or take forever as memory tries to swap to a full disk That's only a problem if your memory is full as well, and even then, I've never encountered a server that uses a swapfile instead of a swap partition.
  • jstanley 6y ago
    That's also only a problem if your swap partition is mounted from a file on your filesystem, which is an exceedingly uncommon configuration.
  • yjftsjthsd-h 6y ago
    Even a swap file shouldn't matter, since it's still not sparse. The one exception is if you're on a system that dynamically adds and removes swap files - I believe darwin does that, and I think it might be possible to do on Linux(?) but I've not actually seen it done.
  • Deathmax 6y ago
    Not quite the same situation as described in the article, but it is still possible for the kernel to swap memory in and out of disk even without a swap file/partition. Memory used for storing executable binaries is allowed to be moved out of memory, as a copy of it lives on disk. This means you can still encounter memory thrashing (and thus system unresponsiveness) under low memory situations.
    • Denvercoder9 6y ago
      > Memory used for storing executable binaries is allowed to be moved out of memory, as a copy of it lives on disk. On Linux, this is not necessarily the case, as you can change the file on disk while the executable is running. I don't know if Linux just keeps executable code in memory all the time, or if it is smart enough to detect whether a copy of executable pages still lives on disk.
      • unilynx 6y ago
        You should get a "text file busy" error if you try that. What you can do is delete and then recreate the executable. Then the deleted data simply sticks around on disk until it's no longer referenced
• AlisdairO 6y ago
  One other option is increasing the reserved block count ( https://ma.ttias.be/change-reserved-blocks-ext3-ext4-filesystem-linux/ https://ma.ttias.be/change-reserved-blocks-ext3-ext4-filesys... ). This has the nice side effect of increasing the space available for critical daemons. If you haven't customised this, in a pinch you can still lower it down a bit to buy some time.
  • stonesweep 6y ago
    I suspect the blog author did not understand this (based on the content) - as a Linode user myself, I just had a look at one of my VMs and they install with the regular 5% reserved space (ext4/Debian).
    • birdyrooster 6y ago
      Funny because I have always tune2fs -m1 or tune2fs -m0 because the reserved space was never supposed to scale linearly with hard drive capacities and is not useful to userspace in anyway. Have never had any issues and been doing it for decades in commercial applications. In some cases, where you probably shouldn't be using ext3/4 anyways, we are talking about reclaiming TBs of reserved space. It's important to note that mkfs doesn't care if you are formatting the root partition or a data volume partition, it will still reserve space for the kernel.
      • stonesweep 6y ago
        Fun trivia: you can actually set per-device settings in /etc/mke2fs.conf with all sorts of alternate defaults. https://man7.org/linux/man-pages/man5/mke2fs.conf.5.html https://man7.org/linux/man-pages/man5/mke2fs.conf.5.html
        • anonymousiam 6y ago
          More fun trivia. The original(?) man page for tunefs had an "easter egg": https://unixhistory.livejournal.com/1808.html?nojs=1 https://unixhistory.livejournal.com/1808.html?nojs=1
• tomaszs 6y ago
  Interesting. I would try to notify myself when space is getting low. But I like the solution because of it's simplicity
• Something1234 6y ago
  I have an empty leader on my hard drive so that I can recover if I accidentally nuke the front of it with dd while making a live usb. So it's not a bad idea, and it's super effective so far it hasn't been tested, and hopefully I never will need to.
  • ttyprintk 6y ago
    A good reason to partition swap before /boot.
• tyingq 6y ago
  Careful how you create it. Several ways to create large files can make a sparse file, which I don't think removing will actually help later.
  • zepearl 6y ago
    I think that in the past I saw that when creating a file with e.g. ... dd if=/dev/zero of=deleteme.file bs=1M count=8196 ...the "free space" shown by "df" slowly decreased while the file was being created, but then once the operation completed that "free space" magically went back to its original value => the big existing file (full of "0"s) was basically not using any storage. Is this what you mean? I just tried to replicate this behaviour but, dammit, I cannot demonstrate that right now as the behaviour so far was the expected one (free storage decreasing when creating the file and sticking to that even after the completion of the operation). I strongly believe that that's what I saw in the past (when I was preallocating image files to then be used by KVM VMs), but now I'm wondering if I'm imagining things... :P EDIT: this happened when using ext4 and/or xfs (don't remember) without using any compression.
    • tyingq 6y ago
      dd will create sparse files if you use the seek option, like: dd if=/dev/zero of=a_sparse_file bs=1 count=0 seek=8G
      • bombcar 6y ago
        You have to beware if you're on a filesystem (such as ZFS) that has compression enabled. A file of all zeros compresses quite well, and may not get you the space you need when you remove it.
        • zepearl 6y ago
          Yep - but I think that I was using ext4 and/or xfs when I was seeing that happening... .
      • zepearl 6y ago
• terramex 6y ago
  Ah, the classic 'speed-up loop' approach: https://thedailywtf.com/articles/The-Speedup-Loop https://thedailywtf.com/articles/The-Speedup-Loop About the blogpost itself: The disk filled up, and that's one thing you don't want on a Linux server—or a Mac for that matter. When the disk is full nothing good happens. I had this happen few times on a Mac and every time I was shocked that if disk gets full you cannot even delete a file and the only option is to do a full system reboot. I was also unable to save any open file, even to external disk and suffered minor data loss every time due to that. What is the proper way of dealing with such issue on macOS? (or other systems, if they behave the same way)
  • TheAdamAndChe 6y ago
    I don't know with Mac, but this is why many Linux distros recommend putting /home is on a separate partition. If it fills, it won't lock up the whole system. Fun story with this. Ubuntu now has an experimental root-on-zfs feature. I installed it and started playing with some docker containers, trying to compile a certain version of pytorch. Suddenly, my computer crashed. Apparently, my root partition filled because docker installed everything on the same partition as my OS, crashing everything immediately.
  • ghostly_s 6y ago
    MacOS has given the user nagging "startup disk is almost full " prompts for as long as I can remember, yours doesn't?
    • dylan604 6y ago
      And users have been ignoring that message as long as MacOS has been giving them. Maybe even longer
      • sillysaurusx 6y ago
        I even hacked my MacOS to disable the message. Computers shouldn't nag their owners repeatedly, even if it's in their best interest, unless the computer is about to catch fire.
• gkarthik92 6y ago
  So far, my first stop to temporarily get more disk space was to reduce the size of the swapfile which on a lot of servers seems to be allotted >1x the requirement. Will be switching to this hack! Perfect illustration of the KISS principle (Keep it simple, stupid).
  • bluedino 6y ago
    Useful for people who still do 2x physical memory and you have a server with 64+ Gb
• rags2riches 6y ago
  Back in my early university days the disks always seemed to be full at inconvenient times on the shared Unix systems we used. Some students resorted to "reserving" disk space when available. Which of course made the overall situation even worse.
• SethTro 6y ago
  Same idea as this game development legend https://www.dodgycoder.net/2012/02/coding-tricks-of-game-developers.html https://www.dodgycoder.net/2012/02/coding-tricks-of-game-dev... > he had put aside those two megabytes of memory early in the development cycle. He knew from experience that it was always impossible to cut content down to memory budgets, and that many projects had come close to failing because of it. So now, as a regular practice, he always put aside a nice block of memory to free up when it's really needed.
  • 295310e0 6y ago
    If true, I hate that story. Think of the better art assets that were needlessly left behind. How is it that said block of memory had never been identified by any profiling?
    • emmab 6y ago
      If it would be detected by profiling that does make the technique asymmetric in that it would only stick around if nobody profiled to find it.
      • hinkley 6y ago
        Or if you didn’t have an understanding with the sort of people who would run the profiler...
    • usefulcat 6y ago
      > Think of the better art assets that were needlessly left behind. Consider how long it takes to edit or recreate art assets to reduce their size. Depending on the asset, you might be basically starting over from scratch. Rewriting code to reduce its size is likely to be an even worse option, introducing new bugs and possibly running slower to boot. At least smaller, simpler art assets are likely to render faster. This is also the kind of problem that's more likely to occur later in the schedule, when time is even more scarce. Between these two factors (lack of time and amount of effort required to get art assets which are both decent looking and smaller), I think in practice you're actually more likely to get better quality art assets by having an artificially reduced memory budget from the outset.
• gfody 6y ago
  having an 8gb file you know you can delete isn't really all that helpful if everything has already gone disk-full-fracked. you should really have an alarm on free space, especially if you're an indie.
  • kelnos 6y ago
    Sure, but sometimes the disk filling up is caused by something runaway and fast. If your "60% full" alarm goes off and the disk fills up 2 minutes later, you're still stuck. With a "ballast file" (as another commenter termed it), you can decide exactly when processes get to start consuming disk again, and that can give you some headroom to fix the problem.
• liaukovv 6y ago
  I'm sorry for meta comment, but this font is barely readable.
  • nielsbot 6y ago
    It's pretty light. Use Reader mode?
  • davidcollantes 6y ago
    Not just the font of choice, but the formatting (hanging indent?). It makes it harder to read when there is little volume of text.
    • patrickserrano 6y ago
      Glad I wasn't the only one that found the indented paragraphs made it difficult to read.
• nycdotnet 6y ago
  This is an old trick for when you need to deploy to media with a fixed size - floppy/CD-ROM/etc. Make a file that is 5-10% the size of your media and don't remove unless you're running out of space in crunch time.
• h4waii 6y ago
  This is like carrying around a pound of beef because you refuse to look up the address of a McDonald's 7 minutes away. Setup quotas or implement some damn monitoring -- if you're not monitoring something as simple and critical as disk usage, what else are you not monitoring?
  • bostonsre 6y ago
    Not all environments require a stringent SLA. I have some servers that don't have a stringent SLA and aren't worth being woken up at night over if their disk is filling up fast.
    • luckylion 6y ago
      Okay, so you wake up with a full disk. What did the spacer accomplish?
      • bostonsre 6y ago
        It allows me to remove that big file, then I'm able to run sudo since I don't allow root ssh and sudo won't work with a full disk, then I can clear up space on the system, bring it up again, then update log rotate or do whatever to prevent that case from happening again.
        • luckylion 6y ago
          That sounds a lot more complicated (and time consuming) than just having monitoring in place, realizing the disk is filling up and fixing it before it leads to downtime.
          • bostonsre 6y ago
            Monitoring is in place and usually it is caught in time. Downtime is acceptable in this environment, I don't think its worth being woken up in the middle of the night when it can just be resolved in the morning.
• moistbar 6y ago
  When I worked at SevOne, we had 10x500 MB files on each disk that were called ballast files. They served the same purpose, but there were a couple nice tools built in to make sure they got repopulated when disk space was under control, plus alerting you whenever one got "blown." IIRC it could also blow ballast automatically in later versions, but I don't remember it being turned on by default.
• njacobs5074 6y ago
  As hacks go, it's a good one. I also like it because you don't have to be root to implement it and you don't have to reconfigure your file system params in ways that might or might not be great for other reasons.
• fractal618 6y ago
  I think i know where this is going without even reading. Any attempt from the outside to pull this 8b Gb file would be a very noticeable red flag.
  • outworlder 6y ago
    Nope. Try again :)
    • qwertox 6y ago
      But the idea isn't that bad. Name it properly, like saudi-arabia-customer-data.sql.pgp next to a directory named pgp-keys and fill it with /dev/random.
• clipradiowallet 6y ago
  An alternative approach here... make sure (all) your filesystems are on top of LVM. This reduces the steps needed to grow your free space. Whether you have a 8gb empty file laying around, or an 8gb block device to attach...LVM will happily take them both as pv's, add them to your vg's, and finally expand your lv's. some reading if LVM is new and you want to know more: https://opensource.com/business/16/9/linux-users-guide-lvm https://opensource.com/business/16/9/linux-users-guide-lvm edit to add: pv=physical volume, vg=volume group, lv=logical volume
  • uniformlyrandom 6y ago
    I would not say this is an alternative, more like yet another tool in a shed: 1. Tunefs 2. spacer.8gb 3. lvm
  • ttyprintk 6y ago
    Added benefit of not waiting to backup and restore 8 GB.
  • rubiquity 6y ago
    Yes LVM can help here. Another approach would be when you create the logical volume to intentionally under allocate. Perhaps only use 80-90% of the physical volume.
  • jonhermansen 6y ago
    If you are using LVM on all of your filesystems, it seems like a bad idea to use a file residing on LVM block device as another PV. And actually I'd be surprised if this was even allowed. Though maybe it is difficult to detect. You'd effectively send all block changes through LVM twice (once through the file, then through the underlying block device(s))
    • labawi 6y ago
      LVM is just fancy orchestration for the device-mapper subsystem with some headers for setup information. For block operations it's no different from manual setup of loop-mounted volumes, that also need to travel a couple of layers to hit the backing device. Though there is an important caveat - LVM is more abstracted, making it easier to mistakenly map a drive onto itself, which may create a spectacular failure (haven't tried).
• midasuni 6y ago
  I do similar, I keep multiple files though - 4GB, 2GB, 1GB and 100M, which I also use for testing speed
• diego 6y ago
  This points to a much more serious problem. This is 2021 and the technology is from the 90s, with a really poor user experience design. Your car warns you when you're low on fuel, but your server doesn't if you're low on critical resources.
  • mtone 6y ago
    Assuming we're talking about VMs (2021 etc.), for a SME is there any downside to giving 2TB of space to your discs and let dynamic allocation do the work? Perhaps consolidate/defrag once a year. Even monitoring total usage more often than that is probably not worth the effort - just buy ample cheap storage. Also, there was a tradition to split drives into OS, DB, DB Logs. That was mostly a rust performance thing and these days is probably just voluntary management overhead. RAM is another story.
    • qw3rty01 6y ago
      One VM using excessively more disk space than it's supposed to can potentially cause data corruption in all the other VMs on that system. For just spinning VMs up and down for testing, you probably won't run into that issue, but on a production system, it could potentially cause some massive downtime
      • cure 6y ago
        Virtual machine disk space (e.g. Xen, Linode, AWS EC2, or similar) does not work this way. Each VM gets a dedicated amount of disk space allocated to it, they don't all share a pool of free space.
        • jodrellblank 6y ago
          Yes they do with the "dyanmic allocation" the parent comment mentions; VMware datastore has 1TB total, you put VMs in with dynamically expanding disks they are sharing the same 1TB of free space and will fill it if they all want their max space at the same time and you've overprovisioned their max space. And if you haven't overprovisioned their max space, you may as well not be using dynamic allocation and use fixed size disks. Even then, snapshots will grow forever and fill the space, and then you hope you have a "spacer.img" file you can delete from the datastore, because you can't remove snapshots when the disk is full and you're stuck. It's the same problem, at a lower level.
• mgarfias 6y ago
  Why not keep an eye on the disk and expand the fs before it goes south?
  • jbverschoor 6y ago
    Because sometimes it can fill up quite quickly. And the extra space will give to the headroom you need. Will def do this on all my servers
    • qwertox 6y ago
      Yes. I like the idea very much and also think that I will do this with a couple of my machines.
• kchoudhu 6y ago
  This is why I insist on data and root partitions on all the machines I administer. Go ahead and kill the data partition, at least the root partition will keep the system up and running.
• gherkinnn 6y ago
  An architect once told me that he always plans for a solid gold block hidden away in the cellar. Once the project invariably goes over budget, he drops the plans for the gold and frees up extra funds. Edit: I think it was a large marble slab. Same thing.
  • emmab 6y ago
    What?!? How does that work? Does he just draw up a blueprint and write "solid gold block goes here" and them some contractor says "yes that gold block will be $NNNNN" and includes it in the budget??
    • marshmallow_12 6y ago
      Check your basement!
    • viraptor 6y ago
      It's likely not literal. He likely quotes for price +50k or something like that, so that people will start thinking about reducing the price before they run out of budget.
• scottlamb 6y ago
  > On Linux servers it can be incredibly difficult for any process to succeed if the disk is full. Copy commands and even deletions can fail or take forever as memory tries to swap to a full disk and there's very little you can do to free up large chunks of space. This reasoning doesn't make sense. On Linux, swap is preallocated. This is true regardless of whether you're using a swap partition or a swap file. See man swapon(8): > The swap file implementation in the kernel expects to be able to write to the file directly, without the assistance of the filesystem. This is a problem on files with holes or on copy-on-write files on filesystems like Btrfs. > Commands like cp(1) or truncate(1) create files with holes. These files will be rejected by swapon. I just verified on Linux 5.8.0-48-generic (Ubuntu 20.10) / ext4 that trying to swapon a sparse file fails with "skipping - it appears to have holes". Now, swap is horribly slow, particularly on spinning rust rather than SSD. I run my systems without any swap for that reason. But swapping shouldn't fail on a full filesystem, unless you're trying to create & swapon a new swapfile after the filesystem is filled.
  • bostonsre 6y ago
    Not sure about their reasoning.. but if you don't have root ssh enabled, sudo can break if there is no free disk space. I do something similar where I write a 500mb file to /tmp and chmod 777 it so anyone can free it up without needing sudo.
    • franga2000 6y ago
      I've experienced far more full disks than I'd want to admit, on many different hardware and software configurations, and I've never seen sudo break. Is this something you've experienced recently? I definitely agree with your advice and will go double check all my servers if /filler is 777 (not in /tmp since it's sometimes mounted tmpfs), but if sudo does break in that situation, that sounds like a pretty severe and most likely fixable bug.
      • bostonsre 6y ago
        Yea, have seen it recently on centos 7. I think it's due to having sudo logging enabled. It won't let you run sudo without it working (at least I think that's the case, but haven't spent the time to investigate too thoroughly).
• andimm 6y ago
  ot: the first two links are "swapped"
  • sonicrocketman 6y ago
    Thanks for pointing this out. Fixed.
• michelb 6y ago
  Showing off that i’m not a sysadmin, but wouldn’t a monitoring daemon work? Once disk usage grows past a certain uncomfortable threshold you get an email/notification to see what’s up. I mean you obviously are monitoring other server vitals anyway right?
  • willbutler 6y ago
    Monitoring is a good idea, regardless. However, there are cases where a bug or some other issue can cause disk usage to ramp too quickly for someone to respond to an alert.
  • jodrellblank 6y ago
    Cases mentioned below where space fills up quickly due to a bug, maybe yes. Outside that there's the problem that you can ignore the emails (or be sick, asleep, etc). Worse if they go to a team and everyone is busy and assumes someone else will deal. Bad if you aren't in charge and tell people in charge and they nod and don't decide anything - they prefer to run at 70/80/90/95% used indefinitely instead of signing a cheque. When the drive fills and everything breaks and, you /have/ to respond, and it becomes socially ok to make it your highest priority and drop everything. An email "only a few weeks until maybe it runs out of space" is much harder to prioritise and get anyone to care. With this system, the time when it fills and breaks has some flex for you to not go down with the server, and save your own bacon. It's as much a fix for the organization as anything else. I see this most in smaller company aging systems where they had ample storage and drives for a few years ago when they were new, now they're crammed full with the growth of software, libraries, updates, data, new services being deployed on them, increased demands, and nobody wants to commit to more storage for an older system towards the end of its warranties, but they definitely don't want to commit to the much larger cost of a replacement and all the project work, and running at 90% full costs nothing and involves no decisions. 91%. 92%.
• IncRnd 6y ago
  This solution is what creates the problem. If you want warnings when the free disk space is low, set-up warnings for when the free disk space is low.
• rgj 6y ago
  In the mid nineties I worked in a research institute. There was a large shared Novell drive which was always on the verge of full. Almost every day we were asked to clean up our files as much as possible. There were no disc quota for some reason. One day I was working with my colleague and when the fileserver was full he went to a project folder and removed a file called balloon.txt which immediately freed up a few percent of disk space. Turned out that we had a number of people who, as soon as the disk had some free space, created large files in order to reserve that free space for themself. About half the capacity of the fileserver was taken up by balloon.txt files.
  • gowld 6y ago
    You had a community-driven quota system.
    • manifoldgeo 6y ago
      Sounds like they had a tragedy of the commons, too haha
      • Dylan16807 6y ago
        Partly that, partly the opposite. It's basically reserving part of the disk for very important things only, which scares off less important uses. Like making the commons seem more polluted than it actually is to get some action taken. If those files weren't there, the space would probably fill up, but now without any emergency relief valves. It would be better if these files were a smaller fraction of space and had more oversight... but that's just a quota system. This is something halfway in between real quotas and full-on tragedy of the commons.
      • agumonkey 6y ago
        is there a model to solve these ?
        • chihuahua 6y ago
          I am far from an expert on game theory, but it seems that the cause of the tragedy of the commons is that people can use the shared resource for free. If there was a price to be paid, and the price was dynamically adjusted depending on conditions, then the overuse could be avoided. Similarly for file storage and "reserving" it by creating huge but useless files. If everyone was charged a fee per gigabyte per day, then people would be less likely to create those placeholder files. You probably have to be careful about how you measure, otherwise you'll get automated processes that delete the placeholder files at 11:59pm and create them at 12:01am.
• danielrhodes 6y ago
  My understanding is this is why one should partition a drive. If you have a data partition, a swap partition, and an OS partition, you can get around issues where a server’s lack of disk space hoses the whole system.
  • yabones 6y ago
    100% agree. I think at the bare minimum every system should have two partitions: `/` and `/var` /var is usually where the most data gets added. Logs, database files, caches, and whatever other junk your app spits out. 99% of the time that's what causes the out of space issues. By keeping that separate from root, you've saved your system from being completely hosed when it fills up (which it will). Obviously there are other places that should get their own mounts, like /home and /usr, but before you know it you've got an OpenBSD install on your hands with 15 partitions :)
    • nijave 6y ago
      I place I used to work achieved something similar with lvm thin provisioning and split out something like /, /home, /var, /var/log and maybe a couple others. I think they also had something clever with lvm snapshots to rollback bad updates (snapshot system, upgrade, verify) so even if an update went rogue and deleted some important, unrelated files it could be undid
    • ClumsyPilot 6y ago
      Sounds like a poor-mans quota system
    • sedachv 6y ago
      OpenBSD default partition allocation is really well thought out: https://man.openbsd.org/disklabel#AUTOMATIC_DISK_ALLOCATION https://man.openbsd.org/disklabel#AUTOMATIC_DISK_ALLOCATION At least put /tmp in its own partition as well. Multiple partitions can also save a lot of recovery time when there is a multiple bad sector event that corrupts an entire partition beyond recovery.
    • kortex 6y ago
• capableweb 6y ago
  I've had my disks so full that a `rm` command doesn't even work, would this workaround work in those cases too?
  • geocrasher 6y ago
    Yes because you could just do > save_my_butt.img and now it's 0 bytes.
    • kiwijamo 6y ago
      Would that work? The fs may actually allocate a new file before deleting the exisiting allocation so the risk of it not working is still there I would think?
      • quesera 6y ago
        It might vary by kernel, filesystem, or shell, but in my experience and confirmed with a quick test: shell redirection does not create a new file/inode.
      • geocrasher 6y ago
        Worst case scenario, if > filename didn't null it, then just echo "0" > filename
• ohazi 6y ago
  Ah yes... good ole' in_case_of_fire_break_glass.bin
• pritambarhate 6y ago
  All my servers have an alarm when disk space goes above 70%. It sends an email every hour once the disk usage goes above 70%. Never had a server go down because of disk space issue after adopting this practise. Also one of the main reasons server disks go full is generally log files. Always remember to "logrotate" your log files and you will not have this issue that much. Yes one more thing, for all user uploaded files use external storage like NFS or S3.
  • ghostly_s 6y ago
    Is there a package I can install to set this up?
    • Moto7451 6y ago
      Icinga is a common solution for monitoring FS and other use metrics. I imagine his setup, if custom rolled, is a shell script checking df and sending an email when the usage is at or above 70
    • epakai 6y ago
      I see Chris Siebenmann (https://utcc.utoronto.ca/~cks/space/blog/ https://utcc.utoronto.ca/~cks/space/blog/) blog about Prometheus (https://prometheus.io/ https://prometheus.io/) often. It seems quite capable of this and a lot more complex alerts.
  • bonestamp2 6y ago
    > for all user uploaded files use external storage like NFS or S3 We send our log files to S3 too. I mean, we write them locally (EC2) and then push them to S3 every minute. Then we have a tool that will let us search the log files in S3 and it will parse these rotated log files and join together the relevant pieces depending on what we're looking for (or all of it for a specific time period if we don't know what we're looking for). This is great because if the server goes down and we can't access it, or the instance is gone, we can still see log files from shortly before the problem occured. We also use bugsnag, etc for real time logging and tracking where possible.
  • 6y ago
• johbjo 6y ago
  No. Careful partitioning is the solution to this problem. Monitor the growth of your partitions and make sure nothing on rootfs or other sensitive partitions grow significantly.
  • slaymaker1907 6y ago
    I don't think this has to be an either/or scenario. Having some bloat you can get rid of quickly is a nice backup in case your monitoring fails for whatever reason.
  • tremon 6y ago
    Or the modern equivalent, subvolume quotas.
    • Cacti 6y ago
      Partitions do far more than enforce quotas.
      • tremon 6y ago
        They also hurt far more than dynamic subvolume allocations, due to their static nature. You still can't repartition an active disk without downtime under Linux FAFAIK, it requires a reboot or unmounting all other partitions on that disk, even for partitions that didn't change. I'll take lvm/zfs/btrfs subvolumes over static partitions any day.
• pgray 6y ago
  tune2fs -m not good enough?
• k__ 6y ago
  I remember a discussion here about a dude who did this with memory in game development. People didn't like the idea very much. To me it has a taste of domain squatting or GPU scalping, but you don't do it with strangers, but your team.
• fggg444 6y ago
  it feels like setting your watch 5 minutes fast, it's not a real solution
  • dylan604 6y ago
    can you do this on smartwatches? I know someone that went to full extreme of hour push, but they did this by setting their system time to the next time zone over.
    • K7PJP 6y ago
      Apple Watch used to offer this ability, but it no seems to be available.
• tshaddox 6y ago
  How is this better than sounding alarms when free disk space drops below 8GB? If you’re going to ignore the alarms, then you’re going to have the same problem after you remove your spacer file and the disk fills up again!
  • frenchie14 6y ago
    This would work if you have sufficient time between alarm and failure. If some issue or process uses up all of your available disk space in a short time span, you won't have that luxury. Hopefully, the author is using alerts on top of having this failsafe
  • _wldu 6y ago
    It requires far less configuration.
    • tshaddox 6y ago
      How so? You'll presumably need to configure some way to be notified when the disk space is full anyway.
  • zokier 6y ago
    Compare the rate an haywire process can fill up disk to your response time to alarms, and you got your answer right there.
    • luckylion 6y ago
      Okay, so now you have a disk full, only become aware of it when it's full and your database throws errors. You have an easy way to fix it, just delete the spacer file. But what good does that do? You're still in the mess where your database is really unhappy. On the other hand, if your monitoring was set up well, you got a notification and had time to react to it before it was at 100%. Granted, if you have a process that just wrote a file at maximum speed, that time window is tiny, but that's not usually what happens in my experience. What happens is that something starts logging more and it slowly builds up while you're happy that your server is running so well that you don't need to pay attention. And then the alert comes and tells you that there's less than 10% space available, and you have plenty of time to investigate and avert the crisis.
• monksy 6y ago
  To prevent the root fs block from filling up. That's why I always partition my home+var+opt partition away from the root partition.
• harperlee 6y ago
  At work, OneDrive does not sync by policy if there is less than 30Gb free space. Apparently for ensuring space for updates when they come...
• raldi 6y ago
  This reminds me of Perl’s esoteric $^M variable. You assign it some giant string, and in an out-of-memory condition, the value is cleared to free up some emergency space for graceful shutdown. “To discourage casual use of this advanced feature, there is no English long name for this variable.” But the language-build flag to enable it has a great name: -DPERL_EMERGENCY_SBRK, obviously inspired by emergency brake.
  • amock 6y ago
    I think it likely relates to https://en.m.wikipedia.org/wiki/Sbrk https://en.m.wikipedia.org/wiki/Sbrk.
    • raldi 6y ago
      Yes, of course. That’s what makes the pun work.
  • deleted 6y ago
    [deleted]
  • Wibjarm 6y ago
    I'd expect the name is also inspired by the sbrk(2) system call, so you can allocate some memory "for emergency use" if needed.
• londons_explore 6y ago
  Linux has this built in... By default, only root can use the last 5% of disk space. That means you can fire up a root shell and know you have a buffer of free space to resolve the issue.
  • mamon 6y ago
    But now we have Docker, which means all the containerized workflows will run as root....
    • londons_explore 6y ago
      I suspect you need to be root in the root user namespace... So docker doesn't get this special power...
      • cmeacham98 6y ago
        Docker does not use user namespaces by default (and some features are unavailable when using them).
  • cmeacham98 6y ago
    I believe this is an ext{2,3,4} feature. Unsure if it exists on btrfs, zfs, etc.
• kaydub 6y ago
  Stories like this and my own past memories make me so happy to work somewhere big.
• aidenn0 6y ago
  This won't work with ZFS, as it may be impossible to delete a file on ZFS when disk is full. The equivalent in ZFS is to create an empty dataset with reserved space.
  • throwaway525142 6y ago
    For me, it was possible to truncate -s 0 a large file on a full disk with ZFS.
  • hikarudo 6y ago
    Thanks, I've been wondering about the "proper" way of doing this in ZFS and this method hadn't come up in my searches.
  • davemtl 6y ago
    A way to prevent this is to create a dataset and reserve n amount of space, typically 10-20% and set it read-only.. before the pool gets full. Then when the pool fills up, you can reduce the reservation to be able to clean up files.
• geocrasher 6y ago
  For everyone saying "This isn't a real solution!" I'd like to explain why I think you're wrong. 1) It's not intended to be a Real Solution(tm). It's intended to buy the admin some time to solve the Real Issue. 2) Having a failsafe on standby such as this will save an admin's butt when it's 2am and PagerDuty won't shut up, and you're just awake enough to apply a temp fix and work on it in the morning. 3) Because "FIX IT NOW OR ELSE" is a thing. Okay, sure. Null the file and then fill it with 7GB. Problem solved, for now. Everybody is happy and now I can work on the Real Problem: Bob won't stop hoarding spam. That is all.
  • luckylion 6y ago
    > 1) It's not intended to be a Real Solution(tm). It's intended to buy the admin some time to solve the Real Issue. If you don't have monitoring, will you even be aware that your disk is filling up? If you do have monitoring, why are you artificially filling up your disk so that it will be at 100% more quickly instead of just setting your monitoring up to alert you when it's at $whateverItWasSetToMinusEightGB?
    • deleted 6y ago
      [deleted]
    • smarx007 6y ago
      An extra failsafe? You can do both. What if your cron/netdata are not forwarding emails for some reason (eg nullmailer gets errors from Mailgun)?
      • luckylion 6y ago
        Right, but again, what good does the spacer file do if you're not aware that you're running low on disk space? That is: if your monitoring isn't working, how do you know that you need to quickly make room? And if your monitoring is working correctly, the spacer file really serves no purpose other than lowering the available disk space.
        • 6y ago
• legulere 6y ago
  Seems like a good example how private property and therefore capitalism can be seen as the same as a tragedy of the commons.
  • deleted 6y ago
    [deleted]
  • dmingod666 6y ago
    What?
    • michaelt 6y ago
      If it's morally incorrect to get to the license server early and snatch a roaming license you might not use to its fullest, is it not also wrong to access the property market early and snatch a bunch of land you might not use to its fullest?
      • jsmith99 6y ago
        The difference is that market mechanisms are meant to allocate items to those that value them the most (assumed to be the same as their willingness to pay). Claiming a licence by contrast has little cost.
        • Kaze404 6y ago
          I think a homeless person values an empty house way more than a real estate owner.
          • Cullinet 6y ago
            not by the way they seem to treat any property I've seen and I have experience of homelessness personally and the prejudices have rational foundation
            • 6y ago
• beervirus 6y ago
  > even deletions can fail or take forever > in a moment of full-disk crisis I can simply delete it and buy myself some critical time to debug and fix the problem Uhh...
• cwt137 6y ago
  In theory, this is a good idea, but doesn't protect you in all cases. I have had instances on a few of my application servers where an event happened that dumped GB's worth of log data to the log files in a matter of a couple of minutes and filled up the drive (Thanks fast SSDs!). If I employed the strategy in the article, it would have only bought me a couple of more minutes worth of time, if that!
• ineedasername 6y ago
  A great idea, but it still leaves the possibility for performance issues prior to an admin's ability to address is. Some like two 4gb blocks might work better: if you get within, say, 200mb of storage limits you remove the first one and trigger an email/text/whatever to the admin, that way they can address it before it goes further. It's an early warning and automated solution. Then, if the situation continues, the second 4gb block is also automatically removed with another message send to the admin. Nothing fails silently.
• znpy 6y ago
  That's a dumb idea? Iirc some filesystems allow you to reserve a percentage of blocks for this particular use case (recovery by root). Ext2/3 for sure, ext4 probably too. Not sure you can do that on linode on the rootfs, since the filesystem is mounted, tho.
  • joana035 6y ago
    tune2fs -m
• ryandrake 6y ago
  A lot of tips in this thread are about how to better alert when you get low on disk space, how to recover, etc. but I'd like to highlight the statement: "The disk filled up, and that's one thing you don't want on a Linux server—or a Mac for that matter. When the disk is full nothing good happens." As developers, we need to be better at handling edge cases like out of disk space, out of memory, pegged bandwidth and pegged CPU. We typically see the bug in our triage queue and think in our minds "Oh! out of disk space: Edge case. P3. Punt it to the backlog forever." This is how we get in this place where every tool in the toolbox simply stops working when there's zero disk space. Especially on today's mobile devices, running out of disk space is common. I know people who install apps, use them, then uninstall them when they're done, in order to save space, because their filesystem is choked with thousands of pictures and videos. It's not an edge case anymore, and should not be treated as such.
  • jandrese 6y ago
    It doesn't help that the base model of many phones had ridiculously undersized storage for so many years. "I have an unlimited data plan, I'll just store everything in the cloud." only to discover later that unlimited has an asterisk by it and a footnote that says "LOL it's still limited".
  • probably_wrong 6y ago
    I know when our server's /tmp directory is full because Bash's tab autocompletion stops working. /home still has space, though, so nothing truly breaks. Perhaps I should file a bug report about that.
  • anyfoo 6y ago
    A lot of measures are preventative, and kind of have to be. Consider the hypothetical scenario of being totally out of memory. I mean completely: not a single page free, all buffers and caches flushed, everything else taken up by data that cannot be evicted. So in result, you cannot spawn a process. You cannot do any filesystem operations that would end up in allocations. You can't even get new page tables. Hence things like Linux's OOM killer, which judiciously kills processes--not necessarily the ones you would like killed in such a situation. And again, a lot of preventative measures to not let it come that far. Our Turing Machines still want infinite tapes, in a way.
• da_big_ghey 6y ago
  Full disc problem in linux macine has been a problem in partialy solved in past many decades. We have had seperated partition /home, /tmp, /var, /usr in each its own partition. This is reduce problem if not completly removing. This is small desadvantage: there is reducion in fungability for a disc space.
• dmingod666 6y ago
  It sounds 'cool' and all for 1995.. but, what about one script that'll email you when the disk is at 80%?
  • ineedasername 6y ago
    So that if by the time you get the email the issue is at 97% you can immediately give yourself enough breathing room to figure things out with downtime or significantly degraded performance.
    • dmingod666 6y ago
      Sure, alerts + this sounds like an approach someone would take
  • dylan604 6y ago
    Besides immediately after first spin-up, when is a drive not at 80% capacity?
• deleted 6y ago
  [deleted]
• CrLf 6y ago
  This is why the invention of LVM was such a good idea even for simpler systems (where some people claimed it was useless overhead). In my old sysadmin days I never allocated a full disk. The "menace" of an almost full filesystem was usually enough to incentivize cleanups but, when necessity came, the volume could be easily expanded. I guess a big file is not a bad idea either.
• nijave 6y ago
  The real question... Why does Linux or at least the common filesystems get stuck so easily running out of disk space? Surely normal commands like `rm` should still function.
  • nemo1618 6y ago
    > Surely normal commands like `rm` should still function They do. In my experience, the only disruption to most terminal operations is that tab completion will fail with an error.
    • hobofan 6y ago
      They sometimes don't. The article even acknowledges this: > Copy commands and even deletions can fail I've had that happen too many times, so I don't know why would fill up my disk with a hacky spacer file, which surely can also fail to be deleted when the disk is already full.
  • npongratz 6y ago
    As recently as 2016 I experienced major problems using `rm` with an intentionally-filled btrfs (and current Linux kernel at the time), and per my notes, it was even mounted as `-o nodatacow`: # rm -f /mnt/data/zero.*.fill rm: cannot remove '/mnt/data/zero.1.fill': No space left on device
• johnchristopher 6y ago
  It's the google photos thumbnails db. /s
• AcerbicZero 6y ago
  In most VMware clusters that use resource pools extensively I've always maintained a small emergency CPU reservation on a pool that would never use it, just in case I had to free up some compute without warning.
• davidelettieri 6y ago
  I always thought that database files should be on a different drive from the os. If the db fills up the HD, the os is still running smoothly.
  • lazyweb 6y ago
    Yep, ideally you'd have seperate partitons for /var, /tmp, /home, root, any application/db data ..
• Saris 6y ago
  It's interesting to me that linux doesn't natively reserve a little space to allow basic commands like directory listing and file deletion to function even with a full disk. Because really the biggest problem when I've had a partition get full, is I sometimes can't even delete the offending log file.
  • myself248 6y ago
    It still boggles my mind that the act of deleting a file can fail because it requires space to perform the act. If y'ask me, that's a fundamental design flaw. Of course nobody asked me...
    • chungy 6y ago
      Depends entirely on the design of the file system. In copy-on-write file systems, it's a necessity: you need to at least allocate a new metadata block that doesn't record the existence of some file... and that's assuming you don't have snapshots keeping it allocated anyway. You can run into real trouble on btrfs if you fill it, it has no reserve space to protected from this scenario. ZFS at least reserves a fraction of the total space so that deletes are allowed to work even when the pool reaches 100% capacity.
    • jcims 6y ago
      No real good way to send messages on HN, i recalled a convo we had a while back about phone switches and thought you might enjoy this YouTube channel - https://www.youtube.com/watch?v=ngRb6mBB9HY https://www.youtube.com/watch?v=ngRb6mBB9HY Tons of recordings from back in the day.
• louwrentius 6y ago
  Please use LVM (Logical Volume Manager) if you really are afraid of filling up disks. If the disk would ever fill up: 1. Buy an additional virtual disk 2. Add the disk to the LVM volume group 3. Expand the Logical volume A really good primer on LVM: https://wiki.archlinux.org/index.php/LVM https://wiki.archlinux.org/index.php/LVM
  • amelius 6y ago
    Can you do all that while blocking the original request for space?
    • louwrentius 6y ago
      Yes you can. And all of this is on-line.
• SpaceInvader 6y ago
  To extend space in any filesystem in the root volume group on AIX you need space in /tmp. Years ago while working for some major bank I proposed to create such dummy file in /tmp exactly for the reason of extending filesystem. It saved us several times :)
• arthurmorgan 6y ago
  It's a really bad problem on iOS where a full disk won't allow you to delete anything and a reboot puts your phone in a boot loop.
• kristjansson 6y ago
  Lots of comments assailing this approach as a poor replacement for monitoring miss the point. Of course monitoring and proactive repair are preferable - but those are systems that can also fail! This is a low cost way to make failure of your first line of defense less painful to recover, and seems like a Good Idea for those managing bare-metal non-cattle systems.
• deeblering4 6y ago
  if you are on an ext filesystem, reducing the reserved percentage on the full filesystem can save the day. its more or less this same trick built in to the filesystem IIRC 5% is reserved when the filesystem os created, and if it gets full you can run: tune2fs -m 4 /dev/whatever which will instantly make 1% of the disk available. of course should be used sparingly and restored when finished
• benibela 6y ago
  I have a dual-boot laptop with windows and linux, and use the ntfs partition to share data between them Recently, I extracted a large archive with Linux on the ntfs, and the partition was full Then Windows did not start anymore Linux would only mount the partition as read-only, because it was marked dirty after the failed start. Finally I found a tool to reset the mark, and delete the files. Now Windows starts again, but my user account is broken. It always says "Your Start menu isn't working. We'll try to fix it the next time you sign in.", then I sign out, and it is still broken I had to make a new user account
  • MayeulC 6y ago
    > Finally I found a tool to reset the mark, and delete the files. fsck?
    • benibela 6y ago
      No, ntfsfix I do not think linux fsck works on ntfs
• cmckn 6y ago
  This really goes to show, there is more than one way to skin a cat. Yeah the guy could probably overhaul his entire approach to system administration, but also...this works. Well-placed hacks are maybe my favorite thing.
• robin21 6y ago
  This is a great idea. Hit this so many times.
• dominotw 6y ago
  sysadmin version of setting the clock 5 mins ahead?
• davidmoffatt 6y ago
  Dumb idea. Read the man page for tunefs. The file system has some thing called min free which does the same thing. However this does not interfer with wear leveling. Dummy data does.
  • 404mm 6y ago
    Not commenting on whether OPs idea is sound or not, however tunefs implies the now less and less used ext4 (many distros are switching to XFS or btrfs :-/). On another note, that limit applies to non-privileged processes only. Some crap running as root will just fill up the disk too.
• CodeBeater 6y ago
  The gastric balloon of linux servers
• Blikkentrekker 6y ago
  > On Linux servers it can be incredibly difficult for any process to succeed if the disk is full. Copy commands and even deletions can fail or take forever as memory tries to swap to a full disk I don't understand this. Swap is either a swap partition, or a specific swap file, all of which allocated in advance, so the fullness of the storage should have no bearing.
• anonymousisme 6y ago
  One thing that many Linux/Unix users do not know is that all commonly used filesystems have a "reserved" amount of space to which only "root" can write. The typical format (mkfs) default is to leave 5% of the disk reserved. The reserved space can be modified (by root) any time, and it can be specified as a block count or a percentage. As long as your application does not have root privileges, it will hit the wall when the free+reserved space runs out. Instead of the clumsy "spacer.img" solution, one could simply (temporarily) reduce the reserved space to quickly recover from a disk full condition.
  • reph2097 6y ago
    Of course the application is running as root, duh.
    • encryptluks2 6y ago
      And that would be another issue that should be addressed.
• tanseydavid 6y ago
  I am surprised WIN did not make the short list for OS-lost when disk space is almost gone. I have been there a couple of times and it is a land of crazy unpredictable behavior.
• TazeTSchnitzel 6y ago
  I am reminded of a tweet that suggested adding a sleep() call to your application that makes some part of it needlessly slow, so that you can give users a reason to upgrade when there's a security fix (it's 1 second faster now)!
  • Black101 6y ago
    Apple did something like that... https://www.npr.org/2020/11/18/936268845/apple-agrees-to-pay-113-million-to-settle-batterygate-case-over-iphone-slowdowns https://www.npr.org/2020/11/18/936268845/apple-agrees-to-pay...
    • TazeTSchnitzel 6y ago
      They did it so old phones wouldn't suddenly hard-shutdown at 30% battery. I appreciate that they did that, it's very annoying.
      • Black101 6y ago
        That's what Apple likes to say. Luckily they got at least a small fine.
• joana035 6y ago
  Mind you can also use tune2fs. It has an option "-m" that one can tune how much reserved space is dedicated to root user.
• rrauenza 6y ago
  I thought this was gonna be about the obscenely large sparse file /var/log/last. I really wish they would move it from a sparse memmap() file to a btree or something.
• zeta0134 6y ago
  If you happen to use ext as your default filesystem, check the output of tune2fs; it's possible your distro has conveniently defaulted some 2-5% of disk space as "reserved" for just such an occasion. As the root user, in a pinch, you can set that to 0% and immediately relieve filesystem pressure, buying you a little bit more time to troubleshoot whatever the real problem is that filled the disk in the first place.
• jedberg 6y ago
  Since the late 90s, this was always my solution: tune2fs -m 2 /dev/hda1 That sets the root reserve on a disk. It's space that only root can use, but also you can change it on the fly. So if you run out of userland space you can make it smaller, and if a process running as root fill your disk, well, you probably did something real bad anyway. :) But yeah, this is a pretty good hack.
• freeone3000 6y ago
  This sounds like you should, instead, use the "Filespace Reserved for Root" functionality of your filesystem, which exists specifically for this contingency. The default for ext3 is 5%.
• crabmusket 6y ago
  This reminds me of a similar story in a classic Gamasutra article[1] (the section is "The Programming Antihero", and I'd recommend the other pages of the article for a few good chuckles). Apocryphal or not, it makes for a good story. > I can see how sometimes, when you're up against the wall, having a bit of memory tucked away for a rainy day can really make a difference. Funny how time and experience changes everything. [1] https://www.gamasutra.com/view/feature/132500/dirty_coding_tricks.php?page=4 https://www.gamasutra.com/view/feature/132500/dirty_coding_t...
• alvarlagerlof 6y ago
  Sorry, but I cannot read this at all. Please increase the font thickness.
• reph2097 6y ago
  This is stupid. Just don't make your servers use up all space. That's why ext can reserve space for root, 5% by default.
• rafaelvasco 6y ago
  This is an old technique. For example, some game developers back in the early days used to put dummy files in the game data space, and code the entire game with less space so that if later more space was needed, it was just a matter of deleting the dummy files. In that context, it kinda forces you to be smarter about your game assets and code.
• mito88 6y ago
  also disable file compression on the 8gb file.
• fpoling 6y ago
  This reminded me of embedded Java project that I worked 20 years ago. The VM had only 10MB of RAM and properly dealing with out-of-memory exceptions was a must. The most effective strategy was to preallocate like 200K array. Then on any memory exception the code released that array and set a global flag. The flag was queried through out the code to aggressively minimize memory usage until it drops to tolerable limit. The preallocated buffer was essential. Without it typical result was recursive out-of-memory that eventually deadlocked/crashed the VM with no recovery.
• DeepYogurt 6y ago
  This is why reserved blocks in fileystems exist no?
• davesque 6y ago
  On the subject of "inverted" thinking like this, I recently added a test to a test suite that is intended to fail some day. The test will eventually fail when a bug (for which we developed a workaround and added the aforementioned test to confirm the fix) is fixed in one of our open source dependencies. When the test fails, we'll know to remove the workaround (and the test)!
• bluedino 6y ago
  It’s nice today to be able just grow a disk using whatever virtualization platform you’re on. Another 50GB? Click. Click. Click.
• aye01 6y ago
  as a sysadmin, i hate this... but I'm also thinking of using it.....
• john_moscow 6y ago
  I happen to run a couple of small servers myself and here's a better version of this approach. Create a cron job that will run a simple self-testing script once every few hours. My self-test does this: 1. Checks that all domains can be accessed via HTTP and HTTPS. If not, DNS might have died. 2. Checks that a few known CMS-generated pages contain some phrases they should contain. If not, SQL might have died. 3. Checks that the HTTPS certificate has enough runway left. If not, certbot might have died. 4. Sends a basic email message from my domain to a gmail account. Receives it via IMAP and sends a reply. Then, verifies the reply. This catches a whole bunch of mail-related issues. 5. Checks the free RAM and disk space. Updates an internal "dashboard" page and sends me an email if they are off. It only took a couple of hours to hack this together and I must say, I get a much better night time sleep ever since.
  • chime 6y ago
    This would be an awesome apt package that I could configure with a few options and install on any server.
    • john_moscow 6y ago
      And I would even love to maintain and support it if there was a culture of paid software on Linux. But because the status quo is that everything should be free and "if you publish it, it's your duty to support it", we're stuck reinventing our own wheels.
• indigodaddy 6y ago
  This isn’t the worst idea in the world. It’s just not a long term or future looking plan..
• grconner 6y ago
  Really good idea. After looking at the linked article about dd, I guess this wouldn't work as well if one was using a file system with compression. In that case maybe /dev/urandom would be better?
• lameiam 6y ago
  summary: people suck
• truetraveller 6y ago
  This is not the right solution. It's like setting your clock 5 minutes ahead, to trick yourself into thinking it's 9:00 am, when it's really 8:55 am. It doesn't work. The better solution is simple monitoring. Alert when limit is passed. Increase limit to 16gb disk space remaining if paranoid.
• KernelMC 6y ago
  I keep my databases on a separate file system from root, var, or anything system critical for this reason. Wouldn't you still have down time when you fill up the filesystem with the 8GB space waster in place? I might be missing something here, but this doesn't seem like a good solution.
• o-__-o 6y ago
  Tunefs -m Been doing this since the UFS days
• KernelMC 6y ago
  I keep my databases on a separate filesystem from root, var, or anything system critical for this reason. Even with the 8GB space waster, if you aren't on top of your disk usage you'd have down time when you fill up the filesystem containing the DB. I might be missing something here, but this does not seem like a good solution to this problem.
• hardwaresofton 6y ago
  Proof that the future is here, but just unevenly distributed -- we have technology for dynamic disk expansion, but implementation & integration just isn't present/slick enough to make it available to even tech-inclined hosting consumers just yet. Guess this is another place of differentiation that some of these platforms could offer.
• 867-5309 6y ago
  8GB seems like an arbitrary number, especially with no explanation or percentage of total. would be no good on e.g. a 10GB slice
• prewett 6y ago
  This reminds me of an old gamedev story that I have no idea how to find. The project was getting near to shipping, they had cut all the space they could cut, but they still needed another megabyte of space. After a week of this, the senior dev told the narrator to meet him in his office, and he closed the door. He opened one of the project files and deleted a 1 MB static array. "At the beginning of development I always reserve space for just this occasion," he said. Shortly afterwards he emerged from his office, announced that he had been able to find some extra space, and was lauded as a hero.
  • sbierwagen 6y ago
    https://www.gamasutra.com/view/feature/132500/dirty_coding_tricks.php?page=4 https://www.gamasutra.com/view/feature/132500/dirty_coding_t...
• noxer 6y ago
  Just because HN likes to bashes Windows. I tell you that Windows runs pretty much normal if the disk is full. Had that happen many times and intentionally did this for tests as well. Even disconnecting the disk technically doesn't break the OS. Because of the "Windows To Go" feature, the OS can detects this and pauses. (Note: Windows To Go is officially removed from current versions but the code that freezes is still there. However, whether that works with your hardware is basically a gambling... so yeah dont try at home/work.)
• herpderperator 6y ago
  The author seems to forget that ext-based filesystems keep 5% of disk space available for root at all times by default, known as "reserved blocks".[0] That means if a non-root user uses all of the available space, it wasn't really all of the space -- root still has access to 5% free space within that partition. That's exactly the same as the useless 8GB file but in an officially-supported manner. If you run out of disk space, you actually have 5% left for root. So log in as root and fix the issue. Simple. Also: > On Linux servers it can be incredibly difficult for any process to succeed if the disk is full. Copy commands and even deletions can fail or take forever as memory tries to swap to a full disk and there's very little you can do to free up large chunks of space. Why would memory swap to disk when the disk is full? I feel like the author is conflating potential memory pressure issues with disk issues. How many serious production-grade servers even use swap, which usually just causes everything to grind to a halt if memory becomes full? [0] https://ma.ttias.be/change-reserved-blocks-ext3-ext4-filesystem-linux/ https://ma.ttias.be/change-reserved-blocks-ext3-ext4-filesys...
  • deleted 6y ago
    [deleted]
  • deleted 6y ago
    [deleted]
  • cellularmitosis 6y ago
    > So log in as root and fix the issue. Simple. "just fix the issue" often takes longer than 'rm spacer.img', which allows non-root processes to begin working immediately. In practice, the 5% root reserve is not really an effective substitute for OP's technique.
    • herpderperator 6y ago
      Running `tune2fs -m 0 /dev/sdaX` as root instantly makes the reserved blocks available. You could even just lower it by 1% which would be more than sufficient in the meantime: `tune2fs -m 4 /dev/sdaX`. When you've fixed the issue and freed up space, you can increase it to 5% again. So why would you bother making a blank file? :)
• rhdxmr 6y ago
  Simple but clever idea
• linuxhansl 6y ago
  ext4 has a default "Reserved block count" of 5%, only usable by root.
• GameOfKnowing 6y ago
  You should probably delete it.
• Diesel555 6y ago
  >The disk filled up, and that's one thing you don't want on a Linux server—or a Mac for that matter. When the disk is full nothing good happens. I found a bug with time machine where it wouldn't delete local copies properly and filled my hard drive until I couldn't do anything. The OS slowly stopped working. At first I couldn't copy or save anything, then deleting files made more files. It was so bad that the `rm` command eventually wouldn't work from recovery or the local OS. I could do nothing. I had to format. It happened again and I learned to manually delete the time machine local snapshots, but it was crazy how hard it was to recover once it took all my storage. That bug is fixed now.
• sriku 6y ago
  On a Mac, I'm often puzzled why the OS says both "you're low on disk space" *and* "you have 23GB available" on thr same disk.
• 1970-01-01 6y ago
  Tell all your SecDevOps friends how this file can also pull double duty as a ransomware canary. https://blog.urbackup.org/371/ransomware-canary https://blog.urbackup.org/371/ransomware-canary https://support.huntress.io/article/136-ransomware-canaries-technical-details https://support.huntress.io/article/136-ransomware-canaries-...
  • hollerith 6y ago
    OK, but any human-written text file can serve as a ransomware canary because it is straightforward for code to distinguish between plaintext written by a human and encrypted text.
• ck2 6y ago
  Another trick you can use is to adjust the size of the FS tables, NFS4 can do this very quickly and free up space. However on a sketchy drive this is obviously not a wise move. Actually wait, NFS2/3/4 has reserved block counts you can free. # tune2fs -m 3 /dev/md2 # Setting reserved blocks percentage to 2%
• godber 6y ago
  All my disks have 5% space reserved for root by default. This reserve can be changed at will.
• layla5alive 6y ago
  Would be better to leave 8GB unpartitioned and then expand the partition. An 8GB file on an SSD is removing 8GB worth of blocks from being able to participate in wear leveling.
  • umanwizard 6y ago
    Expanding a mounted partition on a live server sounds dangerous. Does ext4 even allow it?
• rswail 6y ago
  mkfs has an option to reserve a %age or # of blocks/inodes for root of a file system. It's the file system equivalent of empty files. Usually when free space is exhausted, it's for non-root users. You get that same "time to fix stuff by deleting the file" by using tunefs to change that root reserved space to zero. Plus have /var/log on a separate file system and make sure that your log rotations are based on size as well as time.
• ussrlongbow 6y ago
  This approach is not unique to IT. Some companies are overstaffed for similar reasons
• pikseladam 6y ago
  Simple and beautiful. Thanks for sharing.
• trissylegs 6y ago
  I suppose if you have deduplication (eg ZFS) you'd have filter out that file or it'll probably get converted to a sparse file.
  • sillysaurusx 6y ago
    dd if=/dev/urandom would do the trick. Filling a file with zeros is as fast as filling it with random data.
• superjan 6y ago
  We write a data intensive desktop app, and when you are close to disk full, we reduce functionality so you can’t make the problem worse, or lose work because of the disk full situation. The thing is that we know that more than half of that user’s data is ours, so our data is often the cause.
• AYBABTME 6y ago
  Once upon a time, I wanted to cache large and expensive to pull files on many thousands of servers. Problem is the disk space on these servers was at premium and meant to be sold for customer use. The servers did have scratch space on small disks, but that was used by the OS. So I wrote an on-disk cache system that would monitor disk usage, and start to evict and shrink its disk space usage. It would take up to N gigabytes of disk (configurable) for the purpose of caching, and maintain an M gigabytes free-disk-space buffer. Say you had a 100 GiB total space on a partition, with 8 GiB used for cache with a 2 GiB headroom. As legitimate/regular (customer) space usage increased and reached 91 GiB, the cache would see 9 GiB available, and removing the 2 GiB buffer, would start to evict items to resize to 7 GiB, and so on until it had evicted everything. When this system deployed, it started to trigger low-disk-space alerts earlier than before. At first that seemed like a problem, but the outcome is that we were now getting low-disk-space alerts with more advance warning, and the cache bought some time as it kept resizing down to free up space. It kind of, in a way, served the same purpose as described in this blog post. Overall this cache was pretty neat and still is, I bet. There's probably ways to do similar things with fancy filesystems (or obscure features) but this was a quick thing to deploy across all servers without having to change any system setting or change the filesystem. I sometime wish I had done this in open-source, because it would be convenient to use locally on my laptop, or on many servers.
  • bombcar 6y ago
    Box Drive and OneDrive apps for desktop do this now, but with cached files from the "cloud". It looks to you like everything is there, but in reality it downloads as you click on things, and empties cache if the free space begins to drop below a set amount.
• larusso 6y ago
  I maintain a small fleet of CI machines (mostly macs) and run into this issue as well from time to time. The free space idea is nice but I ram into the problem that under very critical disk space I can’t even shh or when delete a file because there is simply not enough space to execute the simple command. A reboot to get rid of some temp files helps me in these situations to get some control back.
• t0mmel 6y ago
  What I don’t understand about this approach is why you think it actually does anything for you ? What you do instead of this is to setup an alert to monitor disk space at the right threshold for you, and then have a contingency plan for how to add more space to your environment. It seems like you have sort of done that, but in this case you are actually allowing your system to get into a bad state before you react. Perhaps it’s better to be proactive instead of reactive.
  • gitgud 6y ago
    This is true, but alerts and monitoring software can fail in many ways, due to network issues, process crashes... Etc. Due to the simplicity of this solution, there's not much that can go wrong.... unless you can't actually access the server anymore to delete the file... I see this more as a second line of defence
    • luckylion 6y ago
      If the monitoring fails to alert you, you don't get the alert that you're running out of disk space and you can't access the server to delete the spacer file. All you've done in that case is lower your disk space by 8gb and make your server fill up more quickly.
  • Kiro 6y ago
    Setting up alarms sounds complicated. I don't want to deal with that.
• jordinebot 6y ago
  This reminded me of that joke about two guys who meet in the middle of the Savanna. One is carrying a phone booth and the other one an anvil. So, the one with the anvil asks: - Why do you carry a phone booth around? - Oh, you see, it's for the lions. If I see a lion, I drop the booth, step inside and I'm safe. What's with the anvil? - It's for the lions too. If I see a lion, I drop the anvil and I can run way faster! Good trick anyway!
• nix23 6y ago
  Hmmm, Quotas anyone?
• igammarays 6y ago
  Does anyone know if disk resizing works from the Digital Ocean console when the disk is full? Kindof worried now.
• skytreader 6y ago
  Oh man, reminds me of a Game Dev war story I read years ago. This purportedly happened in those console days with very limited memory capabilities. In some game studio, as a project neared its release, the team was still struggling with memory issues. No matter what they did, they had a surplus of just about 2MB. The artists have reduced their polygon counts drastically, the programmers have checked every possible leak, have optimized algorithms and buffers the best they could but the 2MB surplus just kept haunting them. That's when the VP of Engineering stepped-in. Calling the TL of the project into a closed-doors optimization code-review, they had the source code on a large screen and the TL talked the VP through everything the team has done so far to stay within the memory budget. As the TL finished the walkthrough, the VP opens some mother-of-all files and deletes a cryptic variable declaration to the effect of: int toLiveBuffer[2000000]; The VP then explains that he hid this declaration in their codebase after a project that had to optimize drastically late into the development cycle. But first he wanted to make sure that the team did their homework. And poof. They emerge from the closed-doors meeting jubilant and victorious. The game is ready for prime time!
  • alfiedotwtf 6y ago
    ... sounds like a dailywtf post :)
• gautamsomani 6y ago
  fallocate is a better command then dd. Check it out. It doesn't even cause any disk IO. I use it all the time for such cases.
• mulle_nat 6y ago
  This may not work on a filesystem with compression. That empty file might only take up a block, even though its size is 8 GB.
• otikik 6y ago
  Reminds me of the cron task I set up once, long time ago, on a bare metal server. It would kill and relaunch a web service every 4 hours. The service in question didn't require high availability (it was a mailing list processing/interface thing, if I remember correctly) but it had some memory leak which would eventually devour all the memory in the server, in about 2 days. This hack served its purpose well, until the service was eventually replaced by something else.
• hitul-mistry 6y ago
  Why to have empty files? Can't you put disk size monitoring? Regards, Hitul Mistry digiqt.com
• pabs3 6y ago
  Hmm, I thought empty/sparse files didn't take up any space on disk.
• asdofindia 6y ago
  I just delete my docker logs when the disk is full and get a 5GB space :)
• pantalaimon 6y ago
  Doesn't ext[2,3,4] reserve 5% of the space on the disc for this very reason? This can be adjusted with tune2fs -m <percentage> /dev/sda1 You can check the reserved blocks with sudo tune2fs -l /dev/sda1 | grep 'Reserved block count'
• tammerk 6y ago
  I've seen in Cockroach db documents last month : https://www.cockroachlabs.com/docs/v20.2/cockroach-debug-ballast https://www.cockroachlabs.com/docs/v20.2/cockroach-debug-bal... This technique even has a name : ballast file
• glacials 6y ago
  On the chat team at Twitch in the early days after the Twitch Plays Pokemon crisis [1], we started artificially doubling all chat traffic through our systems, then dropping the doubles just before they would be sent to users. [2] Not only did it give us a "big red button" to press during emergencies like OP, but it revealed important logical scaling issues before they became real problems. [1]: tldr; 1 million people playing a single game of Pokemon Red by using chat to send button presses [2]: https://www.twitch.tv/videos/93572955 https://www.twitch.tv/videos/93572955
• sloucher 6y ago
  The great thing about Windows is that there's so much bloatware you'll always find something to delete when you need space.
• cafard 6y ago
  In the days of minicomputers, Data General's first 16-bit operating system, RDOS, required that the main file be "contiguous". Not only that, there was some model of disk they sold where the OS file had to be close to the edge for speed in loading. Prudent sysadms would create empty contiguous files in the favored space against the next upgrade.
• notorandit 6y ago
  I would keep 4x2gb. Just in case...
• ultrafez 6y ago
  A fun problem on a Mac is that if you're using APFS for your filesystem, if it fills up, you can't delete any files. It's caught me out a handful of times, and each time, the only way to recover is to reboot, and thankfully I've had more free disk space after a reboot. I'm not going to try to understand the logic as to why APFS requires free space in order to delete files (via any method, including dd)
  • wilsonthewhale 6y ago
    Probably because it's a log-structured file system, and those _really_ don't like running low on free space. They work by appending to the log then compacting sometime later, not modifying things in-place. As such, you always need a reasonable supply of free blocks so this can occur.
• erlkonig 6y ago
  For ext* filesystems, you can use tune2fs to change the reserved block percentage to accomplish this in what might - depending on your preferences - be a more graceful way. Basically it lets you knock 8 GB or more (although it's a percentage instead, 5% by default) off of the disk space available to non-root users. When it hits 100% and things start breaking, that reserve can be used by root to do compression safely, move things around, and so on. Alternatively the reserve percentage can be changed with a single command (by root), to allow non-root processes more space while the admin contemplates what do do next. One nice aspect of using the reserve instead of a file is that it prevents runs of "du" for including the file in their results. Another is that it's pretty much impossible to accidentally remove the reserve (or for some other admin to find it and decide it's superfluous). This is less effective at sites that have a lot of services running as root, in which case only your approach is fully effective. I want to say "But who does that nowadays...", but it happens. tune2fs apparently also supports allowing members of a certain unix group or user to have access instead of solely root. The core command for all this is: tune2fs -m <reserved-percent> <device> One other thing you might want to worry about: inode exhaustion. tune2fs has an inode reserve % as well - and trying to emulate this by creating a few hundred thousand files instead would be... inelegant.
• tilolebo 6y ago
  Or you could use "tune2fs -m" to temporarily reduce the percentage of reserved blocks
• bobbydreamer 6y ago
  I had used this technique in Dev and ist servers precisely 11 years back. Get storage would be a days task which would stall current activity. This helped. 1.5gb of 5 files.
• dalu 6y ago
  Just don't use cloud, get a cheap dedicated server and never run out of disk space
• arendtio 6y ago
  With virtual servers this should not be necessary, as it is easy enough to add some disk space. After all, this should not be a common issue in production environments, but more like a once in a decade problem. With physical servers it might be a different story and might be a good idea. I tend to size filesystems to the requirements I have and enlarge them when required (it gives you a periodic reminder to think about what waste you have accumulated). That way, I can still add space even if the filesystem has been filled up. However, if you do it just to have some space when you need it, it probably is overkill and to have an empty buffer file is a lot easier to handle.
• EricE 6y ago
  I've been doing this for years too. I also learned instead of one big file, halving several files is also useful so you can release space in "chunks". If you need everything at once you can wildcard the delete. Such flexibility has been invaluable over the years. Thankfully with block storage and modern operating systems/file systems growing volumes can be significantly easier for most servers.