3 ms·
A lot of tips in this thread are about how to better alert when you get low on disk space, how to recover, etc. but I'd like to highlight the statement: "The di
by ryandrake 6y ago
A lot of tips in this thread are about how to better alert when you get low on disk space, how to recover, etc. but I'd like to highlight the statement: "The disk filled up, and that's one thing you don't want on a Linux server—or a Mac for that matter. When the disk is full nothing good happens."
As developers, we need to be better at handling edge cases like out of disk space, out of memory, pegged bandwidth and pegged CPU. We typically see the bug in our triage queue and think in our minds "Oh! out of disk space: Edge case. P3. Punt it to the backlog forever." This is how we get in this place where every tool in the toolbox simply stops working when there's zero disk space.
Especially on today's mobile devices, running out of disk space is common. I know people who install apps, use them, then uninstall them when they're done, in order to save space, because their filesystem is choked with thousands of pictures and videos. It's not an edge case anymore, and should not be treated as such.
- jandrese 6y agoIt doesn't help that the base model of many phones had ridiculously undersized storage for so many years. "I have an unlimited data plan, I'll just store everything in the cloud." only to discover later that unlimited has an asterisk by it and a footnote that says "LOL it's still limited".
- probably_wrong 6y agoI know when our server's /tmp directory is full because Bash's tab autocompletion stops working. /home still has space, though, so nothing truly breaks. Perhaps I should file a bug report about that.
- anyfoo 6y agoA lot of measures are preventative, and kind of have to be. Consider the hypothetical scenario of being totally out of memory. I mean completely: not a single page free, all buffers and caches flushed, everything else taken up by data that cannot be evicted. So in result, you cannot spawn a process. You cannot do any filesystem operations that would end up in allocations. You can't even get new page tables. Hence things like Linux's OOM killer, which judiciously kills processes--not necessarily the ones you would like killed in such a situation. And again, a lot of preventative measures to not let it come that far. Our Turing Machines still want infinite tapes, in a way.
- gardaani 6y agoEarly Symbian apps are an excellent example how to write apps so that they don't crash when storage or memory becomes full. They just show an error dialog and the user can still use the system to free storage or memory. Modern phone apps either crash or the entire phone crashes in similar situations.
- apexalpha 6y agoI had this on my Ubuntu server... The NFS mount died for some reason and the downloading app wrote it all to the local filesystem, filling my SSD to the brink within minutes. By the time I ssh'd in the NFS had remounted, so it took ages to figure out where all that disk space actually was used since all dir scan tools would traverse into the NFS mount again. It felt like everything was falling apart. As soon as I deleted something another app filled it up in minutes. Even Bash Tab completion breaks... There really should be a 98% disk usage threshold in Linux so that you can at least use all system tools to try and fix it.
- herpderperator 6y ago> As developers, we need to be better at handling edge cases like out of disk space, out of memory, pegged bandwidth and pegged CPU In what situation though? Let's consider disk space. This certainly does not apply to all developers or all programs. Making your program understand the fact that the system has no space left does not seem like something that would be very productive in the vast majority of cases. Like running out of memory, it is not something the program can recover from all by itself unless it knows it created temporary files somewhere that it could go and delete. If that scenario does in fact apply to your program, then it's not even an edge case: the program should be deleting temporary files if it doesn't need them anymore. If the P3 was created to add support for that exact function, then I agree that it should be acted upon. A P3 is fine as long as it's reached. If you don't reach your P3s ever, then there are different issues that need addressing. I'd even say for something littering users' disks it should be higher than a P3, but the point is it's a specific case where it makes sense to handle that error. In every other case, your best bet is a _generic_ exception handler for write operations that will catch any failure and inform the user (e.g. "[Errno 28] No space left on device"), but that's something that should already be a habit. There are cases when you want to try to avoid running out of disk space because your program might know that it needs to consume a lot of it (e.g. installers) so it will be checked preemptively. Even then you probably do want to try to handle running out of disk space (e.g. in the unfortunate event that something else consumed the rest of your disk _after_ you preemptively calculated how much was required) so you can attempt a rollback and inform the user to try again. Other than that, when else is that _specific_ error more important than knowing that the data just couldn't be written in general? Let's say you have a camera app that tries to save an image. Surely you'd have a generic exception handler for not being able to save the image, rather than a specific handler for "out of space", which seems oddly specific considering there are literally hundreds of specific errnos you could be encountering that would prohibit you from writing. I'm sure the user doesn't want to see something like "Looks like you're out of disk space. Do you want to try save this image in lower quality instead?" So my point in all of this is I agree that we should _consider_ the impact of disk space but it doesn't need to be prioritized by developers unless it's actually important like in the first few examples I gave.
- scottlamb 6y ago