3 ms·
I see a lot of people asking "How could this possibly happen?" It turns out that there's a common failure mode that makes it past almost every unit test suite:
by sillysaurusx 3y ago
I see a lot of people asking "How could this possibly happen?" It turns out that there's a common failure mode that makes it past almost every unit test suite: a lack of disk space.
Running out of disk space pops up time and again. I've often wondered why software fails so often in this situation. But it's because the basic components of the system start to break -- things like "Call with temporary file" stop working, because creating a tempfile fails. And usually you're writing that code precisely in a spot where it assumes it can't fail.
This just happened on my PS5. I thought I'd play a bit of Armored Core 6 over Christmas, so I sat down and was greeted by "Can't start game." There was a pending update which couldn't be installed. So I skipped update; same error. But now the update was skipped, so it wouldn't even try to reinstall it. I ended up having to reinstall the whole game. Never seen a PS5 game completely brick itself like that.
Disk space woes are also one of the primary reasons websites go down. Ever forget about a server for two years and suddenly see your service break? It's almost always because the process ran out of file handles (resource leak) or because it ran out of disk space (hello imagemagick tempfiles).
Of course, the bug here could be completely different, but disk space errors are so common that it's as good a guess as any. Computers don't do well when you're out of disk space. MacOS freezes and bluescreens instead of going to sleep, because it can't save the hibernation file to disk. PS5 media recording stops working, because the hour-long recording backlog no longer has room to write the video to disk. This software update obviously passed their unit tests, but failed in this particular case; disk space is one of the few variables that they probably didn't test for.
I wish it were easier to write unit tests for all possible out-of-space situations. There are just so many. Even if it technically doesn't break, it often craps up the experience so much that the user ends up wondering why the software is so bad.
- deleted 3y ago[deleted]
- qingcharles 3y agoI had a rock solid app that I wrote die on me yesterday because it tried to access a folder and Windows returned "Drive is corrupt," which of course I hadn't handled because that will never happen. Out of drive space is the worst, though. Like you say, most things don't handle it, they just plough headlong into an install and then die, leaving the whole drive in a horrible unknown mess of files where you might or might not be able to retry the operation, and you might or might not have used up the last few bytes on the drive and now for some reason when you try to reboot it can't even boot the system up because it can't write some temp file it needs.
- somerandomqaguy 3y agoCould also be out of memory errors which are also tricky. Linux's OoM killer just picks something at random and terminates it. If you're lucky is something your app doesn't need but if you're unlucky and you're not paying close attention to syslogs, you're in for a fun time trying to figure out why your app just stopped for no logged reason.
- Szpadel 3y agoBtrfs have (had?) interesting failure mode for this scenario, because this is Copy On Write file system it need to allocate new block to modify metadata, when you are out of space, you cannot do that anymore. You might end up in scenario where your filesystem is full but you cannot delete anything because this required allocating new metadata block and your drive is full
- RecycledEle 3y agoOut of disk space errors is why we need larger disks. I get annoyed at people who put in the smallest possible disk, when a disk 4x larger would cost another $10.