13 ms·

A disk so full, it couldn't be restored

382 points by goranmoomin 3y ago

p0w3n3d 3y ago
Tbh any system cannot be full, because it will malfunction. Not sure about Windows (which warns that it might misbehave once the disk is fully full (￣ー￣)//”” ), but Linux will be unable to spawn processes once the /proc is full. I remember I had problem to login to my linux server, because sshd was unable to spawn bash
ycombiredd 3y ago
I see a few other comments trying to address this with a similar truncation shell command but then a few arguing replies about the redirection operator not working due to some syntax error. So here’s the trick there… preface the redirection operator with a null command, like so :> somefile Note the colon, which stands in for the command as a null operator. An additional hint is that if the system is so starved you can’t run ls, again call On the shell for help and use “echo *”, which will show a list of file names (without sizes) from which hopefully you can select a large file (or few) based on name alone. Colon-greater them into zero byte status and watch your system become useable enough to begin normal recovery efforts. From an old Unix admin; I hope that helps someone.
voidwtf 3y ago
It seems like Time Machine has been steadily declining. I'm not sure why there is no impetus to get it reliable and functioning well. Between sparse bundles becoming corrupt and having to start a new backup and failing functionality I haven't felt like Time Machine is worth setting up anymore. This is in stark contrast to the iOS/iPadOS backups which have worked every time.
- wlesieutre 3y ago
  I don’t know how many times I’ve had Time Machine decide it didn’t want to work anymore and I had to wipe the backups and start fresh to make it work again, but it’s a much larger number than it should be.
- ksec 3y ago
  >It seems like Time Machine has been steadily declining. Because they dont sell Time Capsule anymore. And they want you to backup everything to iCloud to grow their Services Revenue.
  - philistine 3y ago
    Extrapolate it one more step: Apple is clearly working on an iCloud backup for Macs, cause then that's more services revenue. While they're doing that, why would they fix bugs in Time Machine. People can't surely be using this old thing while we're working on the spiffy new thing!
    - firecall 3y ago
      The problem though is that it is likely to be complete environment backup and restore, like iOS. But we shall see! Hopefully they will provide the ability to backup and restore file versions!
      - bducycy 3y ago
        [dead]
  - firecall 3y ago
    Totally. But iCloud isn’t a backup. Its sync. And it will happily sync corrupt files and does not provide any versioning. The best TimeMachine is an SSD connected locally to the Mac. The second best is an SSD running on a Mac setup with TomeMachine Server. Then you are lucky if backups continue to work. And even luckier if you can sensibly restore anything via the hellscape that is the interstellar wormhole travel interface! ;-) BackBlaze is very reliable at least! Not cheap with a house full of computers to backup though :-/
    - realusername 3y ago
      > And it will happily sync corrupt files and does not provide any versioning. It does not even provide a basic progress bar when used on the phone.
- ryukoposting 3y ago
  As a non-Mac user, this sounds like a catastrophic and inexcusable bug the likes of which would inspire a dogpile of hatred against desktop operating systems with penguin mascots and/or headquarters in Washington.
  - folbec 3y ago
    That's the magic of genius marketing
  - ipv6ipv4 3y ago
    Time Machine works fine. Better than anything available on Windows. I’ve used it for more than a decade and a half, with multiple restores, across multiple machines. My current desktop Mac environment is a direct descendant of my original Mac from 2004 thanks, largely, to Time Machine.
- steve1977 3y ago
  The whole of desktop macOS has been steadily declining, so I‘m not surprised by this story in the slightest. But hey, we get new emojis and moving desktop wallpapers…
- fmajid 3y ago
  Mac OS quality control has been declining since they fired Scott Forstall, and it wasn’t amazing under his tenure to begin with.
- aequitas 3y ago
  I don't share this experience. I've been running Time Machine for years now on a Samba share for multiple Macs and if anything I've only seen an improvement. Years back I would regularly get a corrupted Time Machine sparse bundle that had to be recreated (or I would restore a previous ZFS snapshot and it would continue of off that), but it also ran over AFP back then I think, not SMB. Lately I've not have had any of these issues on any of the machines. I do have specific flags enabled in the smb.conf file that are recommended for Time Machine backups.
  - andruby 3y ago
    Have you restored from a backup during that Time? I have a similar setup (sambsa share on a ZFS NAS) and was appalled at how long it took. Both machines on wired gigabit ethernet, yet the restore took more than 24 hours. And that was for just a 1TB disk.
dannyw 3y ago
This happened to me too, without Time Machine. It’s devastating how rm on macOS can’t remove files when the disk is full.
Dalewyn 3y ago
What would the reasoning be for the file system failing this hard? Filling a partition to 99.999999% capacity always produces a nonsensical situation, but it's usually still recoverable without resorting to DBANing it first.
- epmos 3y ago
  I do not know the details of Apple's file system, but I wouldn't be surprised if it needs to allocate space for the log (journal) and can't do so. That isn't reasonable in the sense of "this is what the filesystem should do in this situation" but if the log and user data are allocated from the same pool it is quite possible to exhaust both.
- opello 3y ago
  I'm so curious about this too. I've done this with ext2 and ext3 in the past and truncating a large file solved the problem if rm wouldn't. It's been long enough I don't remember so specifically though, but certainly wasn't "dd to larger disk and grow the partition" to the rescue. If it was a journal issue would something akin to using an initramfs (or live environment) and mounting with data=writeback enable removing files? Or maybe APFS doesn't support that?
sneak 3y ago
Time Machine has been consistently unreliable the entirety of the time since it launched well over a decade ago. It should be common knowledge that it sucks. Use Backblaze if you don’t care about privacy, rsync+ssh to a selfhosted zfs box if you do.
- abhinavk 3y ago
  You can use backup tools like restic/borg/kopia to encrypt/compress before uploading to Backplaze or any other cloud service.
- jen729w 3y ago
  Backblaze supports encryption, as does the excellent Arq, which is a better solution than rolling your own rsync+ssh.
  - lh7777 3y ago
    Backblaze Personal does support encryption, but it's always been incomplete. If you supply your own encryption key, it's true that Backblaze can't read your data at rest. But to restore files, you have to send your key to Backblaze's server, which will then decrypt the data so that you can download it. They say that they never store the key and promptly delete the unencrypted files from the server, but to me this is still an unnecessary risk. There's no reason why they couldn't handle decryption locally on the client device, but they justify on-server decryption in the name of convenience -- you can restore files via the web without downloading an app. If you're concerned about this, the solution is to use B2 with a 3rd party app like Arq.
    - philistine 3y ago
      I actually use Arq to send my Time Machine backups and the rest of my NAS to S3 Glacier, in case the house burns down or the drives fail (whichever comes first). It works great and is very cheap!
      - pitaj 3y ago
bingo-bongo 3y ago
Probably needed to delete some of the mentioned disk snapshots, before trying to delete files. https://support.apple.com/en-gb/guide/disk-utility/dskuf82354dc/mac https://support.apple.com/en-gb/guide/disk-utility/dskuf8235...
jen729w 3y ago
I had this issue in October 2018 as documented in this Stack Overflow question, whose text I’ll paste below. I was lucky: I had an additional APFS partition that I could remove, thus freeing up disk space. Took me a while to figure out, during which time I was in a proper panic. --- https://apple.stackexchange.com/questions/338721/disk-full-terminal-in-recovery-mode-won-t-delete-files-boot-kernel-panics https://apple.stackexchange.com/questions/338721/disk-full-t... I’m in a pickle here. macOS Mojave, just updated the other day. I managed to fill my disk up while creating a .dmg, and the system froze. I rebooted. Kernel panic. Boot to Recovery mode. Mount the disk. Open Terminal. –bash–3.2# rm /path/to/large/file rm: /path/to/large/file: No space left on device Essentially the same issue as this Unix thread from ‘08! https://www.unix.com/linux/69889-unable-remove-file-using-rm-disk-space-full.html https://www.unix.com/linux/69889-unable-remove-file-using-rm... I’ve tried echo x > /path/to/large/file, no good. It’s borked. Does anyone have any suggestions that aren’t “wipe the drive and restore from your backup”?
- bombcar 3y ago
  Sounds like creating a sliver of an extra partition of a gig or so might be valuable insurance. Kind of like the old Unix file systems that would reserve 5% for root.
  - pixelfarmer 3y ago
    Since the SSD days (I started like 15 years ago with that) I keep a bit of space empty for two reasons: Emergency situations where you may need some extra space and to give SSDs a bit more room to relocate blocks to (they have a certain amount internally reserved already).
- bouke 3y ago
  With APFS this is not as straightforward though, as containers are only allocated when written too.
- djmips 3y ago
  I see in that stackexchange post, there's since been another alternative potential solution where you delete you virtual memory partition and if it's large enough it can give you back enough space to allow deleting of files to happen.
owyn 3y ago
Oh! What I would do is take the disk out, mount it on another machine and delete files then put it back........ /s I think the whole stack of operating systems and tools that assume that this is possible get in trouble when it's not possible. I don't want my computer to become a locked down sandbox but it seems like this is where we are headed.
- userbinator 3y ago
  If you mounted the disk on another machine, I suspect in this situation the filesystem was so full that you'd get the same error when trying to delete, like what the author here encountered when booting into Recovery OS. The only solution is to copy all the files you want to keep to another (not full) disk, then reformat and copy them back, or if you don't have another disk to copy to, somehow edit the disk directly to "manually" free some space.
  - owyn 3y ago
    I was just joking about how Macbook drives are not replaceable but some replies on this thread have potential real solutions, which is awesome. I forgot you could attach two Macs during the boot process, which is usually used when you are setting up the new computer and copying files but could be used to fix the broken one too. That's another possible fix, buy a new one with twice the disk space and transfer files on setup?
    - Gigachad 3y ago
      MacBooks have a recovery mode you can boot in to which can download a fresh macOS install off the internet to reinstall the whole system.
  - _wire_ 3y ago
    This is a repost in reply context... It's true that mounting the affected Data partition on another machine won't help. And booting Recovery and mounting is equivalent to mounting on another machine. But it can help, as follows. Before resorting to a wipe, try booting into Recovery, mount the Data partition, then use rm on a large file. When it fails, unmount the Data partition and run fsck -y on the Data partition at the commandline. If it finds errors and fixes, you'll get free space. If you can't figure out how to mount the Data partition read/write at the commandline, close Terminal and run Disk Utility, locate the "Data" partition in the sidebar and right-click to Mount. Quit Disk Utility and restart Terminal. You will find user's data files the /Users folder. You can locate large files on the mounted Data partition with find /Users -size +100M Use df -h /Users to verify that 100M or more are free. Find and rm large files as needed. Then boot normally and finish cleanup from comfort of normal operation. Note— Running fsck via Disk First Aid in Disk Utility should be the same as running fsck -y at commandline, but the UI for Disk Utility can be confusing. For example if it can't unmount the drive to perform the repair, DU will misleadingly advise you that the drive has failed and cannot be repaired. DU has some other odd behaviors, so it's more effective to use fsck at the commandline. Another fine point: It's the specific APFS "Data" partition filesystem that's locked up (user's data), so you need to repair that specific volume, e.g. disk2s2. Look up the Data partition with disutil list Repairing the drive as a whole (e.g., disk2) is not what you want; this just checks that there's a partition table, which will naturally be OK. Similarly, repairing the other APFS system partitions will not help, nor will repairing the APFS Container disk. Fix the specific "Data" partition.
dehrmann 3y ago
I wonder if truncate would have worked.
armchairhacker 3y ago
btrfs used to have this issue, the problem being that the filesystem has to append metadata to do any operation including (ironically) deletion: https://serverfault.com/a/478742 https://serverfault.com/a/478742. AFAIK it's fixed now, because btrfs reserves some space and reports "disk full" before it's reached. macOS probably does the same (I'd hope), but it seems in this case the boundary wasn't enforced properly and the background snapshot caused it to write beyond.
- userbinator 3y ago
  It looks like ZFS also suffers(ed?) the same problem: https://zfs-discuss.opensolaris.narkive.com/BQ7RMcjo/cannot-delete-file-when-fs-100-full https://zfs-discuss.opensolaris.narkive.com/BQ7RMcjo/cannot-...
  - jclulow 3y ago
    Yes, ZFS has the same fundamental issue that all COW file systems have here. We have a reserved pool of space that you can't use except for operations like removing stuff. The new problem with that reserved pool mechanism is that in 2024 it's probably way too big, because it's essentially a small but fixed percentage of the storage size. Don't let people use thresholds of total size without some kind of absolute cap!
userbinator 3y ago
My best guess at what happened (based on a little knowledge of HFS+ disk structures, but not APFS) is that the journal file also filled up, and since deletion requires writing to it and possibly expanding it, you get into the unusual situation where deletion requires, at least temporarily, more space. macOS continued to write files until there was just 41K free on the drive. I've (accidentally) ran both NTFS and FAT32 to 0 bytes free, and it was always possible to delete something even in that situation. Digging around in forums, I found that Sonoma has broken the SMB/Samba-based networking mount procedure for Time Machine restores, and no one had found a solution. This appears to still be the case in 14.4. In my experience SMB became unreliable and just unacceptably buggy many years ago, starting around the 10.12-10.13 timeframe; and now it looks like Apple doesn't care about whether it works at all anymore. I hate to think what people without decades of Mac experience do when confronted with systemic, cascading failures like this when I felt helpless despite what I thought I knew and all the answers I searched for and found on forums. I don't have "decades of Mac experience", but the first thing I'd try is a fsck --- odd not to see that mentioned here. If I were asked to recover from this situation, and couldn't just copy the necessary contents of the disk to another one before formatting it and then copying back, I'd get the APFS documentation (https://developer.apple.com/support/downloads/Apple-File-System-Reference.pdf https://developer.apple.com/support/downloads/Apple-File-Sys...) and figure out what to edit (with dd and a hex editor) to get some free space.
- Lammy 3y ago
  > Sonoma has broken the SMB/Samba-based networking Apple dropped Samba in favor of their own implementation a long time ago after Samba adopted GPLv3: https://lists.samba.org/archive/samba-announce/2007/000122.html https://lists.samba.org/archive/samba-announce/2007/000122.h... https://www.engadget.com/2011-03-24-apple-to-drop-samba-networking-tools-from-lion.html https://www.engadget.com/2011-03-24-apple-to-drop-samba-netw...
  - mkesper 3y ago
    But that's apparently not working with the same features?
  - 3y ago
miles 3y ago
The author might have had better luck by using an external storage device to boot the Mac and delete unneeded files on the internal disk from there: Use an external storage device as a Mac startup disk https://support.apple.com/en-us/111336 https://support.apple.com/en-us/111336 Was surprised to learn that with Apple silicon-based Macs, not all ports are equal when it comes to external booting: If you're using a Mac computer with Apple silicon, your Mac has one or more USB or Thunderbolt ports that have a type USB-C connector. While you're installing macOS on your storage device, it matters which of these ports you use. After installation is complete, you can connect your storage device to any of them. * Mac laptop computer: Use any USB-C port except the leftmost USB-C port when facing the ports on the left side of the Mac. * iMac: Use any USB-C port except the rightmost USB-C port when facing the back of the Mac. * Mac mini: Use any USB-C port except the leftmost USB-C port when facing the back of the Mac. * Mac Studio: Use any USB-C port except the rightmost USB-C port when facing the back of the Mac. * Mac Pro with desktop enclosure: Use any USB-C port except the one on the top of the Mac that is farthest from the power button. * Mac Pro with rack enclosure: Use any USB-C port except the one on the front of the Mac that's closest to the power button.
- userbinator 3y ago
  If the filesystem itself got into a deadlocked state, booting from anything and going through the FS driver to delete files from it won't work.
  - HumanOstrich 3y ago
    What do you mean by "deadlocked state" for a filesystem?
    - pxx 3y ago
      Where `rm`, or more technically unlink(2), fails due to ENOSPC, like in the article... Hilariously this failure case doesn't seem to be listed in the docs. https://developer.apple.com/library/archive/documentation/System/Conceptual/ManPages_iPhoneOS/man2/unlink.2.html https://developer.apple.com/library/archive/documentation/Sy...
deleted 3y ago
[deleted]
whartung 3y ago
I had this happen to me, though I can’t recall how I fixed it. In general I’ve had good success with Time Machine. I, too, have lost TM volumes. I just erased them and started again. Annoying to be sure but 99.99% of the time don’t need a years worth of backups. The author mentioned copying the Time Machine drive. I have never been able to successfully do that. Last time I tried I quit after 3 days. As I understand it, only Finder can copy a Time Machine drive. Terrible experience. That said, I’d rather cope with TM. It’s saved me more than it’s hurt me, and even an idiot like me can get it to work. I did have my machine just complain about one of my partitions being irreparable, but it mounted read only so I was able to copy it, and am currently copying it back. I don’t know if this is random bit rot, or if something is going wrong with the drive. That would be Bad, it’s a 3TB spinning drive. Backed up with BackBlaze (knock on wood), but I’d rather not have to go through the recovery process if I could avoid it. Problem is I don’t know how to prevent it. It’s been suggested that SSDs are potentially less susceptible to bit rot, so maybe switching to one of those is a wise plan. But I don’t know.
- skhr0680 3y ago
  > The author mentioned copying the Time Machine drive. I have never been able to successfully do that. Last time I tried I quit after 3 days. As I understand it, only Finder can copy a Time Machine drive. Terrible experience. rsync -av $SOURCE $DEST has never let me down. Copy or delete on Time Machine files using Finder never worked for me. > Problem is I don’t know how to prevent it. It’s been suggested that SSDs are potentially less susceptible to bit rot, so maybe switching to one of those is a wise plan. But I don’t know. OpenZFS with two drives should protect you from bit rot. ZFS almost became the Mac file system in Snow Leopard.
- pronoiac 3y ago
  I have notes somewhere on roundtripping Time Machine backups between USB drives and network shares. (It's non-trivial, and it's not supported, but it worked.) It was with HFS+ backups, and there were various bits that were "Here Be Dragons", so I never posted them.
  - desro 3y ago
caseyy 3y ago
Windows with NTFS will also break in mysterious ways when the system disk is full. There was a time when OSs could deal with the system disk being full quite well. And not so long ago.
- tredre3 3y ago
  I regularly run my Windows 10 NTFS drive down to 0 bytes free. I'm always amazed at how the file system survives just fine, but the machine doesn't even crash! I'm not sure where you got your experience from.
  - bloomingeek 3y ago
    I tried, accidentally, to over fill the HDD on a Windows Vista machine. Vista popped up a box telling me I couldn't do it. Unfortunately, in my panic, I didn't take a picture of the warning for posterity.
  - caseyy 3y ago
    My work demands that I generate large amounts of data and I don’t know how much I’ll have to generate up-front. So I run out of disk space a lot. My experience is that Windows and many of its programs will become very unstable with 0 b on the system drive. And about 3 times out of maybe 50, the system also became unbootable. I’ve learned to do whatever I can to free up space before restarting for stability. The last time I’d regularly run out of space on Win was around Windows 98 times. I never had a problem then. Now in Windows 11 times, it’s a real headache. Not sure how you’re so lucky.
    - Kwpolska 3y ago
      I manage to get to 0 or close to that sometimes, usually through uncontorlles pagefile expansion. Some apps may misbehave, but explorer is stable enough to let me delete something.
    - juitpykyk 3y ago
      Maybe you should use a secondary partition for work
amarshall 3y ago
> macOS outstripped its ability to throttle filling storage Does it actually do that? I.e. stop the user from writing new data when storage space is extremely low?
1970-01-01 3y ago
A viable solution would be to clone all data onto a bigger drive, then delete some files, wipe the disk, and finally copy everything back.
desro 3y ago
Impressive. I've never dealt with a situation where even `rm` failed, but I have had the displeasure of using and managing modern Macs with 256 GB (or less) of internal storage. I like to keep a "spaceholder" file of around 16GB so when things inevitably fill up and prevent an update or something else, I can nuke the placeholder without having to surgically prune things with `ncdu`
- LeifCarrotson 3y ago
  I find that one of the main benefits of a space holding file is that when it's needed, freeing up that space provides a window of time where you can implement a long-term solution (like buying a new drive with quadruple the storage space of the original for the cost of an hour of that employee/machine's time).
  - derekp7 3y ago
    Had an ex Navy Submariner for a manager many years ago. On some problem systems, he created several such files. He called them ballast files.
- timenova 3y ago
  I noticed CockroachDB does the same thing too, on node startup [0]. [0] https://www.cockroachlabs.com/docs/v23.2/cluster-setup-troubleshooting#automatic-ballast-files https://www.cockroachlabs.com/docs/v23.2/cluster-setup-troub...
mproud 3y ago
Safe Boot is your magical way to have the computer delete purgeable and temporary files, like boot caches. Hold shift down and once it gets to the login window, restart again. Otherwise, go to Recovery mode, mount the disk in Disk Utility, and then open Terminal and rm some shit.
- klausa 3y ago
  The author mentions trying this in recoveryOS to no success.
wyldfire 3y ago
If you can truncate() an existing file (via 'echo > big_file.img' or similar), I would hope the filesystem could deallocate the relevant extents without requiring more space. Seems a bit like a filesystem defect not to reserve enough space to recover from this condition with unlink().
- anticensor 3y ago
  you need delete_and_dealloc(), not unlink()
staticfloat 3y ago
I ran into an issue like this in my first ever job! I accidentally filled up a cluster with junk files and the sysadmin started sending me emails saying I needed to fix it ASAP but rm wouldn’t work. He taught me that file truncation usually works when deletion doesn’t, so you can usually do “cat /dev/null > foo” when “rm foo” doesn’t work.
- jaimehrubiks 3y ago
  Great to know
- pram 3y ago
  You can actually just do >file
- JdeBP 3y ago
  Although note that several comments here report situations where truncation doesn't work either. 21st century filesystem formats are a lot more complex than UFS, and with things like snapshotting and journalling there are new ways for a filesystem to deadlock itself.
  - deleted 3y ago
    [deleted]
- mjevans 3y ago
  In shell :>filepath often works... However sometimes filesystems can't do that. For those cases, hopefully the filesystem supports: resize-grow, resize-shrink, and either additional temporary storage or is on top of an underlying system which can add/remove backing storage. You may also need to use custom commands to restore the filesystem's structure to one intended for a single block device (btrfs comes to mind here).
- JohnMakin 3y ago
  I was once in a situation years ago where a critical piece of infrastructure could brick itself irreparably with a deadlock unless it was always able to write to the file system, so I had a backup process just periodically send garbage directly to dev null and as far as I know that dirty hack is still running years later. /dev/null is magical and worth reading into
quechimba 3y ago
I had an external hard drive that I overfilled by accident while making a manual backup of media files, and after that I couldn't even mount the APFS volume. Apparently it's something that can happen. In the end I was able to mount and rescue the data using https://github.com/libyal/libfsapfs https://github.com/libyal/libfsapfs I followed this guide: https://matt.sh/apfs-object-map-free-recovery https://matt.sh/apfs-object-map-free-recovery
- djmips 3y ago
  Hey thanks for the knowledge!
JdeBP 3y ago
People find it a confusing idea to grasp that deleting things actually requires more space, either temporarily or permanently. Other comments here have gone into the details of why some modern filesystems with snapshotting and journalling and so forth actually end up needing to allocate from free space in order to delete stuff. In a different field: In the early decade of Wikipedia it often had to be explained to people that (at least from roughly 2004 onwards) deleting pages with the intention of saving space on the Wikipedia servers actually did the opposite, since deletion added records to the underlying database. Related situations: * In Rahul Dhesi's ZOO archive file format, deleting an archive entry just sets a flag on the entry's header record. ZOO also did VMS-like file versioning, where adding a new version of a file to an archive did not overwrite the old one. * Back in the days of MS/DR/PC-DOS and FAT, with (sometimes) add-on undeletion utilities installed, deleting a file would need more space to store a new entry into the database that held the restore information for the undeletion utility. * Back in the days of MS/DR/PC-DOS and FAT, some of the old disc compression utilities compressed metadata as well, leading to (rare but possible) situations where metadata changes could affect compressibility and actually increase the (from the outside point of view) volume size. "I delete XYZ in order to free space." is a pervasive concept, but it isn't strictly a correct one.
qiqitori 3y ago
Maybe flamebait, but here is my honest opinion, which I believe is aligned with the hacker ethos: maybe if you were using an open-source operating system you could, with a little experience, write a simple tool that deletes a couple files without allocating new metadata. (Or more likely, somebody else would have been there before you and you could just use their tool.)
- yjftsjthsd-h 3y ago
  Does such a tool exist for BTRFS or ZFS?
  - bjoli 3y ago
    I was in this situation with BTRFS, but it was simple to extend the partition with an old 2gb usb stick and the problem resolved itself.
    - yjftsjthsd-h 3y ago
      Sure, but that's orthogonal to open/closed source.
- Kwpolska 3y ago
  Most users on open-source operating systems can't code as well. And even if they could, this still requires knowledge or guesswork to find the trick for the deletion. Some people suggest truncation, and that's possible with a shell, but what would you do if it failed as well?
  - qiqitori 3y ago
    I have an answer, but I'm not sure you'll find it particularly fulfilling because it is quite hypothetical. The average user just needs to be able to ask the question in a decent place. E.g., Hacker News or a fitting Stack Exchange site. Some developers not afraid of touching the kernel will see the question (or one like it), and if no workaround (e.g. truncation) is found to be acceptable, someone may decide to look into the kernel source to see if it's feasible at all. They may find a lower level function that deletes without writing metadata. Or they may find the function in the filesystem driver's source code where the metadata is written first, and if that was successful, the actual data is written. In the easiest case, you could create a copy of the function with the calls swapped, and a live CD with the modified driver could be created. (Course, this solution is quite unsafe, as writing the metadata could still fail for some other or related reason, so it's a bit of an emergency solution.) There are two other filesystems that were mentioned in the discussion here, btrfs and ZFS. ZFS solved the problem by reserving space, so creating such a tool isn't needed. (However, ZFS is not part of Linux, so I'm not too interested in digging into the details.) btrfs users apparently accept this as a fact-of-life, but have what they consider decent-enough workarounds, see e.g. https://www.reddit.com/r/btrfs/comments/ibjrpm/can_i_somehow_reserve_space_to_prevent_the_btrfs/ https://www.reddit.com/r/btrfs/comments/ibjrpm/can_i_somehow.... (I use neither ZFS nor btrfs; I prefer boring filesystems, thank you very much.)
saurik 3y ago
I know someone who ran out of disk space on her iPhone and then tried to fix some issues she didn't realize were being caused by it by upgrading the device to a new version of iOS; but then it failed the upgrade as it couldn't resize the disk but had already committed to doing such (I later laboriously figured this out by debugging the process using idevicerestore). I feel like this was a bug in its "how much space will I need to have to install successfully, let's verify I have enough before I begin" calculation, and maybe later versions of iOS have fixed the issue, but sadly they are all even larger and the fixed version would just prevent it from trying to upgrade in the first place, not fix it once it got to this point.
thenickdude 3y ago
By contrast, ZFS has "slop space" to avoid this very problem (wedging the filesystem by running out of space during a large operation). By default it reserves 3.2% of your volume's space for this, up to 128GB. So by adjusting the Linux kernel tunable "spa_slop_shift" to shrink the slop space, you can regain up to 128GB of bonus space to successfully complete your file deletion operations: https://openzfs.github.io/openzfs-docs/Performance%20and%20Tuning/Module%20Parameters.html#spa-slop-shift https://openzfs.github.io/openzfs-docs/Performance%20and%20T...
- nobody9999 3y ago
  >By contrast, ZFS has "slop space" to avoid this very problem As does ext4 (although they call the space "reserved blocks"). 'man tune2fs' for details. As well as most other modern (and not so modern[0]) filesystems. [0] As I recall, the same was true for SunOS'[1] UFS back in the 1980s. [1] https://en.wikipedia.org/wiki/SunOS https://en.wikipedia.org/wiki/SunOS
  - _flux 3y ago
    In ext[234]fs the reserved blocks is something else though: they are reserved to a specific user, by default root. So if normal users fill out the filesystem, the root user still has some space to write. Sort of a simple quota system. I believe this problem in is only relevant to CoW filesystems. With ext[234]fs you can set the reserved blocks to 0, fill the fs, and always remove files to fix the situation.
- bell-cot 3y ago
  Yes - and reserving a percentage of disk space (for this reason) was a routine feature of "real" filesystems decades before ZFS (or Linux) even existed. It's kinda like how almost any 1980's MS-DOS shareware terminal program was really good at downloading files over a limited-bandwidth connection, but current versions of MS Windows are utter crap at that should-be-trivial task.
hobs 3y ago
Reminds me of the time I worked at Apple support and got a phone call where a guy had wiped his system drive like 10x+ because he didnt want to lose his data but had run out of inodes so literally nothing was able to create or do anything, and his entire OS was shitting the bed. I showed him how he could type rm -rf and then paste the files in the terminal and it took him a good 15 minutes for him to get it all organized but once he did he literally cried because I got his computer back. Those days were pretty brutal, but there were shining moments.
entropicgravity 3y ago
I ran into a similar situation not long ago on the system partition of a linux installation. The partition was too small to begin with and as new updates piled up there was almost no space left to start deleting stuff. It took me about half an hour to find a subdirectory with a tiny bit of stuff that could be deleted. It was like being in a room so plugged up with junk that you couldn't open the (inward swinging) door to let yourself out. From the tiny beginning I started being able to delete bigger and bigger spaces until finally it was clear and then of course I resized the partition so that wouldn't happen again. The End.
- dataflow 3y ago
  Confused, why didn't you just expand the partition to begin with? And I feel like that ought to be the lesson for power users: always leave a bit of slack space after your partition.
  - entropicgravity 3y ago
    My recollection is that gparted wouldn't allow me to adjust any partitions until I loosened up the available space in the system partition. I guess gparted checks all the partitions to make sure everything is as it should be before it allows changes.
jaredhallen 3y ago
Depending on how dedicated one was, it might work to dd the full contents of the affected disk to a larger one. Then expand the partition, expand the filesystem, and assuming everything is hunky dory, free up some space and reverse all those steps.
atemerev 3y ago
APFS is really sensitive to this situation, more than other file systems. I don’t remember how I managed to resolve this situation, but it involved booting into the single user mode and some magic not accessible to a regular Mac user.
deleted 3y ago
[deleted]
_wire_ 3y ago
Ran into precisely this problem with a friend's Ventura Mini last year. The solution was to boot into recovery and mount the Data partition using Disk Utility. I don't recall where the Data partition gets mounted but I think it is: "/System/Volumes/Macintosh HD - Data" Or just Data, since Sonoma. It will be clear from Disk Utility. Then close Disk Utility and go into Terminal and run rm on a big unseeded file. You can find one using: find <data-mnt> -size +100m Using rm will fail. Unmount the Data partition and run fsck on it. This completes the deletion. From there enough more space can be freed in recovery to have a healthy buffer, then reboot normally and finish cleaning. It seems that when the volume gets so full that rm doesn't work anymore the filesystem also gets corrupted. HTH and that I didn't forget anything.
delta_p_delta_x 3y ago
Huh, something like this happened to my mother's iPhone, too. She kept taking photos until the storage was filled to the brim. One day she had discharged the battery completely, shutting down the phone; after recharging, she tried to restart it, only to be sent into a boot-loop. There is no (official) way to resolve this except repeatedly reboot and hope that at least once, Springboard loads and you can immediately jump into Photos and start mass-deleting, or at least connect to a computer and transfer the media out of the phone.
RamRodification 3y ago
> ... Samba (the disk-sharing protocol), ... Pet peeve (or alternatively: correct me if I'm wrong), Samba is not a protocol. It's a software suite that implements the SMB (Server Message Block) protocol
JackYoustra 3y ago
This happened to me! My solution was to go to an apple store, buy one of their portable SSDs right there, cp everything on to the SSD (that didn't appear to use any additional space!), wipe the mac, and then rm some unneeded stuff on the ssd before cp-ing back and using their no-fee return to return the SSD. There were a few esoteric issues, but for the most part it worked.
albertzeyer 3y ago
When `rm file` gives you "No space left on device", a trick you can do: echo > file # delete content of file first rm file # now it should work
magicalhippo 3y ago
Reminds me of when a customer's database kept crashing with an error code indicating the disk was full. Except Windows Explorer showed the disk having hundreds of gigs free... Took us a little while to figure out that the problem was the database file was so fragmented NTFS couldn't store more fragments for the file[1]. What had happened was they had been running the database in a VM with very low disk space for a long time, several times actually running out of space, before increasing the virtual disk and resizing the partition to match. Hence all the now-available disk space. Just copying the main database file and deleting the old solved it. [1]: https://superuser.com/questions/1315108/ntfs-limitations-max-fragments-per-file https://superuser.com/questions/1315108/ntfs-limitations-max...
TazeTSchnitzel 3y ago
I've had a similar experience on my iPhone. The disk became so full that deleting things was seemingly no longer actually doing anything. Rebooting, the phone couldn't be logged into. Rebooting again, it boot-looped. Rebooting once more, it booted into an inconsistent state where app icons still existed on the home screen, but the actual app was missing, so the icon was blank and the app could not be launched. I became concerned about data integrity and ultimately restored from a backup. I am certain this was a result of APFS being copy-on-write and supporting snapshotting. If no change is immediately permanent, but instead old versions of files stay around in a snapshot, then if you don't have enough space for more snapshot metadata you're in trouble. Maybe they skip the snapshot in low disk space situations, but they still have the copy-on-write metadata problem.
- the-golden-one 3y ago
  I’ve had the exact same thing. Amazing to think in 2024, despite all the clever APFS volume management stuff, you can still put a ‘sealed’ device such as an iPhone into a state where it has to be recovered by DFU just by filling up the user’s storage. In contrast, after accidentally maxing out the space on my windows 11 office laptop which has a single data and boot volume, I was still able to boot it and sort the issue out.
protoman3000 3y ago
> I found that Sonoma has broken... All too familiar. I have two Macs. I upgraded one of them to Sonoma and ever since then it has been nothing but headache and disappointment. Starting from the upgrade having failed (meaning I had to completely wipe the disk and install Sonoma from scratch, luckily I still had data), to problems with Handoff, the firewall seems to not work, Excel very slow etc etc. I don't recommend using Sonoma.
- andrelaszlo 3y ago
  It's terrible :( Bluetooth audio was a joke even before the update, and now it's almost unusable.
Tade0 3y ago
I had a similar issue with a PostgreSQL database - we were running an application that performed simulations and on that day we tried one that was an order of magnitude larger than the previously largest ones. Suddenly queries started failing, so I investigated and even managed to remove some files on the affected volume, but it wasn't enough. Eventually, as the volume's storage class did not allow for expansion, I reached out to the support team to move the data to a larger one.
farkanoid 3y ago
My wife's IPhone 12 Pro Max had the same problem. She somehow managed to fill up the entire 512GB. Updates were unsuccessful, she couldn't make calls and wasn't able to delete anything to make room. She couldn't even back up her phone through iTunes, the only option was to purchase an iCloud subscription and back up to the cloud in order to access her photos.
willyt 3y ago
They mentioned that the problem occurred while steam was downloading, I wonder if because steam is ultra cross platform with bare minimum OS specific UI it is using something quite low level to write data to disk? Maybe NSFile does some checks that posix calls can’t do while remaining compliant with the spec or something weird like that. That would explain why people using various low level ´pro level’ cross platform tools like databases would have issues but typical garage band user is usually ok. If you’re doing database writes you probably don’t want the overhead of these checks making your file system performance look bad so it’s left to the software to check that it’s not going to fill up the file system. Stab in the dark hypothesis. I would hope that however we are writing data to the file system it shouldn’t be able to lock it up like this. I’d be curious for someone with technical knowledge of this to chime in.
- sspiff 3y ago
  Still, you should not be able to brick your device into a state like this with legitimate, normal, non elevated operations. If the POSIX API does have some limitation which would prevent this error from occurring with higher level APIs (which I sincerely doubt), macOS should simply start failing with errno = ENOSPC earlier for POSIX operations. There is no other system that behaves like this, and we wouldn't be making excuses like this if Microsoft messed something basic up like this.
  - willyt 3y ago
    I agree, though others are saying that BTRFS and ZFS can also get into this state.
    - sspiff 3y ago
      I'd have to try, but have never encountered something like it on btrfs (though to be fair I've had many other issues and bugs with it over the years!) I understand the logic, but typically I've seen filesystem implementations block writes once metadata volumes become close enough to full. Also, and I don't know if this is a thing on modern filesystems, you used to be able to reserve free space for root user only, precisely for recovering from issues like this in the past.
kazinator 3y ago
In the shell you can truncate files to zero size with a redirection: $ > large-file It's possible that rm / unlink require working space to perform the transaction of removing the file from the directory, while truncation does not.
Shorel 3y ago
This is a serious failure of the backup mechanism not being able to restore from a backup, and a serious failure of the operating system, not being able to delete files in a mounted external disk. Luckily, all important files were in the cloud, and you could write a blog post describing these monumental failures.
supermatt 3y ago
Sounds like there was no room to write to the journal (journaling is enabled by default on HFS+). Disabling the journaling for that volume (which it doesn’t appear you tried?) will have likely allowed you to perform your deletes. diskutil disableJournal “/Volumes/Macintosh HD” (or whatever volume is)
- djxfade 3y ago
  New macOS versions use APFS, not HFS+. As far as I'm aware, you can't disable journaling on APFS.
resource_waste 3y ago
The sheer number of people who have ran into this is a bit mind boggling. Clearly Apple knows about this, they make no effort to fix it?
EVa5I7bHFq9mnYK 3y ago
The elephant in the room is that it was too expensive to buy a Macbook with more storage for a child, and now it's impossible to upgrade. I usually just replace an SSD if I run over 50% utilization.
- sspiff 3y ago
  SSDs become slower as the fill up, and wear leveling has less available space to choose from when writing data. But filling up an SSD should not brick your volume like that. This is a filesystem implementation bug, not a user error.
LoganDark 3y ago
> they had filled macOS’s startup volume storage so full that the operating system was incapable of deleting files Yeah, this has happened to me too. It became a lot more of an issue when my disk was (forcefully, non-consensually) converted to "APFS" which, along with breaking both alternative operating systems I had installed, also seemed to have a much greater chance of entirely bricking once I ran out of space. I was consistently able to repair it by running a disk check in recovery mode, then deleting the files from the recovery terminal, however that is only accessible by a reboot, which by its very nature, necessarily loses all work that I had open, as it's impossible to save. I never had this issue with HFS+. Everyone who says it can technically lock up is missing the fact that in practice it was still far more resilient than the newer and supposedly "better" APFS.
FartyMcFarter 3y ago
> Terminal: While Terminal would launch, using the standard Unix rm command resulted in a similar error: “No space left on device.” This looks like a huge bug, and the elephant in the room. What's the reason for `rm` requiring space left on the device?
mihau 3y ago
Be careful with iPhones/iOS too. Ignoring „not enough space” warnings will eventually put it into a boot loop. There are threads on Reddit about this. For me, flashing it through Finder with the official firmware was the only option. I lost some photos, the rest I was able to restore from iCloud backup.
m348e912 3y ago
I maxed out the inodes on a partition once. That was a head scratcher trying to figure out what was wrong. It was then I learned about the importance of block sizes when formatting a drive.
voytec 3y ago
I wonder if emptying some large-ish file via `:>file` would help.
Waterluvian 3y ago
I’m most surprised that Steam has apparently done this. It never lets me install anything unless I have the space, and it blocks off the space proactively.
- deathanatos 3y ago
  I've seen it both complain that I didn't have sufficient disk for an install, and then, after I've made room on the disk, have the same installation fail due to a lack of disk space. However they're doing it, their disk space calculations are either wrong or estimates.
Zetobal 3y ago
Fun times old android phones had the same bug.
artgship 3y ago
I had a similar situation happen with running multiple virtual machines where their storage was a .vdi file - the files grew too big and maxed out my storage. On next boot ubuntu would not start, same error telling me disk is too full. I had a recent backup so i just reinstalled everything. I always make sure there is some space on the disks from that moment on.
gadders 3y ago
In the old days (2003) I put a web app live with debug level of logging turned on. It filled the disk up so much the Sys Admin had to get a bus to the data centre to reboot the server (Sun e450 from memory) hands on at the keyboard.
r618 3y ago
i was testing low disk space like situation on a Mac Mini used as small home network storage/server, maybe ~ 1o years ago (running maybe Yosemite at the time or something similar) out of curiosity - so Intel mac and pre-APFS system not sure it this was OS or filesystem feature, but it refused to allocate literally anything if free space reached ~ 100-500MB on system partition so it was being kept _always_ usable, even logging was denied IIRC
deleted 3y ago
[deleted]
jerrysievert 3y ago
this happened to me last Monday. it took over 2 days to recover. it gobsmacked me that I needed space on a filesystem to make space on a filesystem.
slillibri 3y ago
It seems odd that he acknowledges the existence of Time Machine local snapshots but doesn't mention deleting the local snapshots manually. Using `tmutil listlocalsnapshots` and `tmutil deletelocalsnapshots` will actually free up space. I had this experience recently when my local storage was at 99% and simple `rm` wasn't freeing up any space. Once I deleted the local snapshot, 100s of GB were freed. As a side node, Time Machine has been pretty garbage lately. I back up to a local Synology NAS and letting it run automatically will just spin with `connecting to backup disk` (or some such message), but running manually works just fine.
tristor 3y ago
A lot of hate in the comments here for Time Machine. Maybe some of that is justified, but I will say that I've been using Macs professionally since 2012 and while I do use other backup services also, there has never been a competing solution that allows the simplified versioning capabilities and performance of a Time Machine backup. On every single one of my Macs I keep a USB3/USB4 SSD connected just for Time Machine as a target, because it's so useful compared to things like Code42, BackBlaze, Carbonite, SpiderOak, et al.
_wire_ 3y ago
You have to wonder why in this age a trillion dollar computer company that helped usher the PC revolution would allow their systems to become crippled by a full storage device, especially when these devices are applied as indispensable tools for everyday life. I'm begging the question: In the 70s / 80s there was consumer tort litigation for all kinds of "misrepresentation", which was fair, because there was a huge and growing business of dark patterns of false claims, faulty products, and schemes for exploitation of unwitting and/or oppressing customers. But it also became absurd, like class action suits against RCA for selling TVs with 25" inch screens where the picture only measured 24.5 inches due to the cabinet facia overlapping the edge of the tube or the raster not reaching all the way to the edge, etc. Tort reform became a hot-button political issue because an enormous subsector of "consumer-rights" legal practice developed to milk payouts under consumer law. You still see this today for "personal injury." So Apple has trillions in pockets and all their kit is sold with capacity specs. Well, customers has better be able to get access to all that capacity, even if it kills their device. I'm wondering if it's not a bug but a legal calculus, a la intro to Fight Club where Ed Norton is reviewing the share value implications of Ford paying off claims for Pintos that explode when rear-ended versus the cost of a recall?
pvaldes 3y ago
> continued to write files until there was just 41K free on the drive As somebody that has managed several times to lock myself out of my user session by this (and had to learn how to enter again by several means); couldn't they start deleting any files with less than this size and increasing slowly the space available one logfile at a time?. Just wondering.
dredmorbius 3y ago
I'd run into this situation earlier this year, on MacOS, though in my case the restore from Time Machine backup did work. But I found myself in the exceedingly frustrating and confusing situation of not being able to free space by deleting files: - 'rm <filename>' from bash - Emptying Trash. - Deleting ".Trashes" directories. - Booting single-user and attempting to delete files. - Booting to the Recovery system, invoking terminal, and attempting to delete files. - Attempting to remove snapshots using the 'tmutil' utility, either booted normal, safe-mode, single-user, or rescue. Best I can tell, tmutil simply would not run under single-user or rescue modes. Final solution was to repartition the hard drive, re-create filesystem, reinstall the OS, and recover user files from Time Machine. This ... took a while. My take-away is that MacOS behaves exceedingly poorly under any number of high-resource-utilisation modes (memory, CPU, or disk usage). Some discussion from the time on the Fediverse: <https://toot.cat/@dredmorbius/111849495055957910 https://toot.cat/@dredmorbius/111849495055957910>

382 points by goranmoomin 3y ago

Tbh any system cannot be full, because it will malfunction. Not sure about Windows (which warns that it might misbehave once the disk is fully full (￣ー￣)//”” ), but Linux will be unable to spawn processes once the /proc is full. I remember I had problem to login to my linux server, because sshd was unable to spawn bash

ycombiredd 3y ago

I see a few other comments trying to address this with a similar truncation shell command but then a few arguing replies about the redirection operator not working due to some syntax error. So here’s the trick there… preface the redirection operator with a null command, like so :> somefile Note the colon, which stands in for the command as a null operator. An additional hint is that if the system is so starved you can’t run ls, again call On the shell for help and use “echo *”, which will show a list of file names (without sizes) from which hopefully you can select a large file (or few) based on name alone. Colon-greater them into zero byte status and watch your system become useable enough to begin normal recovery efforts. From an old Unix admin; I hope that helps someone.

voidwtf 3y ago

It seems like Time Machine has been steadily declining. I'm not sure why there is no impetus to get it reliable and functioning well. Between sparse bundles becoming corrupt and having to start a new backup and failing functionality I haven't felt like Time Machine is worth setting up anymore. This is in stark contrast to the iOS/iPadOS backups which have worked every time.

wlesieutre 3y ago
I don’t know how many times I’ve had Time Machine decide it didn’t want to work anymore and I had to wipe the backups and start fresh to make it work again, but it’s a much larger number than it should be.
ksec 3y ago
>It seems like Time Machine has been steadily declining. Because they dont sell Time Capsule anymore. And they want you to backup everything to iCloud to grow their Services Revenue.
- philistine 3y ago
  Extrapolate it one more step: Apple is clearly working on an iCloud backup for Macs, cause then that's more services revenue. While they're doing that, why would they fix bugs in Time Machine. People can't surely be using this old thing while we're working on the spiffy new thing!
  - firecall 3y ago
    The problem though is that it is likely to be complete environment backup and restore, like iOS. But we shall see! Hopefully they will provide the ability to backup and restore file versions!
    - bducycy 3y ago
      [dead]
- firecall 3y ago
  Totally. But iCloud isn’t a backup. Its sync. And it will happily sync corrupt files and does not provide any versioning. The best TimeMachine is an SSD connected locally to the Mac. The second best is an SSD running on a Mac setup with TomeMachine Server. Then you are lucky if backups continue to work. And even luckier if you can sensibly restore anything via the hellscape that is the interstellar wormhole travel interface! ;-) BackBlaze is very reliable at least! Not cheap with a house full of computers to backup though :-/
  - realusername 3y ago
    > And it will happily sync corrupt files and does not provide any versioning. It does not even provide a basic progress bar when used on the phone.
ryukoposting 3y ago
As a non-Mac user, this sounds like a catastrophic and inexcusable bug the likes of which would inspire a dogpile of hatred against desktop operating systems with penguin mascots and/or headquarters in Washington.
- folbec 3y ago
  That's the magic of genius marketing
- ipv6ipv4 3y ago
  Time Machine works fine. Better than anything available on Windows. I’ve used it for more than a decade and a half, with multiple restores, across multiple machines. My current desktop Mac environment is a direct descendant of my original Mac from 2004 thanks, largely, to Time Machine.
steve1977 3y ago
The whole of desktop macOS has been steadily declining, so I‘m not surprised by this story in the slightest. But hey, we get new emojis and moving desktop wallpapers…
fmajid 3y ago
Mac OS quality control has been declining since they fired Scott Forstall, and it wasn’t amazing under his tenure to begin with.
aequitas 3y ago
I don't share this experience. I've been running Time Machine for years now on a Samba share for multiple Macs and if anything I've only seen an improvement. Years back I would regularly get a corrupted Time Machine sparse bundle that had to be recreated (or I would restore a previous ZFS snapshot and it would continue of off that), but it also ran over AFP back then I think, not SMB. Lately I've not have had any of these issues on any of the machines. I do have specific flags enabled in the smb.conf file that are recommended for Time Machine backups.
- andruby 3y ago
  Have you restored from a backup during that Time? I have a similar setup (sambsa share on a ZFS NAS) and was appalled at how long it took. Both machines on wired gigabit ethernet, yet the restore took more than 24 hours. And that was for just a 1TB disk.

dannyw 3y ago

This happened to me too, without Time Machine. It’s devastating how rm on macOS can’t remove files when the disk is full.

Dalewyn 3y ago

What would the reasoning be for the file system failing this hard? Filling a partition to 99.999999% capacity always produces a nonsensical situation, but it's usually still recoverable without resorting to DBANing it first.

epmos 3y ago
I do not know the details of Apple's file system, but I wouldn't be surprised if it needs to allocate space for the log (journal) and can't do so. That isn't reasonable in the sense of "this is what the filesystem should do in this situation" but if the log and user data are allocated from the same pool it is quite possible to exhaust both.
opello 3y ago
I'm so curious about this too. I've done this with ext2 and ext3 in the past and truncating a large file solved the problem if rm wouldn't. It's been long enough I don't remember so specifically though, but certainly wasn't "dd to larger disk and grow the partition" to the rescue. If it was a journal issue would something akin to using an initramfs (or live environment) and mounting with data=writeback enable removing files? Or maybe APFS doesn't support that?

sneak 3y ago

Time Machine has been consistently unreliable the entirety of the time since it launched well over a decade ago. It should be common knowledge that it sucks. Use Backblaze if you don’t care about privacy, rsync+ssh to a selfhosted zfs box if you do.

abhinavk 3y ago
You can use backup tools like restic/borg/kopia to encrypt/compress before uploading to Backplaze or any other cloud service.
jen729w 3y ago
Backblaze supports encryption, as does the excellent Arq, which is a better solution than rolling your own rsync+ssh.
- lh7777 3y ago
  Backblaze Personal does support encryption, but it's always been incomplete. If you supply your own encryption key, it's true that Backblaze can't read your data at rest. But to restore files, you have to send your key to Backblaze's server, which will then decrypt the data so that you can download it. They say that they never store the key and promptly delete the unencrypted files from the server, but to me this is still an unnecessary risk. There's no reason why they couldn't handle decryption locally on the client device, but they justify on-server decryption in the name of convenience -- you can restore files via the web without downloading an app. If you're concerned about this, the solution is to use B2 with a 3rd party app like Arq.
  - philistine 3y ago
    I actually use Arq to send my Time Machine backups and the rest of my NAS to S3 Glacier, in case the house burns down or the drives fail (whichever comes first). It works great and is very cheap!
    - pitaj 3y ago

bingo-bongo 3y ago

Probably needed to delete some of the mentioned disk snapshots, before trying to delete files. https://support.apple.com/en-gb/guide/disk-utility/dskuf82354dc/mac https://support.apple.com/en-gb/guide/disk-utility/dskuf8235...

jen729w 3y ago

I had this issue in October 2018 as documented in this Stack Overflow question, whose text I’ll paste below. I was lucky: I had an additional APFS partition that I could remove, thus freeing up disk space. Took me a while to figure out, during which time I was in a proper panic. --- https://apple.stackexchange.com/questions/338721/disk-full-terminal-in-recovery-mode-won-t-delete-files-boot-kernel-panics https://apple.stackexchange.com/questions/338721/disk-full-t... I’m in a pickle here. macOS Mojave, just updated the other day. I managed to fill my disk up while creating a .dmg, and the system froze. I rebooted. Kernel panic. Boot to Recovery mode. Mount the disk. Open Terminal. –bash–3.2# rm /path/to/large/file rm: /path/to/large/file: No space left on device Essentially the same issue as this Unix thread from ‘08! https://www.unix.com/linux/69889-unable-remove-file-using-rm-disk-space-full.html https://www.unix.com/linux/69889-unable-remove-file-using-rm... I’ve tried echo x > /path/to/large/file, no good. It’s borked. Does anyone have any suggestions that aren’t “wipe the drive and restore from your backup”?

bombcar 3y ago
Sounds like creating a sliver of an extra partition of a gig or so might be valuable insurance. Kind of like the old Unix file systems that would reserve 5% for root.
- pixelfarmer 3y ago
  Since the SSD days (I started like 15 years ago with that) I keep a bit of space empty for two reasons: Emergency situations where you may need some extra space and to give SSDs a bit more room to relocate blocks to (they have a certain amount internally reserved already).
bouke 3y ago
With APFS this is not as straightforward though, as containers are only allocated when written too.
djmips 3y ago
I see in that stackexchange post, there's since been another alternative potential solution where you delete you virtual memory partition and if it's large enough it can give you back enough space to allow deleting of files to happen.

owyn 3y ago

Oh! What I would do is take the disk out, mount it on another machine and delete files then put it back........ /s I think the whole stack of operating systems and tools that assume that this is possible get in trouble when it's not possible. I don't want my computer to become a locked down sandbox but it seems like this is where we are headed.

userbinator 3y ago
If you mounted the disk on another machine, I suspect in this situation the filesystem was so full that you'd get the same error when trying to delete, like what the author here encountered when booting into Recovery OS. The only solution is to copy all the files you want to keep to another (not full) disk, then reformat and copy them back, or if you don't have another disk to copy to, somehow edit the disk directly to "manually" free some space.
- owyn 3y ago
  I was just joking about how Macbook drives are not replaceable but some replies on this thread have potential real solutions, which is awesome. I forgot you could attach two Macs during the boot process, which is usually used when you are setting up the new computer and copying files but could be used to fix the broken one too. That's another possible fix, buy a new one with twice the disk space and transfer files on setup?
  - Gigachad 3y ago
    MacBooks have a recovery mode you can boot in to which can download a fresh macOS install off the internet to reinstall the whole system.
- _wire_ 3y ago
  This is a repost in reply context... It's true that mounting the affected Data partition on another machine won't help. And booting Recovery and mounting is equivalent to mounting on another machine. But it can help, as follows. Before resorting to a wipe, try booting into Recovery, mount the Data partition, then use rm on a large file. When it fails, unmount the Data partition and run fsck -y on the Data partition at the commandline. If it finds errors and fixes, you'll get free space. If you can't figure out how to mount the Data partition read/write at the commandline, close Terminal and run Disk Utility, locate the "Data" partition in the sidebar and right-click to Mount. Quit Disk Utility and restart Terminal. You will find user's data files the /Users folder. You can locate large files on the mounted Data partition with find /Users -size +100M Use df -h /Users to verify that 100M or more are free. Find and rm large files as needed. Then boot normally and finish cleanup from comfort of normal operation. Note— Running fsck via Disk First Aid in Disk Utility should be the same as running fsck -y at commandline, but the UI for Disk Utility can be confusing. For example if it can't unmount the drive to perform the repair, DU will misleadingly advise you that the drive has failed and cannot be repaired. DU has some other odd behaviors, so it's more effective to use fsck at the commandline. Another fine point: It's the specific APFS "Data" partition filesystem that's locked up (user's data), so you need to repair that specific volume, e.g. disk2s2. Look up the Data partition with disutil list Repairing the drive as a whole (e.g., disk2) is not what you want; this just checks that there's a partition table, which will naturally be OK. Similarly, repairing the other APFS system partitions will not help, nor will repairing the APFS Container disk. Fix the specific "Data" partition.

dehrmann 3y ago

I wonder if truncate would have worked.

armchairhacker 3y ago

btrfs used to have this issue, the problem being that the filesystem has to append metadata to do any operation including (ironically) deletion: https://serverfault.com/a/478742 https://serverfault.com/a/478742. AFAIK it's fixed now, because btrfs reserves some space and reports "disk full" before it's reached. macOS probably does the same (I'd hope), but it seems in this case the boundary wasn't enforced properly and the background snapshot caused it to write beyond.

userbinator 3y ago
It looks like ZFS also suffers(ed?) the same problem: https://zfs-discuss.opensolaris.narkive.com/BQ7RMcjo/cannot-delete-file-when-fs-100-full https://zfs-discuss.opensolaris.narkive.com/BQ7RMcjo/cannot-...
- jclulow 3y ago
  Yes, ZFS has the same fundamental issue that all COW file systems have here. We have a reserved pool of space that you can't use except for operations like removing stuff. The new problem with that reserved pool mechanism is that in 2024 it's probably way too big, because it's essentially a small but fixed percentage of the storage size. Don't let people use thresholds of total size without some kind of absolute cap!

userbinator 3y ago

My best guess at what happened (based on a little knowledge of HFS+ disk structures, but not APFS) is that the journal file also filled up, and since deletion requires writing to it and possibly expanding it, you get into the unusual situation where deletion requires, at least temporarily, more space. macOS continued to write files until there was just 41K free on the drive. I've (accidentally) ran both NTFS and FAT32 to 0 bytes free, and it was always possible to delete something even in that situation. Digging around in forums, I found that Sonoma has broken the SMB/Samba-based networking mount procedure for Time Machine restores, and no one had found a solution. This appears to still be the case in 14.4. In my experience SMB became unreliable and just unacceptably buggy many years ago, starting around the 10.12-10.13 timeframe; and now it looks like Apple doesn't care about whether it works at all anymore. I hate to think what people without decades of Mac experience do when confronted with systemic, cascading failures like this when I felt helpless despite what I thought I knew and all the answers I searched for and found on forums. I don't have "decades of Mac experience", but the first thing I'd try is a fsck --- odd not to see that mentioned here. If I were asked to recover from this situation, and couldn't just copy the necessary contents of the disk to another one before formatting it and then copying back, I'd get the APFS documentation (https://developer.apple.com/support/downloads/Apple-File-System-Reference.pdf https://developer.apple.com/support/downloads/Apple-File-Sys...) and figure out what to edit (with dd and a hex editor) to get some free space.

Lammy 3y ago
> Sonoma has broken the SMB/Samba-based networking Apple dropped Samba in favor of their own implementation a long time ago after Samba adopted GPLv3: https://lists.samba.org/archive/samba-announce/2007/000122.html https://lists.samba.org/archive/samba-announce/2007/000122.h... https://www.engadget.com/2011-03-24-apple-to-drop-samba-networking-tools-from-lion.html https://www.engadget.com/2011-03-24-apple-to-drop-samba-netw...
- mkesper 3y ago
  But that's apparently not working with the same features?
- 3y ago

miles 3y ago

The author might have had better luck by using an external storage device to boot the Mac and delete unneeded files on the internal disk from there: Use an external storage device as a Mac startup disk https://support.apple.com/en-us/111336 https://support.apple.com/en-us/111336 Was surprised to learn that with Apple silicon-based Macs, not all ports are equal when it comes to external booting: If you're using a Mac computer with Apple silicon, your Mac has one or more USB or Thunderbolt ports that have a type USB-C connector. While you're installing macOS on your storage device, it matters which of these ports you use. After installation is complete, you can connect your storage device to any of them. * Mac laptop computer: Use any USB-C port except the leftmost USB-C port when facing the ports on the left side of the Mac. * iMac: Use any USB-C port except the rightmost USB-C port when facing the back of the Mac. * Mac mini: Use any USB-C port except the leftmost USB-C port when facing the back of the Mac. * Mac Studio: Use any USB-C port except the rightmost USB-C port when facing the back of the Mac. * Mac Pro with desktop enclosure: Use any USB-C port except the one on the top of the Mac that is farthest from the power button. * Mac Pro with rack enclosure: Use any USB-C port except the one on the front of the Mac that's closest to the power button.

userbinator 3y ago
If the filesystem itself got into a deadlocked state, booting from anything and going through the FS driver to delete files from it won't work.
- HumanOstrich 3y ago
  What do you mean by "deadlocked state" for a filesystem?
  - pxx 3y ago
    Where `rm`, or more technically unlink(2), fails due to ENOSPC, like in the article... Hilariously this failure case doesn't seem to be listed in the docs. https://developer.apple.com/library/archive/documentation/System/Conceptual/ManPages_iPhoneOS/man2/unlink.2.html https://developer.apple.com/library/archive/documentation/Sy...

deleted 3y ago

[deleted]

whartung 3y ago

I had this happen to me, though I can’t recall how I fixed it. In general I’ve had good success with Time Machine. I, too, have lost TM volumes. I just erased them and started again. Annoying to be sure but 99.99% of the time don’t need a years worth of backups. The author mentioned copying the Time Machine drive. I have never been able to successfully do that. Last time I tried I quit after 3 days. As I understand it, only Finder can copy a Time Machine drive. Terrible experience. That said, I’d rather cope with TM. It’s saved me more than it’s hurt me, and even an idiot like me can get it to work. I did have my machine just complain about one of my partitions being irreparable, but it mounted read only so I was able to copy it, and am currently copying it back. I don’t know if this is random bit rot, or if something is going wrong with the drive. That would be Bad, it’s a 3TB spinning drive. Backed up with BackBlaze (knock on wood), but I’d rather not have to go through the recovery process if I could avoid it. Problem is I don’t know how to prevent it. It’s been suggested that SSDs are potentially less susceptible to bit rot, so maybe switching to one of those is a wise plan. But I don’t know.

skhr0680 3y ago
> The author mentioned copying the Time Machine drive. I have never been able to successfully do that. Last time I tried I quit after 3 days. As I understand it, only Finder can copy a Time Machine drive. Terrible experience. rsync -av $SOURCE $DEST has never let me down. Copy or delete on Time Machine files using Finder never worked for me. > Problem is I don’t know how to prevent it. It’s been suggested that SSDs are potentially less susceptible to bit rot, so maybe switching to one of those is a wise plan. But I don’t know. OpenZFS with two drives should protect you from bit rot. ZFS almost became the Mac file system in Snow Leopard.
pronoiac 3y ago
I have notes somewhere on roundtripping Time Machine backups between USB drives and network shares. (It's non-trivial, and it's not supported, but it worked.) It was with HFS+ backups, and there were various bits that were "Here Be Dragons", so I never posted them.
- desro 3y ago

caseyy 3y ago

Windows with NTFS will also break in mysterious ways when the system disk is full. There was a time when OSs could deal with the system disk being full quite well. And not so long ago.

tredre3 3y ago
I regularly run my Windows 10 NTFS drive down to 0 bytes free. I'm always amazed at how the file system survives just fine, but the machine doesn't even crash! I'm not sure where you got your experience from.
- bloomingeek 3y ago
  I tried, accidentally, to over fill the HDD on a Windows Vista machine. Vista popped up a box telling me I couldn't do it. Unfortunately, in my panic, I didn't take a picture of the warning for posterity.
- caseyy 3y ago
  My work demands that I generate large amounts of data and I don’t know how much I’ll have to generate up-front. So I run out of disk space a lot. My experience is that Windows and many of its programs will become very unstable with 0 b on the system drive. And about 3 times out of maybe 50, the system also became unbootable. I’ve learned to do whatever I can to free up space before restarting for stability. The last time I’d regularly run out of space on Win was around Windows 98 times. I never had a problem then. Now in Windows 11 times, it’s a real headache. Not sure how you’re so lucky.
  - Kwpolska 3y ago
    I manage to get to 0 or close to that sometimes, usually through uncontorlles pagefile expansion. Some apps may misbehave, but explorer is stable enough to let me delete something.
  - juitpykyk 3y ago
    Maybe you should use a secondary partition for work

amarshall 3y ago

> macOS outstripped its ability to throttle filling storage Does it actually do that? I.e. stop the user from writing new data when storage space is extremely low?

1970-01-01 3y ago

A viable solution would be to clone all data onto a bigger drive, then delete some files, wipe the disk, and finally copy everything back.

desro 3y ago

Impressive. I've never dealt with a situation where even `rm` failed, but I have had the displeasure of using and managing modern Macs with 256 GB (or less) of internal storage. I like to keep a "spaceholder" file of around 16GB so when things inevitably fill up and prevent an update or something else, I can nuke the placeholder without having to surgically prune things with `ncdu`

LeifCarrotson 3y ago
I find that one of the main benefits of a space holding file is that when it's needed, freeing up that space provides a window of time where you can implement a long-term solution (like buying a new drive with quadruple the storage space of the original for the cost of an hour of that employee/machine's time).
- derekp7 3y ago
  Had an ex Navy Submariner for a manager many years ago. On some problem systems, he created several such files. He called them ballast files.
timenova 3y ago
I noticed CockroachDB does the same thing too, on node startup [0]. [0] https://www.cockroachlabs.com/docs/v23.2/cluster-setup-troubleshooting#automatic-ballast-files https://www.cockroachlabs.com/docs/v23.2/cluster-setup-troub...

mproud 3y ago

Safe Boot is your magical way to have the computer delete purgeable and temporary files, like boot caches. Hold shift down and once it gets to the login window, restart again. Otherwise, go to Recovery mode, mount the disk in Disk Utility, and then open Terminal and rm some shit.

klausa 3y ago
The author mentions trying this in recoveryOS to no success.

wyldfire 3y ago

If you can truncate() an existing file (via 'echo > big_file.img' or similar), I would hope the filesystem could deallocate the relevant extents without requiring more space. Seems a bit like a filesystem defect not to reserve enough space to recover from this condition with unlink().

anticensor 3y ago
you need delete_and_dealloc(), not unlink()

staticfloat 3y ago

I ran into an issue like this in my first ever job! I accidentally filled up a cluster with junk files and the sysadmin started sending me emails saying I needed to fix it ASAP but rm wouldn’t work. He taught me that file truncation usually works when deletion doesn’t, so you can usually do “cat /dev/null > foo” when “rm foo” doesn’t work.

jaimehrubiks 3y ago
Great to know
pram 3y ago
You can actually just do >file
JdeBP 3y ago
Although note that several comments here report situations where truncation doesn't work either. 21st century filesystem formats are a lot more complex than UFS, and with things like snapshotting and journalling there are new ways for a filesystem to deadlock itself.
- deleted 3y ago
  [deleted]
mjevans 3y ago
In shell :>filepath often works... However sometimes filesystems can't do that. For those cases, hopefully the filesystem supports: resize-grow, resize-shrink, and either additional temporary storage or is on top of an underlying system which can add/remove backing storage. You may also need to use custom commands to restore the filesystem's structure to one intended for a single block device (btrfs comes to mind here).
JohnMakin 3y ago
I was once in a situation years ago where a critical piece of infrastructure could brick itself irreparably with a deadlock unless it was always able to write to the file system, so I had a backup process just periodically send garbage directly to dev null and as far as I know that dirty hack is still running years later. /dev/null is magical and worth reading into

quechimba 3y ago

I had an external hard drive that I overfilled by accident while making a manual backup of media files, and after that I couldn't even mount the APFS volume. Apparently it's something that can happen. In the end I was able to mount and rescue the data using https://github.com/libyal/libfsapfs https://github.com/libyal/libfsapfs I followed this guide: https://matt.sh/apfs-object-map-free-recovery https://matt.sh/apfs-object-map-free-recovery

djmips 3y ago
Hey thanks for the knowledge!

JdeBP 3y ago

People find it a confusing idea to grasp that deleting things actually requires more space, either temporarily or permanently. Other comments here have gone into the details of why some modern filesystems with snapshotting and journalling and so forth actually end up needing to allocate from free space in order to delete stuff. In a different field: In the early decade of Wikipedia it often had to be explained to people that (at least from roughly 2004 onwards) deleting pages with the intention of saving space on the Wikipedia servers actually did the opposite, since deletion added records to the underlying database. Related situations: * In Rahul Dhesi's ZOO archive file format, deleting an archive entry just sets a flag on the entry's header record. ZOO also did VMS-like file versioning, where adding a new version of a file to an archive did not overwrite the old one. * Back in the days of MS/DR/PC-DOS and FAT, with (sometimes) add-on undeletion utilities installed, deleting a file would need more space to store a new entry into the database that held the restore information for the undeletion utility. * Back in the days of MS/DR/PC-DOS and FAT, some of the old disc compression utilities compressed metadata as well, leading to (rare but possible) situations where metadata changes could affect compressibility and actually increase the (from the outside point of view) volume size. "I delete XYZ in order to free space." is a pervasive concept, but it isn't strictly a correct one.

qiqitori 3y ago

Maybe flamebait, but here is my honest opinion, which I believe is aligned with the hacker ethos: maybe if you were using an open-source operating system you could, with a little experience, write a simple tool that deletes a couple files without allocating new metadata. (Or more likely, somebody else would have been there before you and you could just use their tool.)

yjftsjthsd-h 3y ago
Does such a tool exist for BTRFS or ZFS?
- bjoli 3y ago
  I was in this situation with BTRFS, but it was simple to extend the partition with an old 2gb usb stick and the problem resolved itself.
  - yjftsjthsd-h 3y ago
    Sure, but that's orthogonal to open/closed source.
Kwpolska 3y ago
Most users on open-source operating systems can't code as well. And even if they could, this still requires knowledge or guesswork to find the trick for the deletion. Some people suggest truncation, and that's possible with a shell, but what would you do if it failed as well?
- qiqitori 3y ago
  I have an answer, but I'm not sure you'll find it particularly fulfilling because it is quite hypothetical. The average user just needs to be able to ask the question in a decent place. E.g., Hacker News or a fitting Stack Exchange site. Some developers not afraid of touching the kernel will see the question (or one like it), and if no workaround (e.g. truncation) is found to be acceptable, someone may decide to look into the kernel source to see if it's feasible at all. They may find a lower level function that deletes without writing metadata. Or they may find the function in the filesystem driver's source code where the metadata is written first, and if that was successful, the actual data is written. In the easiest case, you could create a copy of the function with the calls swapped, and a live CD with the modified driver could be created. (Course, this solution is quite unsafe, as writing the metadata could still fail for some other or related reason, so it's a bit of an emergency solution.) There are two other filesystems that were mentioned in the discussion here, btrfs and ZFS. ZFS solved the problem by reserving space, so creating such a tool isn't needed. (However, ZFS is not part of Linux, so I'm not too interested in digging into the details.) btrfs users apparently accept this as a fact-of-life, but have what they consider decent-enough workarounds, see e.g. https://www.reddit.com/r/btrfs/comments/ibjrpm/can_i_somehow_reserve_space_to_prevent_the_btrfs/ https://www.reddit.com/r/btrfs/comments/ibjrpm/can_i_somehow.... (I use neither ZFS nor btrfs; I prefer boring filesystems, thank you very much.)

saurik 3y ago

I know someone who ran out of disk space on her iPhone and then tried to fix some issues she didn't realize were being caused by it by upgrading the device to a new version of iOS; but then it failed the upgrade as it couldn't resize the disk but had already committed to doing such (I later laboriously figured this out by debugging the process using idevicerestore). I feel like this was a bug in its "how much space will I need to have to install successfully, let's verify I have enough before I begin" calculation, and maybe later versions of iOS have fixed the issue, but sadly they are all even larger and the fixed version would just prevent it from trying to upgrade in the first place, not fix it once it got to this point.

thenickdude 3y ago

By contrast, ZFS has "slop space" to avoid this very problem (wedging the filesystem by running out of space during a large operation). By default it reserves 3.2% of your volume's space for this, up to 128GB. So by adjusting the Linux kernel tunable "spa_slop_shift" to shrink the slop space, you can regain up to 128GB of bonus space to successfully complete your file deletion operations: https://openzfs.github.io/openzfs-docs/Performance%20and%20Tuning/Module%20Parameters.html#spa-slop-shift https://openzfs.github.io/openzfs-docs/Performance%20and%20T...

nobody9999 3y ago
>By contrast, ZFS has "slop space" to avoid this very problem As does ext4 (although they call the space "reserved blocks"). 'man tune2fs' for details. As well as most other modern (and not so modern[0]) filesystems. [0] As I recall, the same was true for SunOS'[1] UFS back in the 1980s. [1] https://en.wikipedia.org/wiki/SunOS https://en.wikipedia.org/wiki/SunOS
- _flux 3y ago
  In ext[234]fs the reserved blocks is something else though: they are reserved to a specific user, by default root. So if normal users fill out the filesystem, the root user still has some space to write. Sort of a simple quota system. I believe this problem in is only relevant to CoW filesystems. With ext[234]fs you can set the reserved blocks to 0, fill the fs, and always remove files to fix the situation.
bell-cot 3y ago
Yes - and reserving a percentage of disk space (for this reason) was a routine feature of "real" filesystems decades before ZFS (or Linux) even existed. It's kinda like how almost any 1980's MS-DOS shareware terminal program was really good at downloading files over a limited-bandwidth connection, but current versions of MS Windows are utter crap at that should-be-trivial task.

hobs 3y ago

Reminds me of the time I worked at Apple support and got a phone call where a guy had wiped his system drive like 10x+ because he didnt want to lose his data but had run out of inodes so literally nothing was able to create or do anything, and his entire OS was shitting the bed. I showed him how he could type rm -rf and then paste the files in the terminal and it took him a good 15 minutes for him to get it all organized but once he did he literally cried because I got his computer back. Those days were pretty brutal, but there were shining moments.

entropicgravity 3y ago

I ran into a similar situation not long ago on the system partition of a linux installation. The partition was too small to begin with and as new updates piled up there was almost no space left to start deleting stuff. It took me about half an hour to find a subdirectory with a tiny bit of stuff that could be deleted. It was like being in a room so plugged up with junk that you couldn't open the (inward swinging) door to let yourself out. From the tiny beginning I started being able to delete bigger and bigger spaces until finally it was clear and then of course I resized the partition so that wouldn't happen again. The End.

dataflow 3y ago
Confused, why didn't you just expand the partition to begin with? And I feel like that ought to be the lesson for power users: always leave a bit of slack space after your partition.
- entropicgravity 3y ago
  My recollection is that gparted wouldn't allow me to adjust any partitions until I loosened up the available space in the system partition. I guess gparted checks all the partitions to make sure everything is as it should be before it allows changes.

jaredhallen 3y ago

Depending on how dedicated one was, it might work to dd the full contents of the affected disk to a larger one. Then expand the partition, expand the filesystem, and assuming everything is hunky dory, free up some space and reverse all those steps.

atemerev 3y ago

APFS is really sensitive to this situation, more than other file systems. I don’t remember how I managed to resolve this situation, but it involved booting into the single user mode and some magic not accessible to a regular Mac user.

deleted 3y ago

[deleted]

_wire_ 3y ago

Ran into precisely this problem with a friend's Ventura Mini last year. The solution was to boot into recovery and mount the Data partition using Disk Utility. I don't recall where the Data partition gets mounted but I think it is: "/System/Volumes/Macintosh HD - Data" Or just Data, since Sonoma. It will be clear from Disk Utility. Then close Disk Utility and go into Terminal and run rm on a big unseeded file. You can find one using: find <data-mnt> -size +100m Using rm will fail. Unmount the Data partition and run fsck on it. This completes the deletion. From there enough more space can be freed in recovery to have a healthy buffer, then reboot normally and finish cleaning. It seems that when the volume gets so full that rm doesn't work anymore the filesystem also gets corrupted. HTH and that I didn't forget anything.

delta_p_delta_x 3y ago

Huh, something like this happened to my mother's iPhone, too. She kept taking photos until the storage was filled to the brim. One day she had discharged the battery completely, shutting down the phone; after recharging, she tried to restart it, only to be sent into a boot-loop. There is no (official) way to resolve this except repeatedly reboot and hope that at least once, Springboard loads and you can immediately jump into Photos and start mass-deleting, or at least connect to a computer and transfer the media out of the phone.

RamRodification 3y ago

> ... Samba (the disk-sharing protocol), ... Pet peeve (or alternatively: correct me if I'm wrong), Samba is not a protocol. It's a software suite that implements the SMB (Server Message Block) protocol

JackYoustra 3y ago

This happened to me! My solution was to go to an apple store, buy one of their portable SSDs right there, cp everything on to the SSD (that didn't appear to use any additional space!), wipe the mac, and then rm some unneeded stuff on the ssd before cp-ing back and using their no-fee return to return the SSD. There were a few esoteric issues, but for the most part it worked.

albertzeyer 3y ago

When `rm file` gives you "No space left on device", a trick you can do: echo > file # delete content of file first rm file # now it should work

magicalhippo 3y ago

Reminds me of when a customer's database kept crashing with an error code indicating the disk was full. Except Windows Explorer showed the disk having hundreds of gigs free... Took us a little while to figure out that the problem was the database file was so fragmented NTFS couldn't store more fragments for the file[1]. What had happened was they had been running the database in a VM with very low disk space for a long time, several times actually running out of space, before increasing the virtual disk and resizing the partition to match. Hence all the now-available disk space. Just copying the main database file and deleting the old solved it. [1]: https://superuser.com/questions/1315108/ntfs-limitations-max-fragments-per-file https://superuser.com/questions/1315108/ntfs-limitations-max...

TazeTSchnitzel 3y ago

I've had a similar experience on my iPhone. The disk became so full that deleting things was seemingly no longer actually doing anything. Rebooting, the phone couldn't be logged into. Rebooting again, it boot-looped. Rebooting once more, it booted into an inconsistent state where app icons still existed on the home screen, but the actual app was missing, so the icon was blank and the app could not be launched. I became concerned about data integrity and ultimately restored from a backup. I am certain this was a result of APFS being copy-on-write and supporting snapshotting. If no change is immediately permanent, but instead old versions of files stay around in a snapshot, then if you don't have enough space for more snapshot metadata you're in trouble. Maybe they skip the snapshot in low disk space situations, but they still have the copy-on-write metadata problem.

the-golden-one 3y ago
I’ve had the exact same thing. Amazing to think in 2024, despite all the clever APFS volume management stuff, you can still put a ‘sealed’ device such as an iPhone into a state where it has to be recovered by DFU just by filling up the user’s storage. In contrast, after accidentally maxing out the space on my windows 11 office laptop which has a single data and boot volume, I was still able to boot it and sort the issue out.

protoman3000 3y ago

> I found that Sonoma has broken... All too familiar. I have two Macs. I upgraded one of them to Sonoma and ever since then it has been nothing but headache and disappointment. Starting from the upgrade having failed (meaning I had to completely wipe the disk and install Sonoma from scratch, luckily I still had data), to problems with Handoff, the firewall seems to not work, Excel very slow etc etc. I don't recommend using Sonoma.

andrelaszlo 3y ago
It's terrible :( Bluetooth audio was a joke even before the update, and now it's almost unusable.

Tade0 3y ago

I had a similar issue with a PostgreSQL database - we were running an application that performed simulations and on that day we tried one that was an order of magnitude larger than the previously largest ones. Suddenly queries started failing, so I investigated and even managed to remove some files on the affected volume, but it wasn't enough. Eventually, as the volume's storage class did not allow for expansion, I reached out to the support team to move the data to a larger one.

farkanoid 3y ago

My wife's IPhone 12 Pro Max had the same problem. She somehow managed to fill up the entire 512GB. Updates were unsuccessful, she couldn't make calls and wasn't able to delete anything to make room. She couldn't even back up her phone through iTunes, the only option was to purchase an iCloud subscription and back up to the cloud in order to access her photos.

willyt 3y ago

They mentioned that the problem occurred while steam was downloading, I wonder if because steam is ultra cross platform with bare minimum OS specific UI it is using something quite low level to write data to disk? Maybe NSFile does some checks that posix calls can’t do while remaining compliant with the spec or something weird like that. That would explain why people using various low level ´pro level’ cross platform tools like databases would have issues but typical garage band user is usually ok. If you’re doing database writes you probably don’t want the overhead of these checks making your file system performance look bad so it’s left to the software to check that it’s not going to fill up the file system. Stab in the dark hypothesis. I would hope that however we are writing data to the file system it shouldn’t be able to lock it up like this. I’d be curious for someone with technical knowledge of this to chime in.

sspiff 3y ago
Still, you should not be able to brick your device into a state like this with legitimate, normal, non elevated operations. If the POSIX API does have some limitation which would prevent this error from occurring with higher level APIs (which I sincerely doubt), macOS should simply start failing with errno = ENOSPC earlier for POSIX operations. There is no other system that behaves like this, and we wouldn't be making excuses like this if Microsoft messed something basic up like this.
- willyt 3y ago
  I agree, though others are saying that BTRFS and ZFS can also get into this state.
  - sspiff 3y ago
    I'd have to try, but have never encountered something like it on btrfs (though to be fair I've had many other issues and bugs with it over the years!) I understand the logic, but typically I've seen filesystem implementations block writes once metadata volumes become close enough to full. Also, and I don't know if this is a thing on modern filesystems, you used to be able to reserve free space for root user only, precisely for recovering from issues like this in the past.

kazinator 3y ago

In the shell you can truncate files to zero size with a redirection: $ > large-file It's possible that rm / unlink require working space to perform the transaction of removing the file from the directory, while truncation does not.

Shorel 3y ago

This is a serious failure of the backup mechanism not being able to restore from a backup, and a serious failure of the operating system, not being able to delete files in a mounted external disk. Luckily, all important files were in the cloud, and you could write a blog post describing these monumental failures.

supermatt 3y ago

Sounds like there was no room to write to the journal (journaling is enabled by default on HFS+). Disabling the journaling for that volume (which it doesn’t appear you tried?) will have likely allowed you to perform your deletes. diskutil disableJournal “/Volumes/Macintosh HD” (or whatever volume is)

djxfade 3y ago
New macOS versions use APFS, not HFS+. As far as I'm aware, you can't disable journaling on APFS.

resource_waste 3y ago

The sheer number of people who have ran into this is a bit mind boggling. Clearly Apple knows about this, they make no effort to fix it?

EVa5I7bHFq9mnYK 3y ago

The elephant in the room is that it was too expensive to buy a Macbook with more storage for a child, and now it's impossible to upgrade. I usually just replace an SSD if I run over 50% utilization.

sspiff 3y ago
SSDs become slower as the fill up, and wear leveling has less available space to choose from when writing data. But filling up an SSD should not brick your volume like that. This is a filesystem implementation bug, not a user error.

LoganDark 3y ago

> they had filled macOS’s startup volume storage so full that the operating system was incapable of deleting files Yeah, this has happened to me too. It became a lot more of an issue when my disk was (forcefully, non-consensually) converted to "APFS" which, along with breaking both alternative operating systems I had installed, also seemed to have a much greater chance of entirely bricking once I ran out of space. I was consistently able to repair it by running a disk check in recovery mode, then deleting the files from the recovery terminal, however that is only accessible by a reboot, which by its very nature, necessarily loses all work that I had open, as it's impossible to save. I never had this issue with HFS+. Everyone who says it can technically lock up is missing the fact that in practice it was still far more resilient than the newer and supposedly "better" APFS.

FartyMcFarter 3y ago

> Terminal: While Terminal would launch, using the standard Unix rm command resulted in a similar error: “No space left on device.” This looks like a huge bug, and the elephant in the room. What's the reason for `rm` requiring space left on the device?

mihau 3y ago

Be careful with iPhones/iOS too. Ignoring „not enough space” warnings will eventually put it into a boot loop. There are threads on Reddit about this. For me, flashing it through Finder with the official firmware was the only option. I lost some photos, the rest I was able to restore from iCloud backup.

m348e912 3y ago

I maxed out the inodes on a partition once. That was a head scratcher trying to figure out what was wrong. It was then I learned about the importance of block sizes when formatting a drive.

voytec 3y ago

I wonder if emptying some large-ish file via `:>file` would help.

Waterluvian 3y ago

I’m most surprised that Steam has apparently done this. It never lets me install anything unless I have the space, and it blocks off the space proactively.

deathanatos 3y ago
I've seen it both complain that I didn't have sufficient disk for an install, and then, after I've made room on the disk, have the same installation fail due to a lack of disk space. However they're doing it, their disk space calculations are either wrong or estimates.

Zetobal 3y ago

Fun times old android phones had the same bug.

artgship 3y ago

I had a similar situation happen with running multiple virtual machines where their storage was a .vdi file - the files grew too big and maxed out my storage. On next boot ubuntu would not start, same error telling me disk is too full. I had a recent backup so i just reinstalled everything. I always make sure there is some space on the disks from that moment on.

gadders 3y ago

In the old days (2003) I put a web app live with debug level of logging turned on. It filled the disk up so much the Sys Admin had to get a bus to the data centre to reboot the server (Sun e450 from memory) hands on at the keyboard.

r618 3y ago

i was testing low disk space like situation on a Mac Mini used as small home network storage/server, maybe ~ 1o years ago (running maybe Yosemite at the time or something similar) out of curiosity - so Intel mac and pre-APFS system not sure it this was OS or filesystem feature, but it refused to allocate literally anything if free space reached ~ 100-500MB on system partition so it was being kept _always_ usable, even logging was denied IIRC

deleted 3y ago

[deleted]

jerrysievert 3y ago

this happened to me last Monday. it took over 2 days to recover. it gobsmacked me that I needed space on a filesystem to make space on a filesystem.

slillibri 3y ago

It seems odd that he acknowledges the existence of Time Machine local snapshots but doesn't mention deleting the local snapshots manually. Using `tmutil listlocalsnapshots` and `tmutil deletelocalsnapshots` will actually free up space. I had this experience recently when my local storage was at 99% and simple `rm` wasn't freeing up any space. Once I deleted the local snapshot, 100s of GB were freed. As a side node, Time Machine has been pretty garbage lately. I back up to a local Synology NAS and letting it run automatically will just spin with `connecting to backup disk` (or some such message), but running manually works just fine.

tristor 3y ago

A lot of hate in the comments here for Time Machine. Maybe some of that is justified, but I will say that I've been using Macs professionally since 2012 and while I do use other backup services also, there has never been a competing solution that allows the simplified versioning capabilities and performance of a Time Machine backup. On every single one of my Macs I keep a USB3/USB4 SSD connected just for Time Machine as a target, because it's so useful compared to things like Code42, BackBlaze, Carbonite, SpiderOak, et al.

_wire_ 3y ago

You have to wonder why in this age a trillion dollar computer company that helped usher the PC revolution would allow their systems to become crippled by a full storage device, especially when these devices are applied as indispensable tools for everyday life. I'm begging the question: In the 70s / 80s there was consumer tort litigation for all kinds of "misrepresentation", which was fair, because there was a huge and growing business of dark patterns of false claims, faulty products, and schemes for exploitation of unwitting and/or oppressing customers. But it also became absurd, like class action suits against RCA for selling TVs with 25" inch screens where the picture only measured 24.5 inches due to the cabinet facia overlapping the edge of the tube or the raster not reaching all the way to the edge, etc. Tort reform became a hot-button political issue because an enormous subsector of "consumer-rights" legal practice developed to milk payouts under consumer law. You still see this today for "personal injury." So Apple has trillions in pockets and all their kit is sold with capacity specs. Well, customers has better be able to get access to all that capacity, even if it kills their device. I'm wondering if it's not a bug but a legal calculus, a la intro to Fight Club where Ed Norton is reviewing the share value implications of Ford paying off claims for Pintos that explode when rear-ended versus the cost of a recall?

pvaldes 3y ago

> continued to write files until there was just 41K free on the drive As somebody that has managed several times to lock myself out of my user session by this (and had to learn how to enter again by several means); couldn't they start deleting any files with less than this size and increasing slowly the space available one logfile at a time?. Just wondering.

dredmorbius 3y ago

I'd run into this situation earlier this year, on MacOS, though in my case the restore from Time Machine backup did work. But I found myself in the exceedingly frustrating and confusing situation of not being able to free space by deleting files: - 'rm <filename>' from bash - Emptying Trash. - Deleting ".Trashes" directories. - Booting single-user and attempting to delete files. - Booting to the Recovery system, invoking terminal, and attempting to delete files. - Attempting to remove snapshots using the 'tmutil' utility, either booted normal, safe-mode, single-user, or rescue. Best I can tell, tmutil simply would not run under single-user or rescue modes. Final solution was to repartition the hard drive, re-create filesystem, reinstall the OS, and recover user files from Time Machine. This ... took a while. My take-away is that MacOS behaves exceedingly poorly under any number of high-resource-utilisation modes (memory, CPU, or disk usage). Some discussion from the time on the Fediverse: <https://toot.cat/@dredmorbius/111849495055957910 https://toot.cat/@dredmorbius/111849495055957910>