13 ms·
MacOS may lose data on APFS-formatted disk images
- XJOKOLAT 9y agoLol
- blumomo 9y agoClick bait. What I describe below applies to APFS sparse disk images only — ordinary APFS volumes (e.g. your SSD startup disk) are not affected by this problem. While the underlying problem here is very serious, this is not likely to be a widespread problem, and will be most applicable to a small subset of backups. Disk images are not used for most backup task activity, they are generally only applicable when making backups to network volumes. If you make backups to network volumes, read on to learn more.
- dictum 9y agoThe title clearly qualifies the claim: "MacOS may lose data _on APFS-formatted disk images_" I didn't know there were APFS-formatted disk images (new in 10.13). Even when you consider the many different kinds of disk images that macOS supports, there's a pretty clear distinction between disk image and a backup of your startup disk, made to another partition in another drive. Any additional clarification would get into "MacOS may lose data on APFS-formatted disk images (disk images, not disk-to-disk, as in another volume..." territory.
- GirlsCanCode 9y agoExactly. But the Apple Apologists, who are just doing their job, will say "#FakeNews" or "but Windows!" or anything to deflect from the terrible problems that platform is having.
- condescendence 9y agoYeah this is the opposite of a click-bait title; it clearly explains what the article is about. "may" lose data on "APFS-formatted" disk images.
- deleted 9y ago[deleted]
- jrochkind1 9y agoA lot of people do make backups to disk images though, especially techies. I believe that remains the only method to use 'time machine' with a generic network connected device rather than an apple branded 'time machine' device? The last place you want an unreliable file system is your backups!
- threeseed 9y agoQuite a few non-Apple network devices support Time Machine. I back up every day to my Synology NAS for example.
- banned1 9y agoUnrelated (sorry for being opportunistic): How do you like your Synology NAS? I’m considering it.
- exikyut 9y agoNot the person you replied to, but I want to make a case for ZFS on a generic small motherboard. You wind up with Linux or FreeBSD so it's a general-purpose server, unless you want to use something like FreeNAS. And with ZFS, you get snapshotting, RAIDz, checksumming, etc, as opposed to "oh it has RAID 5 woohoo". RAID (including the software RAID in Linux) doesn't actually do checksumming for file verification. AFAIK, ZFS is the only open system to do so.
- banned1 9y agoThank you
- Bitcoin_McPonzi 9y agoYou appear to have an agenda of protecting Apple. The title is very clear, and the first paragraph which you quoted explains it in detail. It's significant that an established respected company--the makers of Carbon Copy Cloner--will not support APFS formatted disk images for its backups. While your agenda may be driven by the need to protect Apple, the rest of us need to know this important news about APFS so we can be fully informed.
- threeseed 9y agoThis isn't politics. People can have opinions without them needing to be flagged as having hidden agendas. And I have worked at Apple and can assure you that there isn't some payment structure for posting on forums.
- pinewurst 9y agoCarbon Copy Cloner is an excellent program btw - very happy owner here
- dreamcompiler 9y agoCompletely agree. I gave up on Time Machine a long time ago for a variety of reasons, but CCC continues to meet my backup needs perfectly.
- torstenvl 9y agoYes, fairly limited impact, but still. Data loss is fucking important. We aren't talking about an occasional stutter when moving windows or a sound driver that sometimes needs a reboot to resume working. This is a major bug. I would really really like it if Apple would get their shit together so that people who actually rely on their computers to work correctly can upgrade at some point.
- toasterlovin 9y agoAs a counterpoint, APFS was deployed to almost a billion devices over the span of several months when it was release and, IIRC, this is the first major issue.
- Gaelan 9y agoThere was the password vs password hint thing.
- deleted 9y ago[deleted]
- tqkxzugoaupvwqr 9y agoWhich was a bug in Disk Utility, not APFS.
- Gaelan 9y agoRight, my bad.
- nnutter 9y agoSome of the upgrades to High Sierra in my company failed and left the laptop unbootable. Unfortunately, I wasn't involved in the repair so I only have been told it was due to the APFS conversion and that the solution was to wipe and install.
- StillBored 9y agoHow do you know that? Filesystem corruption is frequently silent, and every-time it happens customers don't get on the phone and send the disks to apple so that they can root cause the problem. Its quite possible this bug has happened an untold number of times before it happened to someone who went through the effort to reproduce and isolate it.
- deleted 9y ago[deleted]
- woodfall 9y agoso why exactly does somebody backup into a image with variable size, when they could just dd together a fixed size image that makes these free space calculations unnecessary?
- tqkxzugoaupvwqr 9y agoI use a sparse bundle disk image, i.e. a disk image with variable size and consisting of multiple files under the hood, because it is more efficient to back up over a network. Instead of uploading a 50 GB file to a cloud storage on every backup, only a fraction of data has to be uploaded (the sparse bundle’s changed files) which makes backing up the 50 GB file very fast if only a few megabytes were added. If anyone is curious: I use restic[1] as backup client and Backblaze B2[2] as backup storage. Works well with sparse bundles. [1] https://restic.net https://restic.net [2] https://www.backblaze.com/b2/cloud-storage.html https://www.backblaze.com/b2/cloud-storage.html
- woodfall 9y agoWait, why would the 50 GB need to be transferred fully every time if it was a fixed size image?
- tqkxzugoaupvwqr 9y agoThe very first backup of a fixed size image will be the full size of the image, e.g. 50 GB, no matter which backup software you use. Even if inside the image no files exist. The very first backup of a sparse bundle disk image will be ~100 MB (the initial size of that image). On repeated backups, some backup softwares operate on file level and upload the whole file if it changed. So if you have a fixed size 50 GB image, mount it, add a file, unmount it, it changed, and the whole 50 GB image file has to be uploaded (with some backup softwares).
- Nullabillity 9y agoSure, but any halfway competent backup program would split it into chunks before deduplication anyway, rendering the exercise pointless.
- notadoc 9y agoHigh Sierra seems like a real gem.
- FreakyT 9y agoHonestly, it's the first Mac OS release I'd actively recommend avoiding upgrading to. It offers essentially no benefits over the prior release, and a whole lot of downsides. (Security issues notwithstanding, High Sierra also drops compatibility for a lot of older software.)
- sbuk 9y agoAs did other versions of OSX[0]. This whole meme should get a law like Betteridge’s. Complaints about the current release of macOS are as old as the hills[1] and have become entirely vacuous. That’s not to say the complaints aren’t necessarily without merit; just that there is too much hyperbole around them. [0]https://forums.macrumors.com/threads/apps-that-wont-work-on-snow-leopard.772853/ https://forums.macrumors.com/threads/apps-that-wont-work-on-... [1]https://youtu.be/GWxC8ezE4Dk https://youtu.be/GWxC8ezE4Dk
- freehunter 9y agoAs well as every version of every OS ever. Windows 98 was the best, Windows XP sucked with it's Fisher Price UI. Then Vista/7 came around and the outcry of "if you want the latest DirectX, you have to upgrade". And then Windows XP was discontinued to much wailing and gnashing of teeth, because XP was the best. Then 8/10 came out and everything was terrible and we're just going to stick to Windows 7 because it's the best. To many techies, version n-1 is the best thing ever created, until version n+1 comes out and the "best ever" shifts up one revision.
- torstenvl 9y agoCherry-picked anecdotes are worthless. Few if any people glorified Panther when Tiger came out, or Leopard when Snow Leopard came out, or Win3.11 when Win95 came out, or Win95 when Win98 came out, or WinME when WinXP came out, or Vista when Win7 came out. Stop dismissing legitimate complaints just because you worship "new and shiny"
- tqkxzugoaupvwqr 9y agoThe same behavior (the sparse bundle disk image not updating the free space amount in accordance with the underlying disk’s free space) is present if one selects ExFAT as format in Disk Utility. I haven’t tested if file corruption is the consequence, too, of copying more data into the disk image than the underlying disk has free space.
- rlkf 9y agoIs the incident with the matching checksum that he mentions because APFS only checksum metadata (which are on preallocated space on the image), or is it his own checksum (say sha1sum), I wonder? It seems strange that the filesystem driver would cache 500 GB of sequentially written data in RAM.
- zbentley 9y ago> It seems strange that the filesystem driver would cache 500 GB of sequentially written data in RAM. That was the most interesting/worrying part of TFA, and I would love to see how the checksum tests were conducted clarified in the text. Presumably, the "md5" commandline tool has no special fallback to the filesystem checksum cache (if it does, rather a lot of my life has been a lie, I'm afraid). Since that's the case, could we assume that, if the "lost" writes totalled $X GB of data, that any evil memory-caching of the file will only work in the presence of at least $X GB of free system memory (RAM plus swap). I'd also be interested in learning what happens if there's less than that amount of memory available. Will the checksum fail? Will an error occur elsewhere? Will the system have some sort of memory (and swap) exhaustion failure/panic?
- praseodym 9y agoThe video embedded in TFA shows md5 reporting identical checksums before unmounting the disk image, so it must be reading the data from a cache.
- wiradikusuma 9y agoI upgraded my hackintosh to High Sierra with APFS. The next day, I accidentally switched off the machine while it was in the _process_ of shutting down (the screen had gone blank, but casing still emitting lights). Next time I turned it on, I couldn't get past login screen (giving me forever beach ball). I put the ssd inside my old MBP as slave to recover data. The ssd was corrupted, most data gone, as in shown in Finder but couldn't be copied. I googled for solutions, but it seems I'm the first to experience this.
- im3w1l 9y agoThe data is probably still there just some metadata being corrupted making it inaccesible to standard tools.
- threeseed 9y agoAre you sure the disk wasn't encrypted ? And given you are running on a Hackintosh there isn't much anyone can do given the unsupported hardware.
- voldemort1968 9y ago...So not an actual Apple Product.
- bhj 9y agoMy (limited) understanding of APFS is that it forgoes some integrity checks on the assumption that they have already been done by lower-level hardware. This is of course a debatable design decision, but it may indeed be unwise to use APFS on non-Apple hardware.
- gigatexal 9y agoShould have paid Oracle and just put ZFS on OS X.
- alwillis 9y agoApple killed ZFS on macOS nearly 10 years ago. It wouldn’t make sense on a watch or a phone anyway: http://www.zdnet.com/article/mac-zfs-is-dead-rip/ http://www.zdnet.com/article/mac-zfs-is-dead-rip/
- gigatexal 9y ago200B in the bank and they couldn’t find a way to port it to other platforms? I think they could.
- alwillis 9y agoIt’s not about not being able to port it; it’s not the right tool for the job. ZFS is a file system designed for servers; the Apple Watch, Apple TV and the iPhone and iPad aren’t servers.
- STRML 9y agoThat doesn't mean it can't be adapted and tuned for mobile. The average smartwatch has more processing power than the average server did at the time the ZFS project was launched (2001).
- gigatexal 9y ago+1 to that. Or at least some subset that speaks the ZFS-protocol if such a thing exists.
- agildehaus 9y agoCan you even test a filesystem properly internally? Seems to me we're all involved in a massive public beta.
- tankenmate 9y agoApple authored a tool called fsx (file system exerciser), but I doubt it checks for corner cases for loopback mounts. Maybe they should add such tests.
- koverstreet 9y agoThat tool has been around for ages, I think it might originally be from SGI.
- syncsynchalt 9y agoThey say it takes ten years to get the bugs out of a filesystem. Unfortunately I think a general release is necessary to mature a new FS.
- _phaq 9y agoExternally "testing" a filesystem isn't exactly easy either. If an error occurs, there's not gonna be an exception thrown that you can send off to telemetry. If you're lucky, your customer notices that files are missing, understands that it has to be a bug in the operating system and maybe even has a rough idea what they were doing that caused the bug to occur, then calls up support and your first-level support is competent enough to direct the problem to the filesystem people and then it's still going to require a lot of luck for that department to reproduce the problem and to actually find out what in the code is causing it.
- voldemort1968 9y agoNot to downplay the importance of this, but it reads as clickbait that you wait until the second paragraph to say "oh yeah, it's only sparsebundles. Just those things that almost nobody uses."
- xref 9y agowe use encrypted sparsebundle images to hold sensitive data: ssh keys, SSL keys, license files. I doubt I'm the only one, this is pretty important to our business.
- zbentley 9y agoEh, it's a minority but not an "almost nobody" minority. There are plenty of pieces of software even besides CCC that rely on/give prominent options for the use of sparsebundles. It's more like "the latest OS update leaves you vulnerable to malicious javascript everywhere, but only if you use Opera". It's a minority of users, sure, but it's still really important information to note.
- yborg 9y agoYes, nobody - like for example Apple, to implement Time Machine. But now perhaps I better understand why Time Machine backups aren't supported on APFS.
- saagarjha 9y agoSure, it's a new filesystem; it's bound to have bugs. However, one pain point I've identified is that the existing tools often have no idea of how to deal with APFS. I'm currently typing this from a Mac with an APFS drive that is almost certainly experiencing filesystem corruption–I have folders that suddenly lose track of their contents and become impossible to delete; however, existing tools such as fsck, diskutil, etc. can't do anything to fix the issue, because their idea of how APFS works is woefully inadequate.
- pram 9y agoThere is a fsck_apfs, I’m pretty sure that’s what the repair button in the disk util runs.
- Exuma 9y ago`fsck_apfs -n -l -x /dev/rdisk1s1`
- saagarjha 9y agoYes: # diskutil verifyVolume /dev/disk0s2 Started file system verification on disk2s1 macOS Verifying file system Volume was successfully unmounted Performing fsck_apfs -n -x /dev/rdisk2s1 Checking volume Checking the container superblock Checking the EFI jumpstart record Checking the space manager Checking the object map Checking the APFS volume superblock Checking the object map error: btn: invalid key (210, 16) Object map is invalid The volume /dev/rdisk2s1 could not be verified completely File system check exit code is 8 Restoring the original state found as mounted Error: -69845: File system verify or repair failed Underlying error: 8: Exec format error
- jaaames 9y agoYep, I had an APFS volume that I couldn't even health check. I was pretty sure it was filesystem corruption as it'd hang and shut off randomly then come back up without issues, same as when the SSD died the first time. Also am r APFS time machine volume would never finish encrypting, even after being plugged in for days.
- derefr 9y agoI tend to create sparsebundles to clone my git repos within, to get around the overhead of having huge numbers of inodes on a volume. (Copying, deleting, unpacking archives, Spotlight indexing—all are way slower when you have the worktrees and .git directories from a thousand large repos splayed out across your disk.) So I was a little worried here. Thankfully, I had manually been setting my sparsebundles back to HFS+ on creation, because I saw no reason to make them APFS containers.
- eropple 9y agoThis is a neat idea, I've never thought about doing that. You should write a blog post about it.
- blattimwind 9y agoTBH an I/O system where having a file system inside a loopback device on another file system is faster than using said file system directly in the first place sounds kinda broken-ish/poorly scaling to me.
- eropple 9y agoHaving Spotlight ignore .git directories and the like is probably wise, I would agree with that. But it's text, even if it's basically garbage text (from a user perspective). So I can understand how a sparsebundle is a decent end-around. The Finder in general ends up basically being useless for me for similar reasons; I have dozens of random dependency files I don't even recognize pop up in "All My Files".
- dunham 9y agoSpotlight ignores hidden directories (e.g. .git) and directories whose names end in .noindex. You can create a file in one with a unique name and try to mdfind it to verify this.
- 9y ago
- kbumsik 9y agoTwo bugs are described in this article: 1. An APFS volume's free space doesn't reflect a smaller amount of free space on the underlying disk 2. The diskimages-helper application doesn't report errors when write requests fail to grow the disk image These are not even complex problems of the new format. It is just Apple forgot to have basic checks. It is like the root access with an empty password incident happened 2 months ago. Why these serious but basic problems happen? What is going on with Apple?
- huslage 9y ago(1) is incorrect. Sparsebundles are capable of over-provisioning the storage below them _by design_. They always have been. They are, I believe, APFS snapshots essentially now. This behavior is consistent with most other filesystems with similar constructs. (2) is the real issue here.
- dreamcompiler 9y ago> Sparsebundles are capable of over-provisioning the storage below them _by design_. According to TFA, HFS+ sparsebundles reflect the limitations of their underlying volume, while APFS sparsebundles do not. Seems clear to me that this is a bug.
- praseodym 9y agoSparse bundle disk images [1] are a more efficient way to store disk images on an underlying file system. They are a completely separate concept from the file system in the disk image itself (besides HFS+ apparently doing some intelligent free space reporting) and are definitely not implemented as APFS snapshots. [1] https://en.wikipedia.org/wiki/Sparse_image#Sparse_bundle_disk_images https://en.wikipedia.org/wiki/Sparse_image#Sparse_bundle_dis...
- alwillis 9y agoThis is by far the best explanation of what’s happening at Apple and why we’re noticing these bugs and problems: https://www.imore.com/understanding-apple-scale https://www.imore.com/understanding-apple-scale
- tlo 9y agoMy AFPS volume got corrupted probably during the upgrade to High Sierra. No data lost, but my disk is missing almost half the disk space. See https://apple.stackexchange.com/q/311843/26185 https://apple.stackexchange.com/q/311843/26185
- rangibaby 9y agoI “lost” a lot of space due to time machine local backups. It was frustrating to research and I thought there was something seriously wrong with my computer. Try deleting the local backups and see if you get some space back
- T-N-T 9y agoI had to make room on my MBP SSD to install windows through bootcamp and bashed my head against a wall on the same issue. It took me half an hour to find the reason why so much of my hard drive wasn't available despite me having deleted almost all third party apps and personal data on my macOS partition. Time Machine does 'local backups' and there is NOWHERE in the user interface that fully explains the space they occupy and how to get rid of it. To delete those local backups you need to use the terminal program tmutil. That gave me even more vindication for my move. Also, you really don't know how fast your hardware is until you've used something other than macOS on it. From booting the system to launching software.. everything is snappier now.
- rangibaby 9y agoYes it’s a really poor design decision. They could have at least added “local time machine backups” color to Disk Utility, or added a way to turn them off. Agreed re: snappiness of other OS. Ubuntu flies on my 2013 MBP.
- wila 9y agoThere used to be a way to turn in off before High Sierra, now the only thing you can do is delete the local snapshots. [0] [0] https://forums.macrumors.com/threads/solution-reclaim-storage-back-from-system.2073174/ https://forums.macrumors.com/threads/solution-reclaim-storag...
- hitekker 9y agoYet another reason to stick with El Capitan.
- Apocryphon 9y agoI really want to- but I'm locked out of Xcode 9 if I linger. If only I can run it through some sort of virtualization
- wila 9y agoVMware Fusion supports High Sierra as guest including APFS. I was under the impression that Parallels does the same. Not sure about Virtual Box though.
- deleted 9y ago[deleted]
- BluePAZ 9y agosome things take courage...
- testplzignore 9y ago> To prevent errors when a filesystem inside of a sparse image has more free space than the volume holding the sparse image, HFS+ volumes inside sparse images will report an amount of free space slightly less than the amount of free space on the volume on which image resides. Can anyone explain the "slightly less than" part of this? Why wouldn't it just be "equal to"?
- kalleboo 9y agoTo account for filesystem metadata?
- verisimilitude 9y agoThe sentences following that statement in the hdiutil manual are also helpful: "The image filesystem currently only behaves this way as a result of a direct attach action and will not behave this way if, for example, the filesystem is unmounted and remounted. Moving the image file to a different volume with sufficient free space will allow the image's filesystem to grow to its full size." hdiutil has some of the best man pages I've ever run across.
- praseodym 9y agoAPFS not having block or file-level checksums really seems like a big oversight to me. While the filesystem’s designers considered the hardware-level guarantees to be sufficient [1], this issue shows that there is an entire class of problems that they have not considered. Disk images and loopback-mounted filesystems or even disk-level cloning introduce additional layers of complexity where a filesystem can be silently corrupted, even when the actual physical storage layer is perfectly reliable. A filesystem should be able to last for decades (HFS was designed thirty years ago); I regard not having checksums in a brand new filesystem an over-optimistic tradeoff. [1] http://dtrace.org/blogs/ahl/2016/06/19/apfs-part5/ http://dtrace.org/blogs/ahl/2016/06/19/apfs-part5/
- jchb 9y agoApple HFS+ didn't have file data checksums either. The default filesystem on most Linux distributions, ext4 doesn't, it just stores checksums of the file metadata, not the file data. Same story with Windows NTFS filesystem. Microsofts newer ReFS filesystem has file data checksums disabled by default. So it seems like a tradeoff that most of the major operating system are making. Most likely related to performance. Edit: macOS disk images do have a checksum of the whole image data though. The issue mentioned in the article seems to be caused by an oversight in the disk image helper app, rather than in the APFS filesystem itself.
- temprature 9y agoPerformance is only an issue if your disk can write faster than your CPU can hash. hammer2 changed its hash a couple of years ago because this started happening with newer NVMe drives[0], but before that disk writes weren't CPU-bound. [0] http://lists.dragonflybsd.org/pipermail/commits/2016-June/500610.html http://lists.dragonflybsd.org/pipermail/commits/2016-June/50...
- jchb 9y agoWhat about: * Disk reads may unnecessarily trash the CPU caches, because CPU will need to verify the checksum when the DMA read is done, even if the app isn't going to process the data immediately afterwards * Battery life - without checksums the CPU can stay mostly idle, and go into a lower power mode, while the disk controller does its job
- bonestamp2 9y agoA friend of mine lost a ton of data this week after mac os crashed and he restarted it. The only things he had done since getting the computer a week ago were: 1. update to high sierra 2. copy over files from old mac 3. record about 40gb of screen share data using quicktime (which is what he was doing when it crashed) He spent hours on the phone with apple, the tech said he had never seen anything like it and they weren't able to recover his data... but after reading the other horror stories in this thread there seems to be some serious problems with high sierra and/or APFS.
- benguild 9y agoThat sounds like a failing or faulty SSD. I had this problem on HFS+ on my fairly new 2011 MacBook Air. Downloaded a ton of data, system became unstable and wouldn’t boot after.
- deergomoo 9y agoI had a kernel panic a few weeks ago that left the OS in a state where it was unable to boot. It seemed to think it was mid-upgrade and was complaining about missing the packages an upgrade would read from. Thankfully macOS has an in-place OS reinstall option and as far as I’ve been able to see all my data is totally fine. But it was bizarre and I’ve never experienced anything like it in a decade of using Macs.
- bonestamp2 9y agoEverything running fine now after the OS reinstall?
- bartvk 9y agoLost a ton of data? Doesn't he use Time Machine?
- bonestamp2 9y agoYes, although he was traveling so he wasn't able to recover until he returned home and even then, there was a gap in his backup (his fault) and his screenshare recording wasn't backed up since it was done while traveling. Thankfully he had already sent his presentation ahead to the conference he was presenting at, so he was at least able to complete that obligation. This is a good lesson for everyone... obviously we all know to backup regularly (let Time Machine do its thing multiple times per day of course); but the lesson for me was when traveling, have everything you need backed up on a USB stick at least.
- baxtr 9y agoI get the feeling that many of those who comment didn’t read the article. It says > Note: What I describe below applies to APFS sparse disk images only — ordinary APFS volumes (e.g. your SSD startup disk) are not affected by this problem. While the underlying problem here is very serious, this is not likely to be a widespread problem, and will be most applicable to a small subset of backups.
- eric_lu 9y agoapples quality and design have gone considerably downhill since the decision to get rid of the mac pro and their boner to try to be more like microsoft and stick with flat design. snow leopard was the perfect os RIP
- eric_lu 9y agoalso, https://youtu.be/3spnGnavWFg https://youtu.be/3spnGnavWFg