27 ms·
Apple’s new APFS file system
- comex 10y agoNitpick: You can, in fact, create new folders in the iCloud Drive app on iOS. For some reason it's hidden under the Select option.
- bensummers 10y agoAPFS doesn't do proper checksumming on data, and won't mirror or RAID, so it doesn't have the data recovery properties implied by the article. http://dtrace.org/blogs/ahl/2016/06/19/apfs-part5/ http://dtrace.org/blogs/ahl/2016/06/19/apfs-part5/
- dilap 10y agoWell, that's disappointing to hear. I've run into multiple MP3s bit-rotting on my MB air over several years. Perhaps soley the fault of HFS+, but trusting the underlying hardware just doesn't seem like a good idea.
- itsoggy 10y agoYep, I had a similar thing with audio files corrupting on my old MacBook, I was using time machine to backup and guess what.... the copy in the backup was fried too! Switched back to vinyl for my music, my 20 year old Texhnics 1210 will probably out last my current laptop and the next!
- roryisok 10y agoI smell a hipster.
- amdavidson 10y agoYou switched to a playback means that is inherently destructive to the media as a means to protect against data loss?
- DugFin 10y agoThe physical vinyl won't outlast your digital, it'll just fail more gracefully, slowly losing fidelity every time you scrape it with a needle. You're just replacing a small chance of catastrophic failure with a guaranteed gradual failure.
- semi-extrinsic 10y agoWikipedia says "Record wear can be reduced to virtual insignificance, however, by the use of a high-quality, correctly adjusted turntable and tonearm, a high-compliance magnetic cartridge with a high-end stylus in good condition, and careful record handling, with non-abrasive removal of dust before playing and other cleaning if necessary." In other words, record wear is negligible if you buy good equipment and take good care of it. Just like the risk of bit rot, catastrophic crashes etc. is negligible if you buy good storage equipment and take good care of your data (good backups etc.) Perhaps the one big advantage of vinyl over digital is that the shelf life of unused vinyl is larger than a human lifetime. A disk stored on a shelf, however, can suffer damage in many ways and is guaranteed to be very hard to connect to your computer in ten-twenty years.
- itsoggy 10y agoWith a little calibration of the TT once in a while it will esily outlast me!
- Symbiote 10y agoI add parity archives to add some redundancy to my photographs. I've done it for years, but I think it's useful not to rely on a filesystem to handle this. For example: par2create -r5 -n2 example.par2 *.jpg creates two files, between them giving 5% redundancy. I think that should be more than enough to repair bitrot within a file, but depending how many photos there are, losing a whole file could be more than 5%. par2verify example.par2 will verify, and par2repair will repair corrupted or missing files. https://en.wikipedia.org/wiki/Parchive https://en.wikipedia.org/wiki/Parchive
- indolering 10y agoIt would be neat if there was a parity scheme fast enough to preserve 2% of of all files on disk. It could even be tucked away behind savings from file-level compression.
- mfukar 10y ago> it's useful not to rely on a filesystem to handle this In what way?
- wtallis 10y agoRelatively few filesystems offer thorough data checksumming. Hardly any offer erasure coding. RAID at the filesystem layer is a bit more common, but also more inconvenient. Doing erasure coding at the file archive level rather than in the filesystem gives you the freedom to move your archives onto standard everyday filesystems and devices without silently losing the protection.
- jen20 10y agoYou don't need _a lot_ of filesystems to offer it - just the one you use - and if this is important to you, just use ZFS.
- rovr138 10y agoWhat if I use multiple computers and multiple operating systems want to be able to work on this data on them. Par2 would allow me to create the needed data on any computer and test it on any computer. The other alternative is having a server that's up and running all the time, exposed to the internet (or complicate the setup with a VPN), so I can sync. Operations would take a long time (via the internet) or I would anyway need to transfer the data to my computer, work on it, sync it back. During this time, any protections that ZFS offers are null since anything could happen in my computer and I can't test for it locally. ZFS is great. But it's not the answer to everything.
- beagle3 10y agoDamaged files (that can still be read from disk without error) are more likely than not a result of bad memory. It is incredibly unlikely to have bits mutated on disk without triggering a CRC error -- approx 1 in 4 billion errors would go undetected in a CRC-32; and it has to be spread over more than 32-bits-wide. The fact that you had multiple damaged files basically guarantees that it was a faulty memory (or bus, or controller -- but not actual magnetic media corruption) issue. ZFS can't really help you with that as the data was likely damaged in transit rather than on disk; though with its own 256 bit hash, it is likely to detect those faulty system components earlier than later.
- pstadler 10y agoTo mitigate the likeliness of damage during transit, it's recommened to use ECC memory with ZFS. I had FreeNAS running on a system with a faulty memory module. ZFS correctly discovered damage in random files I just transferred over. Luckily, it wasn't too late to replace the bad module and repeat the transfer. Lessons learned: - Run MemTest86 on hardware before using it as storage - Use an FS that does checksums Unfortunately, the latter doesn't seem to apply to APFS.
- voidz 10y agoCan you always just buy ECC memory (I have DDR3, I guess it's more affordable now since it's a little bit older), or does something else (cpu, motherboard) need to support it as well?
- pstadler 10y agoYou need a compatible motherboard and CPU.
- tw04 10y agoWhich is why Intel kinda sucks. It would cost them basically nothing to have ECC enabled on all of their hardware, but they insist on using it as a differentiator between server and desktop parts. AMD (at least in the past), includes it on all parts so that it's up to the consumer to choose.
- hyperknot 10y agoIsn't it possible that Apple developed this filesystem primarily to be used with their SSD based devices which are using their custom controllers, which simply takes care of this low level protection? SSD controllers are said to be very complicated piece of proprietary technologies, and I believe they are using sophisticated algorithms for integrity protection.
- vetinari 10y agoApple users use external (USB, Thunderbolt) drives too, where they would not get this special hypothetical protection.
- justinjlynn 10y agoClearly telegraphing Apple's intent to release new "Apple approved (and licensed)" storage "accessories" for their systems.
- toyg 10y agoLock-in all the things!
- justinjlynn 10y agoIndeed, sure would be a shame for anything to happen to your data because you chose not to use drives equipped with iStoreSafe(tm) technology by Apple(tm).
- kevin_thibedeau 10y agoBrings back the days when Mac users were paying a 3x markup for standard SCSI drives.
- deleted 10y ago[deleted]
- agildehaus 10y agoUnfortunately bit rot is not something APFS addresses as it only checksums metadata. Wonder if this is something Apple can add to APFS in the future?
- JumpCrisscross 10y agoI assume bit rot isn't an issue on devices running APFS. They're designed to be continuously backed up. EDIT: never mind; didn't realize iCloud didn't keep historical copies for long.
- izacus 10y agoBackup doesn't do anything about bitrot. Worse, it just propagates and compounds it over time.
- razzmatazz 10y agoBackup will not fix your bitrotted files by itself. It will just propagate silently into the future where you don't have access to historically correct files anymore.
- shouldbworking 10y agoI hope. It wouldn't be such a big issue if Intel stopped stripping ECC from desktop CPU's. Traditionally bad ram causes more bit rot than damaged drives. Ram get less reliable every die shrink, everything should be ECC by now.
- DaiPlusPlus 10y agoCPU ECC is for RAM, not storage - checksums/parity is not the concern of the CPU at all. Though we're talking about Optane, then that's different. In the future (with solid OS support), Optane will be both RAM and non-volatile storage - and you really don't want data corruption there. Has Intel announced anything regarding ECC, parity or other systems in-place to protect against bitrot in Optane?
- 10y ago
- anton_gogolev 10y agoBit rot is absolutely not about SSD degradation.
- hoschicz 10y agoHow did the backup software detect it then? Or is the author falsely calling SSD degradation bit-rot
- tyfon 10y agoMost likely, the bit flipped in ram and was bad in transit to the SSD
- cmurf 10y agoHard to say without dmesg, but it could be metadata corruption and in the ensuing confusion backupd gives up; or it could be an explicit read error report by the drive which would result in an I/O error to backupd and it'd likewise give up.
- hackuser 10y agoThe author links to a much more technically informative article: https://www.backblaze.com/blog/apfs-apple-file-system/ https://www.backblaze.com/blog/apfs-apple-file-system/
- robertlagrant 10y agoWhat on earth is nanosecond data resolution? Does it just means file create/modify timestamps are saved with that level of precision? If so, who cares about that?
- aembleton 10y agoApparently it's important for tracking the order of operations. I don't know why a ledger couldn't be kept. https://arstechnica.com/apple/2016/06/digging-into-the-dev-documentation-for-apfs-apples-new-file-system/ https://arstechnica.com/apple/2016/06/digging-into-the-dev-d...
- phaemon 10y agoHah, you were too quick for me! Must type faster!
- moomin 10y agoIf it's anything like systems I'm familiar with, it is a ledger: if two things happen at exactly the same time, one happens a nanosecond later.
- simonh 10y agoA ledger is a separate write that slows down I/O and uses up disk space. Any way the question should by why a ledger wasn't used, not why it couldn't. Of course it could, but it's not necessarily desirable.
- phaemon 10y agoYes, exactly that. I had to follow a few links, but Ars Technica has a better story. To quote: "It also massively increases the granularity of object time-stamping: APFS supports nanosecond time stamp granularity rather than the 1-second time stamp granularity in HFS+. Nanosecond timestamps are important in a modern file system because they help with atomicity—in a file system that keeps a record of writes, having nanosecond-level granularity is important in tracking the order of operations." Full thing is at: https://arstechnica.com/apple/2016/06/digging-into-the-dev-documentation-for-apfs-apples-new-file-system/ https://arstechnica.com/apple/2016/06/digging-into-the-dev-d...
- dingo_bat 10y ago> A director of the largest PC OS licensing company once told me they spent more money on driver development and fixes than on OS code proper. Funny way to refer to Microsoft.
- newjersey 10y agoIsn't this true of gnu/Linux as well though? A lot of patches are drivers and fixes? For example, one could argue trim for solid state storage is a driver issue?
- dingo_bat 10y agoOf course it is. Drivers will be the majority of the work in any OS that is not locked onto a very limited set of hardware.
- tpaschalis 10y agoI think this is the case with both Windows and the Linux base distros. It's a necessity when you need to support a wide array of hardware, but might be different with Apple's ecosystem.
- eridius 10y agoCan we please change the title? The current one sounds like an inflammatory anti-Apple article, and the only reason I even clicked it was in expectation of a good laugh.
- rubatuga 10y agoIt's a rule not to change the titles when posting, unless there's a very good reason not to do so
- teach 10y agoThe current guideline is "please use the original title, unless it is misleading or linkbait." I probably would have left the original title alone, personally, but you could make a case that it was misleading. HN moderators apparently agreed and have since changed it.
- deleted 10y ago[deleted]
- n1000 10y agoSince there seem to be experts here in the discussion: I keep thinking about a little toy open-source project to adress bitrot in TimeMachine backups. A utility that compares chekcksums of the original file against its backup. Apparently Apple started using hashes internally in Time Machine and it would probably be simplest to use those. However, I cannot find any documentation on the internals of Time Machine. Does anyone know where to start looking? Would such a tool make sense at all?
- RJIb8RBYxzAMX9u 10y agotmutil verifychecksums should already do this.
- n1000 10y agoAs far as I understand it, this will verify checksums within the time machine backup by comparing hashes with the ones that were created during backup. However, it will not compare the files against the ones on the system. So it protecte against bitrot on the backup disk only.
- RJIb8RBYxzAMX9u 10y agoWell, you'll need some kind of heuristic to figure out if a mismatch is due to bit-rot, or because the working file's changed. Maybe modification date, but a) what if a file's restored from an older backup, b) mtime itself is corrupted (APFS would guard against this I suppose). In any case, the "hash" appears to be CRC32, stored in extended attributes: $ xattr .inputrc com.apple.finder.copy.source.checksum#N com.apple.metadata:_kTimeMachineNewestSnapshot com.apple.metadata:_kTimeMachineOldestSnapshot $ xattr -px 'com.apple.finder.copy.source.checksum#N' .inputrc 26 E5 4A AB $ cksum .inputrc 2873812262 65 .inputrc $ printf '%x\n' "$(cksum .inputrc | cut -d ' ' -f 1)" ab4ae526
- twohearted 10y agoI really like this idea. It's a relatively simple approach that could work with any fs/backup strategy. It would be great if it wasn't dependent on Time Machine. ZFS/Raid + ECC ram are complicated and expensive and are not even a 100% guarantee against bit rot.
- TonyHou 10y ago16GB iPhone users care!
- rubatuga 10y agoAddressing data integrity is a slippery slope of comparing hard drive bit read errors and ECC ram bit flips. The reality is, for things like video and pictures it probably won't be a problem. However, if the data is sensitive, you should be using a raid 6 with 2 disk redundancy, coupled with constant data scrubbing to ensure that the disks remain bit rot free. The probability of bit rot occurring at the same place is probably less than 1 in 10^16, possibly even less if data scrubbing is constant. Here's a good article addressing bit rot: https://lockss.org/locksswiki/files/ACM2010.pdf https://lockss.org/locksswiki/files/ACM2010.pdf
- roymurdock 10y agoReally interesting to think about how these guys model out reliability estimates and come up with such crazy numbers. Thanks for the share.
- acdha 10y ago> The reality is, for things like video and pictures it probably won't be a problem. However, if the data is sensitive, you should be using a raid 6 with 2 disk redundancy, coupled with constant data scrubbing to ensure that the disks remain bit rot free. The probability of bit rot occurring at the same place is probably less than 1 in 10^16, possibly even less if data scrubbing is constant. The first assertion is simply wrong: the internet is littered with people who have corrupt media files which cause user-visible errors. Beyond that, however, RAID can be part of a strategy but is far from sufficient. The paper you linked provides a good discussion of the reasons why but the root mistake is assuming that failures are perfectly distributed random events at a single point. End-to-end strong integrity checks are so important because e.g. in that scenario with a two-drive RAID array you'd still be subject to errors caused transferring the data from the host to the RAID controller, on the controller itself, in communication to the drives (one reason why it's important to mix vendors & models), and on the path back when reading data. You'd also have to worry about corruption in memory, errors caused by e.g. power fluctuation or other hardware events which may be grossly underestimated because they're silently detected and retried until you get lucky and one of the 100,000 errors happened to generate a valid checksum, etc. Generating a cryptographic hash as early as possible and checking it at every stage is key both because it catches almost all of those scenarios but also because it allows you to confidently report a positive validation result rather than just the absence of detected errors.
- HappyTypist 10y agoWhat's the best way to protect myself against bit rot, as someone with extremely slow upload speeds?
- deleted 10y ago[deleted]
- boomboomsubban 10y agoThe best way is using ZFS with parity.
- spilk 10y agoThe most interesting part of this article (for me at least) are those early Mac screenshots - The Flounder? and that next one with not-quite-Chicago/Geneva fonts in the Finder? Where did those come from? I realize JLG had been with Apple since the early days of Mac development but I've never seen anything like those before.
- joezydeco 10y agoJean-Louis kind of speaks incorrectly here when he says "Very old-timers will remember the pre-Mac File system called The Flounder", unless by old-timers he means Apple employees that were on the Mac team. There's a little more about the early Mac software history at Folklore.org, including another screen shot of what eventually became Finder: http://www.folklore.org/StoryView.py?project=Macintosh&story=Early_Demos.txt http://www.folklore.org/StoryView.py?project=Macintosh&story... http://www.folklore.org/StoryView.py?project=Macintosh&story=Busy_Being_Born,_Part_2.txt http://www.folklore.org/StoryView.py?project=Macintosh&story...
- rbanffy 10y agoFunny... I converted Cream 12 for use in Apple II programs. For the old-timers, it was a shape table and multiple draw commands of numbered (ascii value - 32) shapes would allow you to write proportionally spaced text to the graphical screen. It was the same format Take-1 used (we used their programmer's toolkit). So, Cream 12 came down from Altos all the way to our Apple II+'s.
- nsxwolf 10y agoYou can see a lot of this stuff if you read through the early Mac team stories on http://www.folklore.org http://www.folklore.org.
- mark_l_watson 10y agoHe is joking with the title. Money quote: "encryption will be easier to use, disk space will be better utilized, backups will be more reliable" Both my iPhone 7 and my iPad Pro only have 32G of storage and the new file system gave me back extra storage due to better utilization. It was interesting how smoothly the transition went.
- bluejekyll 10y ago> the “iOS First” release once again speaks to Apple’s priorities, to its vision of its future I actually disagree about the reasoning behind this strategy. iOS may be more important to Apple at the moment, in terms of market value, but there's a better reason to target it first. iOS is deployed to constrained systems, iPhone, iPad, appleTV... this is easier to test because there will be less variance to deal with than on macOS deployments. It makes a ton of sense to start there for technical reasons so that they can validate it in more consistent environments.
- Someone 10y agoAlso, they _know_ that many of those devices are backed up regularly to iCloud (they may know which ones were recently sync’ed with Mac, too). That diminishes the risks even further.
- Karunamon 10y agoI'm not so sure.. if you look at Apple's direction over the last ten years or so, you see a lot of convergence happening. Off the top of my head: * Siri on the desktop * Gatekeeper getting more restrictive * SIP * Neglect of pro users * New stuff showing up on iOS first
- bluejekyll 10y agoI was speaking specifically of this filesystem released to the wild, not about other new features outside of this context.
- eric_h 10y agoI'd also like to mention that it is extremely impressive that Apple was able to transparently swap out the filesystem, with no data loss, on millions of devices. I've not heard of anyone who had serious issues with the update. It was an incredible technical achievement.
- linkmotif 10y agoYeah that's amazing. More generally, Apple was providing working OS upgrades for years while the notion of upgrading Windows between major versions was basically laughable. Wasn't Windows 8 the first version of Windows that you could actually upgrade to and have the system boot successfully? Maybe Vista to 7 worked for people. I never bothered trying. But I remember upgrading from OS X 10.3 to 10.4 and being floored when everything just worked.
- Exuma 10y agoWhat is the release date for APFS on MacOS?
- pmarreck 10y agoFor those who don't know of him, Jean-Louis Gassée (the author here) is a great. Check out some of his quotes: http://www.azquotes.com/author/38975-Jean_Louis_Gassee http://www.azquotes.com/author/38975-Jean_Louis_Gassee my favorite probably being http://www.azquotes.com/quote/754436 http://www.azquotes.com/quote/754436 about Apple Computer. He also invented the legendary BeOS.
- luckydude 10y agoIsn't the file system guy from BeOS the guy behind the apple fs? Yeah, he's referenced it's Dominic Giampaolo. Not sure how much coding he did but it says he's one of the architects. BeFS was way way ahead of its time. Really a shame that OS didn't get picked up by more people.
- savoytruffle 10y agoI think he's also credited as the father of the add-on filesystem journaling that was enabled on consumer OS X 10.2.2 back in fall 2002, and many of the shims and patches that have kept HFS+ going to this day.
- rs86 10y agoThis article is so boring I considered killing myself
- mehrdada 10y agoWorth noting that iOS HFS+ already had some non-standard features [hacks] that Mac did not use like encrypting files with different keys based on the data protection level—that APFS provides—so it makes sense to upgrade iOS first, which can readily leverage such functionality, and it not necessarily a direct result of it being the "vision for the future".