11 ms·

HFS+ Bit Rot

189 points by adamzegelin 12y ago

billyhoffman 12y ago
Somewhere, John Siracusa's spider sense just twitched
- zw 12y ago
  If a table bell rings to signify such an event and it isn't recorded on a podcast, was it ever actually rung?
- mwfunk 12y ago
  If he ends up being satisfied by Swift, I wonder what happens to his Copland 2010 angst? Does it go away, or does it bandwagon with the filesystem angst? My brain seems to follow a law of Conservation of Pessimism; it just gets redistributed when one of the sources goes away. He may be the same way. Regardless, his cause is true and his dedication to it is admirable.
userbinator 12y ago
Hard drives (of both the magnetic and flash-based variety) all have built-in error detection and correction. If you are getting corrupt files that's not the filesystem's fault, it's most likely a problem with the hardware. Checksums at the FS level are very rare; the majority of the ones in use don't have them (http://en.wikipedia.org/wiki/Comparison_of_file_systems http://en.wikipedia.org/wiki/Comparison_of_file_systems ) and yet they function perfectly fine. HFS+ is not the problem here.
- rcthompson 12y ago
  Are you saying that hard drives store everything on disk with checksums or redundancy? That would be news to me. How else could they correct (or even detect) errors? Anyway, are you arguing that filesystems shouldn't bother with checksumming at all?
  - kevinday 12y ago
    Hard drives store things with varying forms of ECC. Each sector has an ECC field, allowing it to detect many errors and automatically correct some. This isn't a replacement for something better, it just allows simple bit errors to be corrected automatically by the drive. The problem is that it's not really obvious when it's happening, and you only notice when it can't fix something. Drives eventually throw a SMART error when it's had to do too many corrections though.
    - userbinator 12y ago
      Correctable errors are logged by SMART - see the Raw Read Error Rate, Seek Error Rate, Hardware ECC Recovered, and related attributes.
  - Serow225 12y ago
    There are many levels and types of error checking performed in the various layers between the physical signals on the media and the bits that get sent over the drive interface. They are not simple checksums either. Otherwise a modern hard drive would not function...
  - mnw21cam 12y ago
    Most modern media does error checking, and usually error correction as well. This isn't primarily because the media is unreliable, although that is true. The main reason to do ECC is to increase the capacity of the media. Having ECC able to cope with a reasonable number of bit errors allows the bits to be crammed closer together on the media while still being reliable.
- Freaky 12y ago
  > If you are getting corrupt files that's not the filesystem's fault, it's most likely a problem with the hardware. I don't think the implication is that the fs is at fault for the corruption - it's just at fault for failing to detect it. Hardware problems tend toward certainty over long enough time scales - doesn't it make sense to defend against it given the relatively minimal cost of doing so? > Checksums at the FS level are very rare; the majority of the ones in use don't have them .. and yet they function perfectly fine. No, they too allow data to silently become corrupt in face of imperfectly functioning hardware. Sure, it normally doesn't happen, but it's certainly not rare enough to warrant ignoring if your data is in any way valuable to you.
- acdha 12y ago
  The problem is that there are other sources of error - e.g. data corruption in transit rather than on the disk itself – and the legacy methods have error rates which are too high for modern data volumes. There are a couple of implementation problems as well: the lower-level error correction mechanisms tend hide information from the higher-level interfaces, making it hard to measure real error-rates, and some classes of errors aren't randomly distributed and were more likely to produce errors which simple schemes couldn't detect. http://queue.acm.org/detail.cfm?id=1317403 http://queue.acm.org/detail.cfm?id=1317403 is a good article by someone at NetApp describing everything which can go wrong with hard drives, including this class of error. There are two good papers on measured real-world error rates: http://indico.cern.ch/event/13797/session/0/material/paper/1?contribId=3 http://indico.cern.ch/event/13797/session/0/material/paper/1... http://www.cs.toronto.edu/~bianca/papers/fast08.pdf http://www.cs.toronto.edu/~bianca/papers/fast08.pdf The good news is that many of these errors were caught but there are examples which were not and the real message is that the entire stack has enough complexity lurking in it that you wouldn't want to simply assume it handles something as critical as data integrity. Something like the ZFS / brtfs approach is nice because it doesn't depend on all of those layers working as expected, is guaranteed to be monitorable and is much less likely to silently change without notice.
  - jsz0 12y ago
    ZFS / births are great for their intended purposes but using them on devices with limited RAM and/or connected via external bus/power would likely introduce a whole new set of problems.
    - yungchin 12y ago
      What's the problem with having those on external buses?
  - thrownaway2424 12y ago
    I thought one of those links was going to be "Parity Lost and Parity Regained", but no? One of them is by the same authors, on the same topic, from the proceedings of the same conference, but it's a different paper? Weird. http://research.cs.wisc.edu/adsl/Publications/parity-fast08.pdf http://research.cs.wisc.edu/adsl/Publications/parity-fast08....
nandhp 12y ago
> HFS+ is seriously old Sure, in computing terms 1998 is an eon ago. But that's not a good reason to stop using a file system. Lots of older file systems are still in use: FAT32 (1996; still the default for SD cards <32GB and anytime you need a cross-platform filesystem), XFS (1994; the new default filesystem for RHEL7), NTFS (July 1993; WinFS still hasn't materialized), and ext2 (January 1993; still commonly used, particularly in situations where a Journal is not required). Of course, I'm perfectly happy to believe that HFS+ is more badly-designed than any of the other filesystems from that time. But a filesystem doesn't need to be replaced just because it's no longer trendy.
- gtaylor 12y ago
  > But a filesystem doesn't need to be replaced just because it's no longer trendy. The author wasn't just saying that its age was damning, it was the fact that the FS hasn't changed much since 1998. Since XFS was mentioned, I wanted to mention that it's hard to compare to HFS. The author mentions HFS' stagnation, but XFS has seen tons of improvement since 1994. It seems like I'm reading about some cool new XFS development every year. NTFS has made progress since its early years, too.
  - coldtea 12y ago
    >The author wasn't just saying that its age was damning, it was the fact that the FS hasn't changed much since 1998. And yet, he goes to mention tons of changes to the FS over the years, including journaling...
    - zymhan 12y ago
      Yeah, I'm not sure what the difference is between what the author calls "hacks" and others call "updates."
      - jackjeff 12y ago
        Some of these updates are clearly hacks, in the sense that if you did not have to worry about backward compatibility, they would certainly not be implemented in that way. They're not like upgrades from ext2 to ext3, it's like ext2 was modified in a weird way to support new features that older ext2 implementations do not know about. Hard Links for example are implemented by adding files which look a lot like symblinks. As soon as you create hard links to a file, the file is moved into a invisible folder named "Private Data" and becomes a "node file". The original file is replaced by a stub containing the CNID (special type, special content). That's not as efficient as having iNodes and hard links from day 1. If you mount such a volume on Mac OS Classic, the "Private Data" folder is totally accessible and modifiable by the user, and of course hard links do not work. Hot file clustering and journaling are implemented in similar ways to hard links (invisible folders). The opposite example are extended attributes (xattr) which were added in 10.4. A special filesystem structure named the "Attributes File" was reserved in HFS+ from day 1 but never used. De-defragmentation does not rely on any changes to the volume, it's purely done in the Filesystem driver stack. Things like Copy-on-Write or Snapshotting, are probably impossible to implement in a backward compatible manner though...
hosay123 12y ago
I don't know the numbers, but the probability of getting 26 corrupted at-rest files through natural causes sounds pretty much like winning the lottery twice on the same day you were struck by lightning twice Checksums wouldn't have fixed this, they'd only alert the user to the fact the damage had already been done, which is exactly what the decompressor did in its own special way. As another comment points out, error correcting codes are the way to handle this, and its already done in hardware, and probably too expensive to do in software in the general case
- clord 12y ago
  You're right, that reported failure rate is way too low, I'd be glad to only have 26 corrupt files. I have one disk with about 300gb of CR2 files (HFS+). Not long ago I did a similar procedure as in the article (except that I also had the files on a ZFS mirror (I was cleaning up the HFS disk and checking for duplicates before formatting) and found 250 files with bit errors compared to the reference. Fortunately only about 1 in 3 of the bit errors resulted in corrupt files. Needless to say I no longer keep anything of consequence on non-checksummed filesystems now.
  - Buge 12y ago
    It annoys me when someone says "you're right" and then completely disagrees with the first person.
    - vertis 12y ago
      It's easy to read the first persons post either way. That you're lucky they ONLY got 26 corrupt files, or that 26 is extraordinarily unlikely.
      - Buge 12y ago
        I think the "at-rest files through natural causes" gives it away.
- deleted 12y ago
  [deleted]
chuckup 12y ago
I'm surprised how little is done to protect data in modern systems. The two big things we should be doing - ECC ram and checksumming filesystems - are still nowhere close to being mainstream. I recently bought and returned three(!) 3TB drives because I found they were silently corrupting data every 500GB of writes, or so (verified by testing on multiple systems) - I switched to 2TB from another brand, and had zero issues. I only knew there was a problem because I wrote a burn-in script to copy & verify over and over. File system, OS does not care. It's the drive's job. Almost every USB stick I've ever used eventually had some small corruption issue - again, no way to catch this unless you are looking for this kind of thing. Average consumer does not think this is even possible - the idea of your data silently getting scrambled seems impossible, like a car randomly being unable to break - it is just assumed this sort of thing does not happen. But as hard drives get bigger and people put more RAM in their systems I think this will become a huge issue. Of course, consumers will blame "a virus" or something along those lines.
- binarycrusader 12y ago
  It doesn't help that Intel has artificially limited ECC support to specific processors. As the amount of memory in PC increases and the frequency at which components operate, the probability of memory errors increases, yet Intel still foolishly refuses to support ECC in their consumer line selling it as a "server" feature and limiting it to Xeons.
  - chuckup 12y ago
    Linus posted about this a while ago: http://www.realworldtech.com/forum/?threadid=114758&curpostid=114766 http://www.realworldtech.com/forum/?threadid=114758&curposti... "Who the f*ck are they to send me reliability patches, when they can't even get the basics right?" I remember reading Microsoft made a push to get hardware vendors to use ECC Ram with Vista - they recognized a lot of crashes were due to this (but XP would get the blame). No go.
  - theandrewbailey 12y ago
    Perhaps AMD could enable ECC on their consumer CPUs?
myrandomcomment 12y ago
Just ordered this: http://www.ixsystems.com/storage/freenas/ http://www.ixsystems.com/storage/freenas/ ZFS to stop the bit rot! I found the same issues on my music, video and photos collections.
- ajtaylor 12y ago
  I've been using an HP MicroServer (the N54L to be exact) + FreeNAS with great success. My outlay was <$300 since I used disks I already had lying around and the performance is much better than the cheap home-grade NAS products.
- ansible 12y ago
  Note that ZFS can detect the bit rot, but you'll need to actually run some kind of RAID (in this case RAID-Z) to have it fixed automatically. And if you do want bit rot detected on a timely basis, you'll need to have the data on the drives scrubbed on a regular basis (weekly? monthly?, not sure what is sufficient).
  - Freaky 12y ago
    You can set the "copies" property to 2 or 3 even with a single disk, so if one dodgy sector blats one of your files, ZFS can still recover: https://blogs.oracle.com/relling/entry/zfs_copies_and_data_protection https://blogs.oracle.com/relling/entry/zfs_copies_and_data_p...
    - ansible 12y ago
      I didn't know that. Thanks.
- PhantomGremlin 12y ago
  I think it's indisputable that ZFS stops "bit rot". It's an excellent tool for that. However, as a casual observer and long time ZFS wanna-be, I've noticed that the following two issues haven't really gone away: 1) Which version of ZFS? After Sun was assimilated by the Borg and after the great ZFS developer diaspora, everyone seems to have their fingers in the ZFS development pie. There are a plethora of derivative versions. Nothing wrong with that, but when there was a single canonical version there were "many eyes" on it. Bugs were hunted down and squashed. But now? Every "port" to a different OS variant introduces new opportunities for bugs, doesn't it? Is ZFS on FreeBSD as stable as on Solaris? And FreeNAS isn't pure FreeBSD, have they tweaked ZFS? 2) When ZFS is working, it's great. But it doesn't seem to simply fray around the edges. When it fails, it fails catastrophically. I've seen much advice on mailing lists to "restore from backup" after something goes wrong.
ksec 12y ago
The reason why ZFS didn't plan out were simply because it was using too much CPU and Memory. Not something worth considering when the majority of Apple's devices now are Mobile ( Phone / Tablet / Laptop ) It may have been great if Time Capsule, or an Apple NAS uses ZFS. But it seems Apple will likely wants you to move everything to iCloud Drive( Finally! ). I think Apple's new FileSystem will be based entirely for Flash. Something similar to Samsung's F2FS. Since F2FS is GPLv2 license it is not possible for Apple to use it within their own Kernel.
- rodgerd 12y ago
  ZFS isn't particularly consumer-friendly, either. Explaining why you can't delete files from a 100% full filesystem, or the pain and complexity of trying to move from 512 byte to 4096 byte sectors and so on would be a nightmare. And of course the design assumes high-quality hardware (ECC RAM for example).
  - XorNot 12y ago
    The design doesn't assume ECC RAM. It's recommended because you can't talk about end-to-end checksumming and then not explain that it can't proof you against unreliable memory. All other filesystems are equally susceptible - if your memory is getting errors, they'll happily write those to disk too.
    - rodgerd 12y ago
      Actually, XFS checksumming will alert on errors caused by bad RAM.
      - XorNot 12y ago
        Depends what you mean by this. ZFS will alert if it reads data into ram that then doesn't match the checksum during verification. But it can't prevent a bitflip of data in memory from getting written to disk before the checksum is calculated. Nor can it prevent data being bitflipped after its been verified and handed off to the application. Which I'm pretty sure XFS can't do either.
jsz0 12y ago
HFS+ is old an ugly but in practical terms it's good enough. Obviously you cannot rely on any single file system no matter how many mirror/parity disks you throw at it. You need at least one local backup on an independent file system / disks and ideally also an offsite backup. The odds of HFS+ corruption of the same file 3 different file systems are incredibly low.
- lallysingh 12y ago
  Backing up a ZFS volume would have detected the corruption error, giving you a chance to (a) not make a backup of the corrupted version and (b) restore a good version from disk. In the same situation, HFS+ would let you make a backup of corrupted data. If you don't keep all your historical backups, you may end up unwittingly tossing out the last backups with good versions of those files. I'm sorry, I don't understand your last sentence.
  - XorNot 12y ago
    If you're backing up without versions, you're inviting disaster.
rcthompson 12y ago
So, what is a better filesystem that I can use with OS X for storing my data? EXT3 via FUSE driver? NTFS via NTFS-3G driver? ZFS via whichever ZFS-on-OSX fork is the current one for this month? FAT? ExFAT? Or just back it up daily over the network to a Linux server in my closet?
- therealmarv 12y ago
  I agree. What is the alternative? Are the 26 files silently bad or does OS X say about this? Is Ext2-4 better than HFS+, is NTFS better? For sure ZFS is better... but is any of this standard file systems better (NTFS, EXT)? Is there any research? Maybe he would also get 26 bad files with NTFS and EXT3. This whole article is blurry for me.
  - lallysingh 12y ago
    The article described how the photos decoded wrong. Sounds like OS X left them alone without noticing the problem. Instead, the user's attempt to read the files personally (when else do you decide?) detected the errors.
- justincormack 12y ago
  Use a FreeBSD based NAS with ZFS I guess is the best option. Not sure daily backups will help if it is being corrupted, as you will overwrite the good data at some point.
atmosx 12y ago
> HFS+ lost a total of 28 files over the course of 6 years. That's a good number for average users. No one is using HFS+ as a file server. Users who have lots of data use external backup devices (which are prone to HW failure, especially the WD external HDs) or oversight backup services. ps. Anyone used this[1] on macosx? Can it replace HFS+ in the root partition? UPDATE: The faq clearly states that it can't be used as a root partition: Q) Can I boot my computer off of O3X?[edit] A) No. O3X cannot be used as your main system partition. what a pitty :-( [1] https://openzfsonosx.org/ https://openzfsonosx.org/
- klapinat0r 12y ago
  I've used it and still am, in somewhat of an experiment, but with great results: I used to work with ZFS for a living, so I want it to work - maybe that makes me biased. I wrote an article on the wiki[0] on CoreStorage and Encryption together with ZFS and it's been working as expected for a couple of months now. I currently use it to test family/friends backup with SyncThing[1] to see if it can make a (although bit hacky) viable backup solution for the common man, with file history based on routinely snapshots (which will make problems, e.g. how Dropbox explicitly doesn't sync open word documents, how a VMs disk image might not be super great to backup "as is" while in use). As a final note: it can not replace HFS+ for TimeMachine backup either. [0]: https://openzfsonosx.org/wiki/Encryption https://openzfsonosx.org/wiki/Encryption [1]: http://syncthing.net/ http://syncthing.net/
  - atmosx 12y ago
    Great, thanks for the heads up. Do you think that there is any hope to use ZFS as a full-featured replacement in the future for OSX?
    - klapinat0r 12y ago
      I'm not knowledgable enough in that area, but as far as I know the OS X root needs to be HFS. I can only answer by similarity: to boot from Btrfs in ArchLinux you'll need to load kernel modules in the bootloader. Two reason: 1. btrfs isn't "built in" and needs to be loaded. 2. the boot loader needs to be able to read btrfs (obvious, but mentioning for the sake of mentioning) That's doable for btrfs/Arch because we can put our own bootloader with btrfs compatability in. Which leads me to my speculative answer: No. It might be possible with a third party bootloader (iBoot comes to mind), but to have Apple adopt the OpenZFS implementation, or to revive their own, seems unlikely. My guess would be that they are working on something, mainly because they have to find an alternative. Other people have talked about this being a necessary next step for Mac for years, but it does seem plausible that its in the works. Replacement for HFS+ - ZFS? Probably not. Something else? Maybe WWDC 2015-2016. EDIT: In case you mean full-featured as in "ZFS full featured", then yes, that's already there. Not Oracle full-featured, but ZoL (ZFS on Linux port) and somewhat IlluminOS compatible.
zurn 12y ago
> Modern file systems like ZFS, which Apple considered but abandoned as a replacement, include checksums of all meta data structures. That means that when the file is accessed, the filesystem detects the corruption and throws an error. This is exactly backwards - metadata checksums don't protect file contents. They just cover the integrity of the FS so when the FS internals are corrupted, it knows not to write to random places on the disk and can with redudancy can try to recover the metadata.
- alcari 12y ago
  The quote is wrong: ZFS checksums both the user data and metadata in a Merkle tree, optionally using cryptographic hashes (i.e. SHA256).
therealmarv 12y ago
http://www.idt.mdh.se/kurser/ct3340/ht09/ADMINISTRATION/IRCSE09-submissions/ircse09_submission_16.pdf http://www.idt.mdh.se/kurser/ct3340/ht09/ADMINISTRATION/IRCS... Page 8 and 9 of this analysis of file systems. NTFS and Ext3 would not be better. The problem the author describes is more and more a hardware problem. No standard filesystem will automatically repair bad blocks.
therealmarv 12y ago
Ars Technica also has a in deep review of HFS+. It seems HFS+ deeply trusts the hardware in finding errors on disk: http://arstechnica.com/apple/2011/07/mac-os-x-10-7/12/ http://arstechnica.com/apple/2011/07/mac-os-x-10-7/12/ BUT all normal filesystems like NTFS, Ext2-3 are no difference there.
huxley 12y ago
Dominic Giampaolo who did a lot of the work on the BeOS File system has been working for Apple since around 2002: http://www.nobius.org/~dbg/ http://www.nobius.org/~dbg/

HFS+ Bit Rot

189 points by adamzegelin 12y ago

billyhoffman 12y ago

Somewhere, John Siracusa's spider sense just twitched

zw 12y ago
If a table bell rings to signify such an event and it isn't recorded on a podcast, was it ever actually rung?
mwfunk 12y ago
If he ends up being satisfied by Swift, I wonder what happens to his Copland 2010 angst? Does it go away, or does it bandwagon with the filesystem angst? My brain seems to follow a law of Conservation of Pessimism; it just gets redistributed when one of the sources goes away. He may be the same way. Regardless, his cause is true and his dedication to it is admirable.

userbinator 12y ago

Hard drives (of both the magnetic and flash-based variety) all have built-in error detection and correction. If you are getting corrupt files that's not the filesystem's fault, it's most likely a problem with the hardware. Checksums at the FS level are very rare; the majority of the ones in use don't have them (http://en.wikipedia.org/wiki/Comparison_of_file_systems http://en.wikipedia.org/wiki/Comparison_of_file_systems ) and yet they function perfectly fine. HFS+ is not the problem here.

rcthompson 12y ago
Are you saying that hard drives store everything on disk with checksums or redundancy? That would be news to me. How else could they correct (or even detect) errors? Anyway, are you arguing that filesystems shouldn't bother with checksumming at all?
- kevinday 12y ago
  Hard drives store things with varying forms of ECC. Each sector has an ECC field, allowing it to detect many errors and automatically correct some. This isn't a replacement for something better, it just allows simple bit errors to be corrected automatically by the drive. The problem is that it's not really obvious when it's happening, and you only notice when it can't fix something. Drives eventually throw a SMART error when it's had to do too many corrections though.
  - userbinator 12y ago
    Correctable errors are logged by SMART - see the Raw Read Error Rate, Seek Error Rate, Hardware ECC Recovered, and related attributes.
- Serow225 12y ago
  There are many levels and types of error checking performed in the various layers between the physical signals on the media and the bits that get sent over the drive interface. They are not simple checksums either. Otherwise a modern hard drive would not function...
- mnw21cam 12y ago
  Most modern media does error checking, and usually error correction as well. This isn't primarily because the media is unreliable, although that is true. The main reason to do ECC is to increase the capacity of the media. Having ECC able to cope with a reasonable number of bit errors allows the bits to be crammed closer together on the media while still being reliable.
Freaky 12y ago
> If you are getting corrupt files that's not the filesystem's fault, it's most likely a problem with the hardware. I don't think the implication is that the fs is at fault for the corruption - it's just at fault for failing to detect it. Hardware problems tend toward certainty over long enough time scales - doesn't it make sense to defend against it given the relatively minimal cost of doing so? > Checksums at the FS level are very rare; the majority of the ones in use don't have them .. and yet they function perfectly fine. No, they too allow data to silently become corrupt in face of imperfectly functioning hardware. Sure, it normally doesn't happen, but it's certainly not rare enough to warrant ignoring if your data is in any way valuable to you.
acdha 12y ago
The problem is that there are other sources of error - e.g. data corruption in transit rather than on the disk itself – and the legacy methods have error rates which are too high for modern data volumes. There are a couple of implementation problems as well: the lower-level error correction mechanisms tend hide information from the higher-level interfaces, making it hard to measure real error-rates, and some classes of errors aren't randomly distributed and were more likely to produce errors which simple schemes couldn't detect. http://queue.acm.org/detail.cfm?id=1317403 http://queue.acm.org/detail.cfm?id=1317403 is a good article by someone at NetApp describing everything which can go wrong with hard drives, including this class of error. There are two good papers on measured real-world error rates: http://indico.cern.ch/event/13797/session/0/material/paper/1?contribId=3 http://indico.cern.ch/event/13797/session/0/material/paper/1... http://www.cs.toronto.edu/~bianca/papers/fast08.pdf http://www.cs.toronto.edu/~bianca/papers/fast08.pdf The good news is that many of these errors were caught but there are examples which were not and the real message is that the entire stack has enough complexity lurking in it that you wouldn't want to simply assume it handles something as critical as data integrity. Something like the ZFS / brtfs approach is nice because it doesn't depend on all of those layers working as expected, is guaranteed to be monitorable and is much less likely to silently change without notice.
- jsz0 12y ago
  ZFS / births are great for their intended purposes but using them on devices with limited RAM and/or connected via external bus/power would likely introduce a whole new set of problems.
  - yungchin 12y ago
    What's the problem with having those on external buses?
- thrownaway2424 12y ago
  I thought one of those links was going to be "Parity Lost and Parity Regained", but no? One of them is by the same authors, on the same topic, from the proceedings of the same conference, but it's a different paper? Weird. http://research.cs.wisc.edu/adsl/Publications/parity-fast08.pdf http://research.cs.wisc.edu/adsl/Publications/parity-fast08....

nandhp 12y ago

> HFS+ is seriously old Sure, in computing terms 1998 is an eon ago. But that's not a good reason to stop using a file system. Lots of older file systems are still in use: FAT32 (1996; still the default for SD cards <32GB and anytime you need a cross-platform filesystem), XFS (1994; the new default filesystem for RHEL7), NTFS (July 1993; WinFS still hasn't materialized), and ext2 (January 1993; still commonly used, particularly in situations where a Journal is not required). Of course, I'm perfectly happy to believe that HFS+ is more badly-designed than any of the other filesystems from that time. But a filesystem doesn't need to be replaced just because it's no longer trendy.

gtaylor 12y ago
> But a filesystem doesn't need to be replaced just because it's no longer trendy. The author wasn't just saying that its age was damning, it was the fact that the FS hasn't changed much since 1998. Since XFS was mentioned, I wanted to mention that it's hard to compare to HFS. The author mentions HFS' stagnation, but XFS has seen tons of improvement since 1994. It seems like I'm reading about some cool new XFS development every year. NTFS has made progress since its early years, too.
- coldtea 12y ago
  >The author wasn't just saying that its age was damning, it was the fact that the FS hasn't changed much since 1998. And yet, he goes to mention tons of changes to the FS over the years, including journaling...
  - zymhan 12y ago
    Yeah, I'm not sure what the difference is between what the author calls "hacks" and others call "updates."
    - jackjeff 12y ago
      Some of these updates are clearly hacks, in the sense that if you did not have to worry about backward compatibility, they would certainly not be implemented in that way. They're not like upgrades from ext2 to ext3, it's like ext2 was modified in a weird way to support new features that older ext2 implementations do not know about. Hard Links for example are implemented by adding files which look a lot like symblinks. As soon as you create hard links to a file, the file is moved into a invisible folder named "Private Data" and becomes a "node file". The original file is replaced by a stub containing the CNID (special type, special content). That's not as efficient as having iNodes and hard links from day 1. If you mount such a volume on Mac OS Classic, the "Private Data" folder is totally accessible and modifiable by the user, and of course hard links do not work. Hot file clustering and journaling are implemented in similar ways to hard links (invisible folders). The opposite example are extended attributes (xattr) which were added in 10.4. A special filesystem structure named the "Attributes File" was reserved in HFS+ from day 1 but never used. De-defragmentation does not rely on any changes to the volume, it's purely done in the Filesystem driver stack. Things like Copy-on-Write or Snapshotting, are probably impossible to implement in a backward compatible manner though...

hosay123 12y ago

I don't know the numbers, but the probability of getting 26 corrupted at-rest files through natural causes sounds pretty much like winning the lottery twice on the same day you were struck by lightning twice Checksums wouldn't have fixed this, they'd only alert the user to the fact the damage had already been done, which is exactly what the decompressor did in its own special way. As another comment points out, error correcting codes are the way to handle this, and its already done in hardware, and probably too expensive to do in software in the general case

clord 12y ago
You're right, that reported failure rate is way too low, I'd be glad to only have 26 corrupt files. I have one disk with about 300gb of CR2 files (HFS+). Not long ago I did a similar procedure as in the article (except that I also had the files on a ZFS mirror (I was cleaning up the HFS disk and checking for duplicates before formatting) and found 250 files with bit errors compared to the reference. Fortunately only about 1 in 3 of the bit errors resulted in corrupt files. Needless to say I no longer keep anything of consequence on non-checksummed filesystems now.
- Buge 12y ago
  It annoys me when someone says "you're right" and then completely disagrees with the first person.
  - vertis 12y ago
    It's easy to read the first persons post either way. That you're lucky they ONLY got 26 corrupt files, or that 26 is extraordinarily unlikely.
    - Buge 12y ago
      I think the "at-rest files through natural causes" gives it away.
deleted 12y ago
[deleted]

chuckup 12y ago

I'm surprised how little is done to protect data in modern systems. The two big things we should be doing - ECC ram and checksumming filesystems - are still nowhere close to being mainstream. I recently bought and returned three(!) 3TB drives because I found they were silently corrupting data every 500GB of writes, or so (verified by testing on multiple systems) - I switched to 2TB from another brand, and had zero issues. I only knew there was a problem because I wrote a burn-in script to copy & verify over and over. File system, OS does not care. It's the drive's job. Almost every USB stick I've ever used eventually had some small corruption issue - again, no way to catch this unless you are looking for this kind of thing. Average consumer does not think this is even possible - the idea of your data silently getting scrambled seems impossible, like a car randomly being unable to break - it is just assumed this sort of thing does not happen. But as hard drives get bigger and people put more RAM in their systems I think this will become a huge issue. Of course, consumers will blame "a virus" or something along those lines.

binarycrusader 12y ago
It doesn't help that Intel has artificially limited ECC support to specific processors. As the amount of memory in PC increases and the frequency at which components operate, the probability of memory errors increases, yet Intel still foolishly refuses to support ECC in their consumer line selling it as a "server" feature and limiting it to Xeons.
- chuckup 12y ago
  Linus posted about this a while ago: http://www.realworldtech.com/forum/?threadid=114758&curpostid=114766 http://www.realworldtech.com/forum/?threadid=114758&curposti... "Who the f*ck are they to send me reliability patches, when they can't even get the basics right?" I remember reading Microsoft made a push to get hardware vendors to use ECC Ram with Vista - they recognized a lot of crashes were due to this (but XP would get the blame). No go.
- theandrewbailey 12y ago
  Perhaps AMD could enable ECC on their consumer CPUs?

myrandomcomment 12y ago

Just ordered this: http://www.ixsystems.com/storage/freenas/ http://www.ixsystems.com/storage/freenas/ ZFS to stop the bit rot! I found the same issues on my music, video and photos collections.

ajtaylor 12y ago
I've been using an HP MicroServer (the N54L to be exact) + FreeNAS with great success. My outlay was <$300 since I used disks I already had lying around and the performance is much better than the cheap home-grade NAS products.
ansible 12y ago
Note that ZFS can detect the bit rot, but you'll need to actually run some kind of RAID (in this case RAID-Z) to have it fixed automatically. And if you do want bit rot detected on a timely basis, you'll need to have the data on the drives scrubbed on a regular basis (weekly? monthly?, not sure what is sufficient).
- Freaky 12y ago
  You can set the "copies" property to 2 or 3 even with a single disk, so if one dodgy sector blats one of your files, ZFS can still recover: https://blogs.oracle.com/relling/entry/zfs_copies_and_data_protection https://blogs.oracle.com/relling/entry/zfs_copies_and_data_p...
  - ansible 12y ago
    I didn't know that. Thanks.
PhantomGremlin 12y ago
I think it's indisputable that ZFS stops "bit rot". It's an excellent tool for that. However, as a casual observer and long time ZFS wanna-be, I've noticed that the following two issues haven't really gone away: 1) Which version of ZFS? After Sun was assimilated by the Borg and after the great ZFS developer diaspora, everyone seems to have their fingers in the ZFS development pie. There are a plethora of derivative versions. Nothing wrong with that, but when there was a single canonical version there were "many eyes" on it. Bugs were hunted down and squashed. But now? Every "port" to a different OS variant introduces new opportunities for bugs, doesn't it? Is ZFS on FreeBSD as stable as on Solaris? And FreeNAS isn't pure FreeBSD, have they tweaked ZFS? 2) When ZFS is working, it's great. But it doesn't seem to simply fray around the edges. When it fails, it fails catastrophically. I've seen much advice on mailing lists to "restore from backup" after something goes wrong.

ksec 12y ago

The reason why ZFS didn't plan out were simply because it was using too much CPU and Memory. Not something worth considering when the majority of Apple's devices now are Mobile ( Phone / Tablet / Laptop ) It may have been great if Time Capsule, or an Apple NAS uses ZFS. But it seems Apple will likely wants you to move everything to iCloud Drive( Finally! ). I think Apple's new FileSystem will be based entirely for Flash. Something similar to Samsung's F2FS. Since F2FS is GPLv2 license it is not possible for Apple to use it within their own Kernel.

rodgerd 12y ago
ZFS isn't particularly consumer-friendly, either. Explaining why you can't delete files from a 100% full filesystem, or the pain and complexity of trying to move from 512 byte to 4096 byte sectors and so on would be a nightmare. And of course the design assumes high-quality hardware (ECC RAM for example).
- XorNot 12y ago
  The design doesn't assume ECC RAM. It's recommended because you can't talk about end-to-end checksumming and then not explain that it can't proof you against unreliable memory. All other filesystems are equally susceptible - if your memory is getting errors, they'll happily write those to disk too.
  - rodgerd 12y ago
    Actually, XFS checksumming will alert on errors caused by bad RAM.
    - XorNot 12y ago
      Depends what you mean by this. ZFS will alert if it reads data into ram that then doesn't match the checksum during verification. But it can't prevent a bitflip of data in memory from getting written to disk before the checksum is calculated. Nor can it prevent data being bitflipped after its been verified and handed off to the application. Which I'm pretty sure XFS can't do either.

jsz0 12y ago

HFS+ is old an ugly but in practical terms it's good enough. Obviously you cannot rely on any single file system no matter how many mirror/parity disks you throw at it. You need at least one local backup on an independent file system / disks and ideally also an offsite backup. The odds of HFS+ corruption of the same file 3 different file systems are incredibly low.

lallysingh 12y ago
Backing up a ZFS volume would have detected the corruption error, giving you a chance to (a) not make a backup of the corrupted version and (b) restore a good version from disk. In the same situation, HFS+ would let you make a backup of corrupted data. If you don't keep all your historical backups, you may end up unwittingly tossing out the last backups with good versions of those files. I'm sorry, I don't understand your last sentence.
- XorNot 12y ago
  If you're backing up without versions, you're inviting disaster.

rcthompson 12y ago

So, what is a better filesystem that I can use with OS X for storing my data? EXT3 via FUSE driver? NTFS via NTFS-3G driver? ZFS via whichever ZFS-on-OSX fork is the current one for this month? FAT? ExFAT? Or just back it up daily over the network to a Linux server in my closet?

therealmarv 12y ago
I agree. What is the alternative? Are the 26 files silently bad or does OS X say about this? Is Ext2-4 better than HFS+, is NTFS better? For sure ZFS is better... but is any of this standard file systems better (NTFS, EXT)? Is there any research? Maybe he would also get 26 bad files with NTFS and EXT3. This whole article is blurry for me.
- lallysingh 12y ago
  The article described how the photos decoded wrong. Sounds like OS X left them alone without noticing the problem. Instead, the user's attempt to read the files personally (when else do you decide?) detected the errors.
justincormack 12y ago
Use a FreeBSD based NAS with ZFS I guess is the best option. Not sure daily backups will help if it is being corrupted, as you will overwrite the good data at some point.

atmosx 12y ago

> HFS+ lost a total of 28 files over the course of 6 years. That's a good number for average users. No one is using HFS+ as a file server. Users who have lots of data use external backup devices (which are prone to HW failure, especially the WD external HDs) or oversight backup services. ps. Anyone used this[1] on macosx? Can it replace HFS+ in the root partition? UPDATE: The faq clearly states that it can't be used as a root partition: Q) Can I boot my computer off of O3X?[edit] A) No. O3X cannot be used as your main system partition. what a pitty :-( [1] https://openzfsonosx.org/ https://openzfsonosx.org/

klapinat0r 12y ago
I've used it and still am, in somewhat of an experiment, but with great results: I used to work with ZFS for a living, so I want it to work - maybe that makes me biased. I wrote an article on the wiki[0] on CoreStorage and Encryption together with ZFS and it's been working as expected for a couple of months now. I currently use it to test family/friends backup with SyncThing[1] to see if it can make a (although bit hacky) viable backup solution for the common man, with file history based on routinely snapshots (which will make problems, e.g. how Dropbox explicitly doesn't sync open word documents, how a VMs disk image might not be super great to backup "as is" while in use). As a final note: it can not replace HFS+ for TimeMachine backup either. [0]: https://openzfsonosx.org/wiki/Encryption https://openzfsonosx.org/wiki/Encryption [1]: http://syncthing.net/ http://syncthing.net/
- atmosx 12y ago
  Great, thanks for the heads up. Do you think that there is any hope to use ZFS as a full-featured replacement in the future for OSX?
  - klapinat0r 12y ago
    I'm not knowledgable enough in that area, but as far as I know the OS X root needs to be HFS. I can only answer by similarity: to boot from Btrfs in ArchLinux you'll need to load kernel modules in the bootloader. Two reason: 1. btrfs isn't "built in" and needs to be loaded. 2. the boot loader needs to be able to read btrfs (obvious, but mentioning for the sake of mentioning) That's doable for btrfs/Arch because we can put our own bootloader with btrfs compatability in. Which leads me to my speculative answer: No. It might be possible with a third party bootloader (iBoot comes to mind), but to have Apple adopt the OpenZFS implementation, or to revive their own, seems unlikely. My guess would be that they are working on something, mainly because they have to find an alternative. Other people have talked about this being a necessary next step for Mac for years, but it does seem plausible that its in the works. Replacement for HFS+ - ZFS? Probably not. Something else? Maybe WWDC 2015-2016. EDIT: In case you mean full-featured as in "ZFS full featured", then yes, that's already there. Not Oracle full-featured, but ZoL (ZFS on Linux port) and somewhat IlluminOS compatible.

zurn 12y ago

> Modern file systems like ZFS, which Apple considered but abandoned as a replacement, include checksums of all meta data structures. That means that when the file is accessed, the filesystem detects the corruption and throws an error. This is exactly backwards - metadata checksums don't protect file contents. They just cover the integrity of the FS so when the FS internals are corrupted, it knows not to write to random places on the disk and can with redudancy can try to recover the metadata.

alcari 12y ago
The quote is wrong: ZFS checksums both the user data and metadata in a Merkle tree, optionally using cryptographic hashes (i.e. SHA256).

therealmarv 12y ago

http://www.idt.mdh.se/kurser/ct3340/ht09/ADMINISTRATION/IRCSE09-submissions/ircse09_submission_16.pdf http://www.idt.mdh.se/kurser/ct3340/ht09/ADMINISTRATION/IRCS... Page 8 and 9 of this analysis of file systems. NTFS and Ext3 would not be better. The problem the author describes is more and more a hardware problem. No standard filesystem will automatically repair bad blocks.

therealmarv 12y ago

Ars Technica also has a in deep review of HFS+. It seems HFS+ deeply trusts the hardware in finding errors on disk: http://arstechnica.com/apple/2011/07/mac-os-x-10-7/12/ http://arstechnica.com/apple/2011/07/mac-os-x-10-7/12/ BUT all normal filesystems like NTFS, Ext2-3 are no difference there.

huxley 12y ago

Dominic Giampaolo who did a lot of the work on the BeOS File system has been working for Apple since around 2002: http://www.nobius.org/~dbg/ http://www.nobius.org/~dbg/