10 ms·

What terrifies me (as a MacBook Pro owner) is the recent realisation (due to some random corruption of iphoto data, and yes I have a backup device to restore fr

What terrifies me (as a MacBook Pro owner) is the recent realisation (due to some random corruption of iphoto data, and yes I have a backup device to restore from) that OS X has a fundamentally unreliable filesystem sitting on top of ever-expanding storage devices that many of us are depending on to archive our photos and other data. I don't understand why Apple keeps pushing the capabilities of the system further and their hardware but haven't yet addressed the critical data storage issues that are eating away at the edges.

the_mitsuhiko 12y ago
> OS X has a fundamentally unreliable filesystem Unfortunately so is every other filesystem. Which is also why Apple heavily emphasizes making backups.
- dman 12y ago
  Have you looked at ZFS?
  - coldtea 12y ago
    Have YOU looked at ZFS, and how it's unfit for desktop use?
    - laumars 11y ago
      That's unsubstantiated FUD. You only have to glance through this thread to see that there are people (myself included) who run ZFS on their desktop machine (laptop in my case). People love to over exaggerate the RAM requirements for ZFS because they see build logs of enterprise-grade storage servers. But you wouldn't expect that kind of throughput on your desktop systems even with lighter file systems. And most desktops have their 64bit CPU sat idle for most of the time, so there's not even an argument for the additional instruction overhead. Obviously the real crux of argument is "what do you primarily use your desktop for"? If it's gaming, then there's little point running ZFS since you're going to be disinterested in the benefits of ZFS anyway, so it's not worth the marginal overhead. However if your desktop is a development machine, used for multimedia authoring or even just an internet terminal (like most PCs are these days), then ZFS is a viable option.
      - coldtea 11y ago
        >And most desktops have their 64bit CPU sat idle for most of the time, so there's not even an argument for the additional instruction overhead. Most "desktops" today are in fact laptops, and the reason you want to have the CPU sit idle is battery time. >However if your desktop is a development machine, used for multimedia authoring or even just an internet terminal (like most PCs are these days), then ZFS is a viable option. If you're not pooling disks together but just have the built in disk, and usb hard disks that you conenct occasionally you don't get much if anything from ZFS, but you get the penalties it incurs... And it's not a substitute for backing up your disks either...
        laumars 11y ago
        > Most "desktops" today are in fact laptops, and the reason you want to have the CPU sit idle is battery time. That's a fair point. I'd argue that ZFS doesn't chuck that much extra CPU time that most people would notice. However if you're one of those guys that likes to tweak your power settings to the nth degree, then ZFS probably isn't best. But those kinds of people are more likely to run a tiling WM instead of a full fat compositing desktop environment; and such like. So they're definitely not your regular desktop users > If you're not pooling disks together but just have the built in disk, and usb hard disks that you conenct occasionally you don't get much if anything from ZFS, but you get the penalties it incurs... You seem to have missed quite a number of useful ZFS features off your list. Ignoring deduplication (as that is extremely memory and CPU hungry), ZFS has snapshotting, checksumming, superblock recovery (even if your journal gets trashed, you can recover it), online data integrity checks (no annoying wait screens for fsck to run through), better compression algorithms than is supported in most other file systems... I could go on, but suffice to say that ZFS is an improvement over many other file systems in other areas than just software RAIDing. That all said, in an ideal world desktops would be running HAMMER (DragonflyBSD's file system) over ZFS. I do love ZFS, but HAMMER seems a slightly better fit for desktops / laptops as it has the aforementioned features but without many of the data centre stuff that home users wouldn't need. A better compromise. However I don't see much interest for HAMMER outside of Dragonfly. > And it's not a substitute for backing up your disks either... Nobody suggested it was. However ZFS does also have backup features built into it too.
- santaclaus 12y ago
  Yea but there is a continuum of reliability -- ZFS is way better than HFS+ in this regard. Also, Btrfs.
  - 286c8cb04bda 12y ago
    I've used btrfs (at several-petabytes-of-data scale) and wouldn't trust it any more than HFS+. Use ZFS if you like, or XFS, or even ext4 instead.
    - yungchin 12y ago
      Would love to hear more about your findings with btrfs.
      - 286c8cb04bda 11y ago
        The biggest recurring issue was deadlocks in the garbage collector. It would start cleanup in a subvolume and trip all over itself. After that, any I/O to that specific directory would never return. The only solutions was to reboot the server and fsck for a few hours. Second frequent problem: Hitting 90% capacity in a filesystem has a non-trivial chance to ruin it forever. Hit the wrong code path, and, even if you immediately delete a bunch of things, I/O to that filesystem would be forever 3000% slower.
    - diroussel 11y ago
      I would also like to learn more about your view of btrfs. Is it a fundamental problem? Or an implementation, or community issue you have?
- nodata 12y ago
  What are you talking about? HFS is creaky and ancient and doesn't have checksums, ZFS and btrfs do.
  - Someone 12y ago
    The way I see it: HFS+ is creaky, ancient, but battle-hardened. ZFS is solid and battle-hardened, but not that suitable for many of Apple's customers, who want the ability to unmount disks they added, and who will not understand that they need huge amounts of RAM (they their Macoboks don't have) to keep their disks speedy, btrfs is promising but not battle-hardened yet, and it is very unlikely Apple will use it as long as it has a GPL license (I think it is more likely that Apple will add checksums to HFS+ in some somewhat clumsy, but working, way than that it starts supporting btrfs)
    - rodgerd 12y ago
      > HFS+ is creaky, ancient, but battle-hardened. If you think HFS and HFS+ are battle-hardened I don't know what to say to you. "Shell-shocked" would be a better description.
      - threeseed 12y ago
        Tens of millions of users have been using HFS for decades mostly with zero issues. I personally have used a Mac since the Mac+ and not one had corruption due to the filesystem. I would describe it as battle hardened for sure.
        rodgerd 12y ago
        > I personally have used a Mac since the Mac+ and not one had corruption due to the filesystem. You're a very lucky man, then. A large part of my time, at one point in my life, was spent fixing up corrupted HFS filesystems.
        acdha 12y ago
        “mostly with zero issues” is the same as saying “with issues”. I would generally agree that it's rare for e.g. HFS to corrupt its own metadata due to bugs but bit-rot happens. If you think you haven't encountered it, you probably haven't actually checked your files closely enough – this is particularly easy to miss with things like video or image formats which were designed to tolerate a few bit-flips. The reason why so many Mac users were interested in ZFS was that a modern filesystem incorporates strong integrity checks so you are guaranteed to either read back the bytes you originally wrote or get an error. That's a much better way to work than requiring you to use a tool which does strong hashes periodically to see if anything has broken.
- ChrisLTD 12y ago
  A backup doesn't help you if the photo is corrupted, and then that corrupted photo is propagated to your backups. You need a way to catch file corruption, like what ZFS and Btrfs have.
  - vacri 12y ago
    btrfs still isn't 'production ready', unfortunately.
    - samf 12y ago
      ZFS is, and Apple almost had it. Damn.
      - coldtea 12y ago
        ZFS is totally broken for regular desktop use. It's use case is a different thing altogether. Do people just read the "pros" and not the "cons"?
        nroach 12y ago
        Have you tried https://openzfsonosx.org https://openzfsonosx.org ? I use it daily on my primary desktop for my home directory and with an external time machine backup drive and have had zero problems.
        laumars 11y ago
        ZFS doesn't have a specific "use case". In fact a large amount of effort went into ensuring that ZFS was tolerant of consumer hardware, so that makes it as sensible option for desktop use as ext4 or HFS+ (when looking strictly at "use cases"). You've also failed to quantify why ZFS is "totally broken for regular desktop use" (likely because you've never actually used ZFS so can only blow smoke about it's issues). People love to over exaggerate the RAM requirements for ZFS because they see build logs of enterprise-grade storage servers. But you wouldn't expect that kind of throughput on your desktop systems even with lighter file systems. And most desktops have their 64bit CPU sat idle for most of the time, so there's not even an argument for the additional instruction overhead. Obviously the real crux of argument is "what do you primarily use your desktop for"? If it's gaming, then there's little point running ZFS since you're going to be disinterested in the benefits of ZFS (plus likely running Windows anyway). However if your desktop is a development machine, used for multimedia authoring or even just an internet terminal (like most PCs are these days), then ZFS is a viable option.
- galago 12y ago
  I've head John Siracusa discuss this, but I can't find a good link at the moment. Yes, with cloud storage systems, there is a chance that with HFS+ you're just backing up garbage and possibly won't notice until you use the file. HFS+ is not really equivalent to its peers.
  - 72deluxe 12y ago
    Here you are, it's from the Lion review I think. Pretty scary reading, and particularly since we've had numerous releases since then with not many changes: http://arstechnica.com/apple/2011/07/mac-os-x-10-7/12/ http://arstechnica.com/apple/2011/07/mac-os-x-10-7/12/
abalone 12y ago
Literally the first feature listed is iCloud Photo Library, a solution to critical photo data storage. It backs it up offsite which is better protection than even the best filesystem software.
- binarycrusader 12y ago
  If I'm offline for some period of time and upload photos to my laptop, an unreliable filesystem has every chance to corrupt my data before it gets uploaded to the cloud when I eventually reconnect. The "cloud" option itself simplifies the backup process and could potentially reduce the window of a chance for corruption, but it does not eliminate the need for a reliable, local filesystem.
  - uptown 12y ago
    Wouldn't you still have the copy on the device or memory card you took the photo from as an alternate source to restore from? Just avoid wiping that until you have two other copies.
    - cbhl 12y ago
      If the device is an iPhone[0], then there is only one copy on one filesystem on one memory card. Granted, if your iPhone is offline for extended periods of time, you're also doing it wrong. But maybe you have an iPod Touch instead. [0] For the last few years, the newest iPhone is the #1 most common camera model found in EXIF data on Flickr, http://petapixel.com/2015/01/09/popular-cameras-flickr-2014/ http://petapixel.com/2015/01/09/popular-cameras-flickr-2014/. IIRC, Facebook saw similar results for photo uploads.
    - vacri 12y ago
      This seems terrifying if the filesystem is that bad that you have to do this to guard against corruption. And isn't half the point of offloading images from a card to clear up the card for reuse?
      - 12y ago
bitL 12y ago
Get a NAS...
- 72deluxe 12y ago
  I've got a NAS but these things come with gigabit Ethernet and Marvel processors that will no way be able to cope with gigabit speeds! Doing a Time Machine backup to NAS isn't fun, I can tell you. Particularly if you use VMs (which you can exclude, but why would you when it's an entire machine image eh?)
  - mrsteveman1 12y ago
    HP's Microservers can cope with gigabit speeds and the AMD models are dirt cheap, both compared to poorly engineered "NAS" devices, and especially for what they can provide in terms of server level features. I've had the older N36L model, an AMD Athlon NEO II processor with 8GB ECC RAM running for years with ZFS handling a few big spinning drives for storage, it's been quite nice and trouble free.
    - 72deluxe 11y ago
      That beats my piece-of-junk NAS then! Thanks Freecom! USB3 and Gigabit Ethernet (and an always open SSH port 22 yet no responsive server on that address...)
- furyg3 11y ago
  This is my solution, but honestly there's a lot to be desired, especially with regard to photos. Storing Aperture, iPhoto, or Lightroom databases on the network is not ideal. Wifi, even 802.11ac, is barely usable. Gigabit means you're wired and also has issues. Apple frequently has implementation bugs on both AFP and SMB which slow things down way below gigabit speeds, and your NAS's processor also becomes a factor. Forget about using this setup if you're not actually on the local network (i.e. VPN). The result is having to manually manage your photo databases (which one's are on my laptop? which ones are on my NAS?) which works ok in Lightroom but Aperture and iPhoto are not very good at. This also creates backup stress (um, can I delete this?) and means I have to decide in advance if I want to look at those photos from two weeks ago before I leave the house. If so, grab a cup of coffee while they copy. Ideally you would have a system where a) your photos are automatically stored in the cloud b) you can select which albums are stored locally and c) there is some sort of cloud backup option that you control (computer with more storage, NAS, or other cloud provider that archives your cloud data).
RexRollman 12y ago
"fundamentally unreliable" I don't know if I buy that. But I do agree with you that Apple needs to improve its file system offerings.
- feld 11y ago
  HFS is a terrible fs design. HFS+ marginally improves it. Consider that in HFS (and early HFS+) had a fun limitation -- you could only ever create 65535 files. Didn't matter if you deleted; the Catalog (CNID) incremented with each new file created until it ran out. Also HFS+ only has 1 second resolution for timestamps and can't do timestamps later than February 6, 2040. Other dreadful details here: http://blog.barthe.ph/2014/06/10/hfs-plus-bit-rot/ http://blog.barthe.ph/2014/06/10/hfs-plus-bit-rot/
Xixi 12y ago
I've lost data (well, they were backed up, so not really lost...) to playing with beta filesystems (reiserfs 4, if memory serves well), never to HFS+. I'm pretty sure somewhere in Cupertino a team is working on a new file-system, but I seriously hope they don't ship it before it's ready. I wouldn't mind playing with it in beta though...
- njloof 11y ago
  Yeah, I'd heard that with reiserfs your data could just disappear one day without a trace. Once I got the developer to admit fault he was able to show me where he'd hidden the data. It was badly corrupted by then but still identifiable.
zw 12y ago
For something that foundational, you have to move as one giant, lumbering beast. Recall that Apple had at least one (that we know of) false start. A fair bit of the OS (CoreServices; Carbon, and the apps that still rely on it) still relies on HFS+-specific data structures and features. Core Storage is already moving forward as an effort to dissociate the bytes in logical storage from the bits on physical media. In the short term, we might be able to sandbox the evils of HFS+ by having it just be a software abstraction on top of a much more modern file-system-like-thing. Yosemite already moves any machine it upgrades to Core Storage, even if you don't use FileVault.
- MCRed 12y ago
  Plus, since the day they added journaling to HFS+, I haven't lost a single byte of data due to corruption of any kind. I've had drives failed, where I lost a day's data (Was traveling and away from my time machine drive) and that sucked. But near as I can tell, beyond losing short periods of work due to catastrophic failures of drives (early SSDs were problematic!) I haven't lost any data since Journalling was added (of course, Time Machine has saved my bacon from drive failures.) I think Apple has done a really great job in this regard. Yes, HFS+ is based on filesystem work going back to the original Macintosh, but in use it's working fine.
  - diroussel 11y ago
    I've had HFS+ corrrupt my TimeMachine partition, and Disk Utility was unable to repair it. DiskWarroir was no use as the volume was too big (the new 64-bit version addresses this). HFS+ is still unreliable. The journalling is only for file system meta data. The data in the files is not journaled or checksumed, and can be corrupted without detection.
  - jahewson 11y ago
    I experience HFS+ corruption frequently on my TimeMachine disk. There's nothing wrong with with the disk, it's the file system causing it. Disk Utiliy is usually able to repair it, but I did loose some backups once. How are you verifying that you've never lost a byte of data? What makes you so sure?
zw 12y ago
For something that foundational, you have to move as one giant, lumbering beast. Recall that Apple had at least one (that we know of) false start. A fair bit of the OS (CoreServices; Carbon, and the apps that still rely on it) still relies on HFS+-specific data structures and features. Core Storage is already moving forward as an effort to dissociate the bytes in logical storage from the bits on physical media. In the short term, we might be able to sandbox the evils of HFS+ by having it just be a software abstraction on top of a much more modern file-system-like-thing. Yosemite already moves any machine it upgrades to Core Storage, even if you don't use FileVault.
BinaryIdiot 12y ago
As a former Apple Technical Support Rep of about 5 years ago iPhoto has always had a reliability issue. Back then once you approached 2GB it would begin going slow and corruption had a higher chance. It's slowly gotten better over the years but reading all of the internal support articles at Apple it was alarming just how many issues it had at the time. Perhaps it's far better today, I'm not really sure but it completely turned me off to using it at all.
- eridius 12y ago
  You realize you're commenting on a post about Apple's brand new app Photos, which is not the same thing as iPhoto, right? It's a replacement, sure, but it's a new codebase rather than an update to iPhoto.
  - BinaryIdiot 12y ago
    > You realize you're commenting on a post about Apple's brand new app Photos, which is not the same thing as iPhoto, right? I was commenting on a comment which talked about Apple's file system in a critical manner discussing how it can become corrupted. I shared a story regarding how iPhoto had many issues like this years ago which makes me wary of using any Apple photo application. Seemed on topic with the thread to me. > it's a new codebase rather than an update to iPhoto. Source? I had not heard this. I heard multiple things mostly which pointed to basing the new application off of Aperture's code base.
    - eridius 11y ago
      The point of your comment seems to be "iPhoto is untrustworthy, I don't think I'm willing to give it another shot". Photos.app isn't iPhoto. I think I was perfectly warranted in pointing that out. > I heard multiple things mostly which pointed to basing the new application off of Aperture's code base. I actually hadn't heard that, but I guess what I meant was Photos.app isn't based on the iPhoto codebase. Whether it's 100% fresh code, or whether it's based on the Aperture codebase, I don't know.
wrcwill 12y ago
I don't know much about filesystems. How can a filesystem have higher chance of corrupting data? And what is different about HFS that makes it likely to corrupt data?
- MCRed 12y ago
  It's got Journalling, and since that point I've seen no evidence of it corrupting any data, despite several catastrophic drive failures (early SSDs) and the need to restore from Time Machine.
jsz0 12y ago
HFS+ is not fundamentally unreliable. The potential exists for data corruption to occur in the amount of time required for the kernel to panic and halt. Even when that happens it still requires the stars to align and catch the HFS+ process writing data to the wrong place with exactly the right bits of memory being flipped. Even when that happens the chances or those bits landing on non-free user data with no file format supplied error correction further reduces the chances of data corruption. Even when that happens many file formats suffering from data corruption can be salvaged. Of course if you have a backup of any kind the odds of the same exact file / bits on both drives being corrupted causing permanent data loss are astronomical. Add a third offsite / cloud backup and it'd be way more worthwhile to worry about being eaten by sharks.
- johnm1019 12y ago
  As the great teacher of my Monte Carlo course taught us, "If something can happen, it will."
- diroussel 11y ago
  It's not fundamentally reliable either. From: http://en.wikipedia.org/wiki/Hierarchical_File_System http://en.wikipedia.org/wiki/Hierarchical_File_System "The Catalog File, which stores all the file and directory records in a single data structure, results in performance problems when the system allows multitasking, as only one program can write to this structure at a time, meaning that many programs may be waiting in queue due to one program "hogging" the system.[2] It is also a serious reliability concern, as damage to this file can destroy the entire file system." See also: https://news.ycombinator.com/item?id=7876217 https://news.ycombinator.com/item?id=7876217
coldtea 12y ago
>that OS X has a fundamentally unreliable filesystem Let's put it this way: OS X's filesystem is in no way "fundamentally unreliable".

the_mitsuhiko 12y ago

> OS X has a fundamentally unreliable filesystem Unfortunately so is every other filesystem. Which is also why Apple heavily emphasizes making backups.

dman 12y ago
Have you looked at ZFS?
- coldtea 12y ago
  Have YOU looked at ZFS, and how it's unfit for desktop use?
  - laumars 11y ago
    That's unsubstantiated FUD. You only have to glance through this thread to see that there are people (myself included) who run ZFS on their desktop machine (laptop in my case). People love to over exaggerate the RAM requirements for ZFS because they see build logs of enterprise-grade storage servers. But you wouldn't expect that kind of throughput on your desktop systems even with lighter file systems. And most desktops have their 64bit CPU sat idle for most of the time, so there's not even an argument for the additional instruction overhead. Obviously the real crux of argument is "what do you primarily use your desktop for"? If it's gaming, then there's little point running ZFS since you're going to be disinterested in the benefits of ZFS anyway, so it's not worth the marginal overhead. However if your desktop is a development machine, used for multimedia authoring or even just an internet terminal (like most PCs are these days), then ZFS is a viable option.
    - coldtea 11y ago
      >And most desktops have their 64bit CPU sat idle for most of the time, so there's not even an argument for the additional instruction overhead. Most "desktops" today are in fact laptops, and the reason you want to have the CPU sit idle is battery time. >However if your desktop is a development machine, used for multimedia authoring or even just an internet terminal (like most PCs are these days), then ZFS is a viable option. If you're not pooling disks together but just have the built in disk, and usb hard disks that you conenct occasionally you don't get much if anything from ZFS, but you get the penalties it incurs... And it's not a substitute for backing up your disks either...
      - laumars 11y ago
        > Most "desktops" today are in fact laptops, and the reason you want to have the CPU sit idle is battery time. That's a fair point. I'd argue that ZFS doesn't chuck that much extra CPU time that most people would notice. However if you're one of those guys that likes to tweak your power settings to the nth degree, then ZFS probably isn't best. But those kinds of people are more likely to run a tiling WM instead of a full fat compositing desktop environment; and such like. So they're definitely not your regular desktop users > If you're not pooling disks together but just have the built in disk, and usb hard disks that you conenct occasionally you don't get much if anything from ZFS, but you get the penalties it incurs... You seem to have missed quite a number of useful ZFS features off your list. Ignoring deduplication (as that is extremely memory and CPU hungry), ZFS has snapshotting, checksumming, superblock recovery (even if your journal gets trashed, you can recover it), online data integrity checks (no annoying wait screens for fsck to run through), better compression algorithms than is supported in most other file systems... I could go on, but suffice to say that ZFS is an improvement over many other file systems in other areas than just software RAIDing. That all said, in an ideal world desktops would be running HAMMER (DragonflyBSD's file system) over ZFS. I do love ZFS, but HAMMER seems a slightly better fit for desktops / laptops as it has the aforementioned features but without many of the data centre stuff that home users wouldn't need. A better compromise. However I don't see much interest for HAMMER outside of Dragonfly. > And it's not a substitute for backing up your disks either... Nobody suggested it was. However ZFS does also have backup features built into it too.
santaclaus 12y ago
Yea but there is a continuum of reliability -- ZFS is way better than HFS+ in this regard. Also, Btrfs.
- 286c8cb04bda 12y ago
  I've used btrfs (at several-petabytes-of-data scale) and wouldn't trust it any more than HFS+. Use ZFS if you like, or XFS, or even ext4 instead.
  - yungchin 12y ago
    Would love to hear more about your findings with btrfs.
    - 286c8cb04bda 11y ago
      The biggest recurring issue was deadlocks in the garbage collector. It would start cleanup in a subvolume and trip all over itself. After that, any I/O to that specific directory would never return. The only solutions was to reboot the server and fsck for a few hours. Second frequent problem: Hitting 90% capacity in a filesystem has a non-trivial chance to ruin it forever. Hit the wrong code path, and, even if you immediately delete a bunch of things, I/O to that filesystem would be forever 3000% slower.
  - diroussel 11y ago
    I would also like to learn more about your view of btrfs. Is it a fundamental problem? Or an implementation, or community issue you have?
nodata 12y ago
What are you talking about? HFS is creaky and ancient and doesn't have checksums, ZFS and btrfs do.
- Someone 12y ago
  The way I see it: HFS+ is creaky, ancient, but battle-hardened. ZFS is solid and battle-hardened, but not that suitable for many of Apple's customers, who want the ability to unmount disks they added, and who will not understand that they need huge amounts of RAM (they their Macoboks don't have) to keep their disks speedy, btrfs is promising but not battle-hardened yet, and it is very unlikely Apple will use it as long as it has a GPL license (I think it is more likely that Apple will add checksums to HFS+ in some somewhat clumsy, but working, way than that it starts supporting btrfs)
  - rodgerd 12y ago
    > HFS+ is creaky, ancient, but battle-hardened. If you think HFS and HFS+ are battle-hardened I don't know what to say to you. "Shell-shocked" would be a better description.
    - threeseed 12y ago
      Tens of millions of users have been using HFS for decades mostly with zero issues. I personally have used a Mac since the Mac+ and not one had corruption due to the filesystem. I would describe it as battle hardened for sure.
      - rodgerd 12y ago
        > I personally have used a Mac since the Mac+ and not one had corruption due to the filesystem. You're a very lucky man, then. A large part of my time, at one point in my life, was spent fixing up corrupted HFS filesystems.
      - acdha 12y ago
        “mostly with zero issues” is the same as saying “with issues”. I would generally agree that it's rare for e.g. HFS to corrupt its own metadata due to bugs but bit-rot happens. If you think you haven't encountered it, you probably haven't actually checked your files closely enough – this is particularly easy to miss with things like video or image formats which were designed to tolerate a few bit-flips. The reason why so many Mac users were interested in ZFS was that a modern filesystem incorporates strong integrity checks so you are guaranteed to either read back the bytes you originally wrote or get an error. That's a much better way to work than requiring you to use a tool which does strong hashes periodically to see if anything has broken.
ChrisLTD 12y ago
A backup doesn't help you if the photo is corrupted, and then that corrupted photo is propagated to your backups. You need a way to catch file corruption, like what ZFS and Btrfs have.
- vacri 12y ago
  btrfs still isn't 'production ready', unfortunately.
  - samf 12y ago
    ZFS is, and Apple almost had it. Damn.
    - coldtea 12y ago
      ZFS is totally broken for regular desktop use. It's use case is a different thing altogether. Do people just read the "pros" and not the "cons"?
      - nroach 12y ago
        Have you tried https://openzfsonosx.org https://openzfsonosx.org ? I use it daily on my primary desktop for my home directory and with an external time machine backup drive and have had zero problems.
      - laumars 11y ago
        ZFS doesn't have a specific "use case". In fact a large amount of effort went into ensuring that ZFS was tolerant of consumer hardware, so that makes it as sensible option for desktop use as ext4 or HFS+ (when looking strictly at "use cases"). You've also failed to quantify why ZFS is "totally broken for regular desktop use" (likely because you've never actually used ZFS so can only blow smoke about it's issues). People love to over exaggerate the RAM requirements for ZFS because they see build logs of enterprise-grade storage servers. But you wouldn't expect that kind of throughput on your desktop systems even with lighter file systems. And most desktops have their 64bit CPU sat idle for most of the time, so there's not even an argument for the additional instruction overhead. Obviously the real crux of argument is "what do you primarily use your desktop for"? If it's gaming, then there's little point running ZFS since you're going to be disinterested in the benefits of ZFS (plus likely running Windows anyway). However if your desktop is a development machine, used for multimedia authoring or even just an internet terminal (like most PCs are these days), then ZFS is a viable option.
galago 12y ago
I've head John Siracusa discuss this, but I can't find a good link at the moment. Yes, with cloud storage systems, there is a chance that with HFS+ you're just backing up garbage and possibly won't notice until you use the file. HFS+ is not really equivalent to its peers.
- 72deluxe 12y ago
  Here you are, it's from the Lion review I think. Pretty scary reading, and particularly since we've had numerous releases since then with not many changes: http://arstechnica.com/apple/2011/07/mac-os-x-10-7/12/ http://arstechnica.com/apple/2011/07/mac-os-x-10-7/12/

abalone 12y ago

Literally the first feature listed is iCloud Photo Library, a solution to critical photo data storage. It backs it up offsite which is better protection than even the best filesystem software.

binarycrusader 12y ago
If I'm offline for some period of time and upload photos to my laptop, an unreliable filesystem has every chance to corrupt my data before it gets uploaded to the cloud when I eventually reconnect. The "cloud" option itself simplifies the backup process and could potentially reduce the window of a chance for corruption, but it does not eliminate the need for a reliable, local filesystem.
- uptown 12y ago
  Wouldn't you still have the copy on the device or memory card you took the photo from as an alternate source to restore from? Just avoid wiping that until you have two other copies.
  - cbhl 12y ago
    If the device is an iPhone[0], then there is only one copy on one filesystem on one memory card. Granted, if your iPhone is offline for extended periods of time, you're also doing it wrong. But maybe you have an iPod Touch instead. [0] For the last few years, the newest iPhone is the #1 most common camera model found in EXIF data on Flickr, http://petapixel.com/2015/01/09/popular-cameras-flickr-2014/ http://petapixel.com/2015/01/09/popular-cameras-flickr-2014/. IIRC, Facebook saw similar results for photo uploads.
  - vacri 12y ago
    This seems terrifying if the filesystem is that bad that you have to do this to guard against corruption. And isn't half the point of offloading images from a card to clear up the card for reuse?
    - 12y ago

bitL 12y ago

Get a NAS...

72deluxe 12y ago
I've got a NAS but these things come with gigabit Ethernet and Marvel processors that will no way be able to cope with gigabit speeds! Doing a Time Machine backup to NAS isn't fun, I can tell you. Particularly if you use VMs (which you can exclude, but why would you when it's an entire machine image eh?)
- mrsteveman1 12y ago
  HP's Microservers can cope with gigabit speeds and the AMD models are dirt cheap, both compared to poorly engineered "NAS" devices, and especially for what they can provide in terms of server level features. I've had the older N36L model, an AMD Athlon NEO II processor with 8GB ECC RAM running for years with ZFS handling a few big spinning drives for storage, it's been quite nice and trouble free.
  - 72deluxe 11y ago
    That beats my piece-of-junk NAS then! Thanks Freecom! USB3 and Gigabit Ethernet (and an always open SSH port 22 yet no responsive server on that address...)
furyg3 11y ago
This is my solution, but honestly there's a lot to be desired, especially with regard to photos. Storing Aperture, iPhoto, or Lightroom databases on the network is not ideal. Wifi, even 802.11ac, is barely usable. Gigabit means you're wired and also has issues. Apple frequently has implementation bugs on both AFP and SMB which slow things down way below gigabit speeds, and your NAS's processor also becomes a factor. Forget about using this setup if you're not actually on the local network (i.e. VPN). The result is having to manually manage your photo databases (which one's are on my laptop? which ones are on my NAS?) which works ok in Lightroom but Aperture and iPhoto are not very good at. This also creates backup stress (um, can I delete this?) and means I have to decide in advance if I want to look at those photos from two weeks ago before I leave the house. If so, grab a cup of coffee while they copy. Ideally you would have a system where a) your photos are automatically stored in the cloud b) you can select which albums are stored locally and c) there is some sort of cloud backup option that you control (computer with more storage, NAS, or other cloud provider that archives your cloud data).

RexRollman 12y ago

"fundamentally unreliable" I don't know if I buy that. But I do agree with you that Apple needs to improve its file system offerings.

feld 11y ago
HFS is a terrible fs design. HFS+ marginally improves it. Consider that in HFS (and early HFS+) had a fun limitation -- you could only ever create 65535 files. Didn't matter if you deleted; the Catalog (CNID) incremented with each new file created until it ran out. Also HFS+ only has 1 second resolution for timestamps and can't do timestamps later than February 6, 2040. Other dreadful details here: http://blog.barthe.ph/2014/06/10/hfs-plus-bit-rot/ http://blog.barthe.ph/2014/06/10/hfs-plus-bit-rot/

Xixi 12y ago

I've lost data (well, they were backed up, so not really lost...) to playing with beta filesystems (reiserfs 4, if memory serves well), never to HFS+. I'm pretty sure somewhere in Cupertino a team is working on a new file-system, but I seriously hope they don't ship it before it's ready. I wouldn't mind playing with it in beta though...

njloof 11y ago
Yeah, I'd heard that with reiserfs your data could just disappear one day without a trace. Once I got the developer to admit fault he was able to show me where he'd hidden the data. It was badly corrupted by then but still identifiable.

zw 12y ago

For something that foundational, you have to move as one giant, lumbering beast. Recall that Apple had at least one (that we know of) false start. A fair bit of the OS (CoreServices; Carbon, and the apps that still rely on it) still relies on HFS+-specific data structures and features. Core Storage is already moving forward as an effort to dissociate the bytes in logical storage from the bits on physical media. In the short term, we might be able to sandbox the evils of HFS+ by having it just be a software abstraction on top of a much more modern file-system-like-thing. Yosemite already moves any machine it upgrades to Core Storage, even if you don't use FileVault.

MCRed 12y ago
Plus, since the day they added journaling to HFS+, I haven't lost a single byte of data due to corruption of any kind. I've had drives failed, where I lost a day's data (Was traveling and away from my time machine drive) and that sucked. But near as I can tell, beyond losing short periods of work due to catastrophic failures of drives (early SSDs were problematic!) I haven't lost any data since Journalling was added (of course, Time Machine has saved my bacon from drive failures.) I think Apple has done a really great job in this regard. Yes, HFS+ is based on filesystem work going back to the original Macintosh, but in use it's working fine.
- diroussel 11y ago
  I've had HFS+ corrrupt my TimeMachine partition, and Disk Utility was unable to repair it. DiskWarroir was no use as the volume was too big (the new 64-bit version addresses this). HFS+ is still unreliable. The journalling is only for file system meta data. The data in the files is not journaled or checksumed, and can be corrupted without detection.
- jahewson 11y ago
  I experience HFS+ corruption frequently on my TimeMachine disk. There's nothing wrong with with the disk, it's the file system causing it. Disk Utiliy is usually able to repair it, but I did loose some backups once. How are you verifying that you've never lost a byte of data? What makes you so sure?

zw 12y ago

BinaryIdiot 12y ago

As a former Apple Technical Support Rep of about 5 years ago iPhoto has always had a reliability issue. Back then once you approached 2GB it would begin going slow and corruption had a higher chance. It's slowly gotten better over the years but reading all of the internal support articles at Apple it was alarming just how many issues it had at the time. Perhaps it's far better today, I'm not really sure but it completely turned me off to using it at all.

eridius 12y ago
You realize you're commenting on a post about Apple's brand new app Photos, which is not the same thing as iPhoto, right? It's a replacement, sure, but it's a new codebase rather than an update to iPhoto.
- BinaryIdiot 12y ago
  > You realize you're commenting on a post about Apple's brand new app Photos, which is not the same thing as iPhoto, right? I was commenting on a comment which talked about Apple's file system in a critical manner discussing how it can become corrupted. I shared a story regarding how iPhoto had many issues like this years ago which makes me wary of using any Apple photo application. Seemed on topic with the thread to me. > it's a new codebase rather than an update to iPhoto. Source? I had not heard this. I heard multiple things mostly which pointed to basing the new application off of Aperture's code base.
  - eridius 11y ago
    The point of your comment seems to be "iPhoto is untrustworthy, I don't think I'm willing to give it another shot". Photos.app isn't iPhoto. I think I was perfectly warranted in pointing that out. > I heard multiple things mostly which pointed to basing the new application off of Aperture's code base. I actually hadn't heard that, but I guess what I meant was Photos.app isn't based on the iPhoto codebase. Whether it's 100% fresh code, or whether it's based on the Aperture codebase, I don't know.

wrcwill 12y ago

I don't know much about filesystems. How can a filesystem have higher chance of corrupting data? And what is different about HFS that makes it likely to corrupt data?

MCRed 12y ago
It's got Journalling, and since that point I've seen no evidence of it corrupting any data, despite several catastrophic drive failures (early SSDs) and the need to restore from Time Machine.

jsz0 12y ago

HFS+ is not fundamentally unreliable. The potential exists for data corruption to occur in the amount of time required for the kernel to panic and halt. Even when that happens it still requires the stars to align and catch the HFS+ process writing data to the wrong place with exactly the right bits of memory being flipped. Even when that happens the chances or those bits landing on non-free user data with no file format supplied error correction further reduces the chances of data corruption. Even when that happens many file formats suffering from data corruption can be salvaged. Of course if you have a backup of any kind the odds of the same exact file / bits on both drives being corrupted causing permanent data loss are astronomical. Add a third offsite / cloud backup and it'd be way more worthwhile to worry about being eaten by sharks.

johnm1019 12y ago
As the great teacher of my Monte Carlo course taught us, "If something can happen, it will."
diroussel 11y ago
It's not fundamentally reliable either. From: http://en.wikipedia.org/wiki/Hierarchical_File_System http://en.wikipedia.org/wiki/Hierarchical_File_System "The Catalog File, which stores all the file and directory records in a single data structure, results in performance problems when the system allows multitasking, as only one program can write to this structure at a time, meaning that many programs may be waiting in queue due to one program "hogging" the system.[2] It is also a serious reliability concern, as damage to this file can destroy the entire file system." See also: https://news.ycombinator.com/item?id=7876217 https://news.ycombinator.com/item?id=7876217

coldtea 12y ago

>that OS X has a fundamentally unreliable filesystem Let's put it this way: OS X's filesystem is in no way "fundamentally unreliable".