6 ms·

Click-bait title again. “You can lose some of your file changes in case of hard-reboot” is more correct. It was a given truth for me all the time and I can to

Click-bait title again. “You can lose some of your file changes in case of hard-reboot” is more correct. It was a given truth for me all the time and I can tolerate some data losses if power was accidentally turned off for my desktop, or if OS panicked (it happens ~ once per year to me). If this is a price for a 1000x speed increase - I’m more than happy they have implemented it this way.

marcan_42 5y ago
You can lose some file changes even after asking the OS to make sure they don't get lost, the normal way. That's a problem. It means e.g. transactional databases (which cannot afford to lose data like that) have a huge performance hit on these machines, since they have to use F_FULLFSYNC. And since that "no really, save my data" feature is not the standard fsync(), it means any portable software compiled for Linux will be safe, but will be unsafe on macOS, by default. That is a significant gotcha. The question is why do other NVMe manufacturers not have such a performance penalty? 10x is fine; 1000x is not. This is something Apple should fix. It's a firmware problem.
- EugeneOZ 5y ago
  No, it’s not a problem, it is expected. If you are running a transactional database on your desktop - at least add a UPS to your system.
  - olliej 5y ago
    The whole point of a transactional database is that even in the case of a power loss you do not lose data. If you UPS blows up, and so you lose power, you should not lose data. The point here is that on the apple systems if you do the correct thing your performance drops to that of spinning disks.
    - EugeneOZ 5y ago
      Add a secondary UPS. What will be the next excuse? It's ridiculous to think that in case of power loss you expect 100% data integrity - it might happen in the middle of the command execution. If the system should be unkillable, it should have an unkillable power source in the first place.
      - marcan_42 5y ago
        The entire point of modern journaling filesystems and properly designed transactional databases is to ensure 100% data integrity in case of power loss, every time, no matter what. The thinking you have is from the 1990s. We can (and do) do better today. A properly designed transactional database will only ever "fail ahead". If power fails a transaction that was in the process of committing might commit without an ack, but will never return an ack and then be lost on the next startup. The ack means the data is safe, regardless of what happened afterwards.
        EugeneOZ 5y ago
        After reading this comment I’m not so sure about that: https://news.ycombinator.com/item?id=30371181 https://news.ycombinator.com/item?id=30371181
        marcan_42 5y ago
        That comment is about the semantics of OS APIs; filesystems are designed not to corrupt themselves in case of hard shutdown, and this is true as long as the underlying storage is well-behaved (e.g. honors flush requests). Databases on macOS already use F_FULLFSYNC (if they noticed this issue) to provide those guarantees. On Linux they just use fsync().
      - freemint 5y ago
        Since no Mac device has two power support adding a second UPS means chaining which will only increase the risk of something going wrong in the chain. Nobody expects 100% data integrity on power of. What is expected is that data that what was fsynced has 100% data integrity once that system call returns. This information is also used when moving files across the network, the file gets deleted on the sender when the receiver said fsync is completed. This means you could loose entire files of data when moving things over the network onto a Mac
Maxion 5y ago
I guess I am old but the assumption I live by is that if power is suddenly cut from a computer - no matter desktop or laptop - it can damage the FS and/or cause data loss. For any mission critical stuff, I have it behind a UPS.
- olliej 5y ago
  No, modern filesystems aren't expected to be corrupted by sudden power loss, and "put it behind a ups" assumes that it's impossible for a UPS to fail.
- marcan_42 5y ago
  At least your thinking is old. Modern filesystems and databases are designed to prevent data loss in that scenario. The last time I saw a modern filesystem eat itself on sudden power loss was when I was evaluating btrfs in a datacenter setting, and that absolutely told me it was not a reliable FS and we went with something else. I've never seen it happen with ext4 or XFS (configured properly) in over a decade, assuming the underlying storage is well-behaved. OTOH, I've seen cases of e.g. data in files being replaced by zeroes and applications crashing due to that (it's pretty common that zsh complains about .zsh_history being corrupted after a crash due to a trailing block of zeroes). This happens when filesystems are mounted with metadata journaling but no data journaling. If you use data journaling (or a filesystem designed to inherently avoid this, e.g. COW cases), that situation can't happen either. Most databases would be designed to gracefully handle this kind of situation without requiring systemwide data journaling though. That's a tradeoff that is available to the user depending on their specific use case and whether the applications are designed with that in mind or not.
  - freedomben 5y ago
    how long ago did you last test btrfs?
- sudosysgen 5y ago
  Modern filesystems are designed so that you can unplug the hard drive and it will not be in a corrupted state. UPSes can and do fail.
newaccount74 5y ago
The problem isn't that the default case is unsafe -- the problem is that the safe case is so extremely slow.

marcan_42 5y ago

You can lose some file changes even after asking the OS to make sure they don't get lost, the normal way. That's a problem. It means e.g. transactional databases (which cannot afford to lose data like that) have a huge performance hit on these machines, since they have to use F_FULLFSYNC. And since that "no really, save my data" feature is not the standard fsync(), it means any portable software compiled for Linux will be safe, but will be unsafe on macOS, by default. That is a significant gotcha. The question is why do other NVMe manufacturers not have such a performance penalty? 10x is fine; 1000x is not. This is something Apple should fix. It's a firmware problem.

EugeneOZ 5y ago
No, it’s not a problem, it is expected. If you are running a transactional database on your desktop - at least add a UPS to your system.
- olliej 5y ago
  The whole point of a transactional database is that even in the case of a power loss you do not lose data. If you UPS blows up, and so you lose power, you should not lose data. The point here is that on the apple systems if you do the correct thing your performance drops to that of spinning disks.
  - EugeneOZ 5y ago
    Add a secondary UPS. What will be the next excuse? It's ridiculous to think that in case of power loss you expect 100% data integrity - it might happen in the middle of the command execution. If the system should be unkillable, it should have an unkillable power source in the first place.
    - marcan_42 5y ago
      The entire point of modern journaling filesystems and properly designed transactional databases is to ensure 100% data integrity in case of power loss, every time, no matter what. The thinking you have is from the 1990s. We can (and do) do better today. A properly designed transactional database will only ever "fail ahead". If power fails a transaction that was in the process of committing might commit without an ack, but will never return an ack and then be lost on the next startup. The ack means the data is safe, regardless of what happened afterwards.
      - EugeneOZ 5y ago
        After reading this comment I’m not so sure about that: https://news.ycombinator.com/item?id=30371181 https://news.ycombinator.com/item?id=30371181
        marcan_42 5y ago
        That comment is about the semantics of OS APIs; filesystems are designed not to corrupt themselves in case of hard shutdown, and this is true as long as the underlying storage is well-behaved (e.g. honors flush requests). Databases on macOS already use F_FULLFSYNC (if they noticed this issue) to provide those guarantees. On Linux they just use fsync().
    - freemint 5y ago
      Since no Mac device has two power support adding a second UPS means chaining which will only increase the risk of something going wrong in the chain. Nobody expects 100% data integrity on power of. What is expected is that data that what was fsynced has 100% data integrity once that system call returns. This information is also used when moving files across the network, the file gets deleted on the sender when the receiver said fsync is completed. This means you could loose entire files of data when moving things over the network onto a Mac

Maxion 5y ago

I guess I am old but the assumption I live by is that if power is suddenly cut from a computer - no matter desktop or laptop - it can damage the FS and/or cause data loss. For any mission critical stuff, I have it behind a UPS.

olliej 5y ago
No, modern filesystems aren't expected to be corrupted by sudden power loss, and "put it behind a ups" assumes that it's impossible for a UPS to fail.
marcan_42 5y ago
At least your thinking is old. Modern filesystems and databases are designed to prevent data loss in that scenario. The last time I saw a modern filesystem eat itself on sudden power loss was when I was evaluating btrfs in a datacenter setting, and that absolutely told me it was not a reliable FS and we went with something else. I've never seen it happen with ext4 or XFS (configured properly) in over a decade, assuming the underlying storage is well-behaved. OTOH, I've seen cases of e.g. data in files being replaced by zeroes and applications crashing due to that (it's pretty common that zsh complains about .zsh_history being corrupted after a crash due to a trailing block of zeroes). This happens when filesystems are mounted with metadata journaling but no data journaling. If you use data journaling (or a filesystem designed to inherently avoid this, e.g. COW cases), that situation can't happen either. Most databases would be designed to gracefully handle this kind of situation without requiring systemwide data journaling though. That's a tradeoff that is available to the user depending on their specific use case and whether the applications are designed with that in mind or not.
- freedomben 5y ago
  how long ago did you last test btrfs?
sudosysgen 5y ago
Modern filesystems are designed so that you can unplug the hard drive and it will not be in a corrupted state. UPSes can and do fail.

newaccount74 5y ago

The problem isn't that the default case is unsafe -- the problem is that the safe case is so extremely slow.