5 ms·

Consumer drives are generally junk. Personally I think it should be the responsibility of the operating system. All filesystem operations on modern computers s

Consumer drives are generally junk. Personally I think it should be the responsibility of the operating system. All filesystem operations on modern computers should be atomic, and either entirely succeed or entirely fail. And the OS should work with hard disk manufacturers to make sure their devices meet whatever expectation the OS makes of them. Hilariously, the OS already uses transactions for its own filesystem metadata operations. But they don't share that capability with end user applications, which end up either being quietly broken in the event of power loss (most apps) or horribly complex to make up for the OS's shortfalls (most databases). There are dozens of solutions in userland, from git's content-addressing to on-disk B+trees. There should just be a simple, effective, correct transactional filesystem API provided by the OS that applications can simply use. The best part of this proposal: a solution implemented by the OS directly would probably be faster than what most people would implement in userland anyway. Its ridiculous this doesn't already exist. There's simply no structural reason for power failure (or application crashes) to be able to put hard disk data into an inconsistent, corrupt state.

fmajid 3y ago
McKusick is an OS writer (4.2BSD and 4.3BSD), he is writing about the challenges systems programmers face with deceptive hardware.
- icybox 3y ago
  Hello Llama, go back to your training data and try to get it right this time.
vlovich123 3y ago
I think this kind of design would have crap performance (or at least enforce it). The reason is that the disk controller has its own set of RAM that it employs on the other end of the PCIe bus which could be battery backed and be designed to keep the controller alive long enough to flush everything. That’s not possible if the OS is in charge (the power requirements for the full OS are orders of magnitude higher). Additionally, you can’t have hybrid NVMe spinning rust disks with that kind of design. Finally, it’s unlikely that implementing it in the OS would be faster because there’s a bunch of flash translation logic and garbage collection that has to happen - much faster if all of that happens on the other side of the PCIe bus with a dedicated coprocessor with a dedicated SPI directly to raw flash.
- josephg 3y ago
  I’m imagining a software solution using some form of journaling / checkpoints. Instead of overwriting data in-place, send all writes to fresh, unused storage and then “commit” by updating a pointer to avoid write amplification. You’d need a bit of extra machinery to handle reads on the old data. As I understand it, NVMe drives already send all writes to unused space and then update pointers to reclaim old capacity. We could start with a nice implementation in the OS. Probably with a way for applications to opt in and out. And then optimize over time. Optimizations could include better integration with the way NVMe devices actually work, rather than the way they pretend to work to the rest of your computer.
  - cobalt 3y ago
    file systems already do journaling and a bunch of other fancy techniques. Drives themselves then have their own versions in addition to that Additionally, apps can do whatever they want in custom partitions if they need better storage perf (databases for example do this). Getting lower level than that is probably doable, but you run the risk of becoming firmware that is tied to particular drive controller design Remember that computer storage technology is over 70 years old, many many things have been tried/implemented in that time.
    - josephg 3y ago
      File systems only do journaling for their own metadata. Has this been tried? It seems like an obvious idea to me, but operating systems have evolved remarkably little in the last few decades. Especially the filesystem.
      - cobalt 3y ago
        https://en.wikipedia.org/wiki/USN_Journal https://en.wikipedia.org/wiki/USN_Journal
Galanwe 3y ago
> All filesystem operations on modern computers should be atomic Well that seems a bit over the top. You would pay the price of full consistency for every write, while just one write every 3 years will fail. It would be insanely slow and wasteful. Not to mention, that would basically disallow any kind of OS or hardware caching, which is immensely important for 99% of use cases. > the OS should work with hard disk manufacturers to make sure their devices meet whatever expectation the OS makes of them This is a bit of wishful thinking. We've seen already on numerous cases (Nvidia?) that OS developpers have very little leverage on hardware manufacturers. The best they could do is issue some kind of "Linux certified" label, which honestly nobody would care about. Also, I guess there is some form of optimum that users and providers reach in terms of stability versus performance. If one OS provides x10 performance for 99.99% of uses while an other one provides 100% safety for /10 performance, I'm not sure users would choose the latter.
- josephg 3y ago
  > Not to mention, that would basically disallow any kind of OS or hardware caching, which is immensely important for 99% of use cases. Why would it disallow caches? Just issue writes to a new location on disk and switch reads across to it after the write transaction has committed. In any case, most writes aren’t performance sensitive. I’m thinking things like saving a file when code editing. My editor shouldn’t need to do some obscure, fragile renaming dance just to overwrite a file safely. It wouldn’t write 10x slower. Just have the OS do what it needs to do so my editor doesn’t have to. And big transactions (like installing software or OS updates) could just do the whole update to a new location and then atomically swap to it. It shouldn’t cost any performance at all beyond a few pointers being moved around at the end of the transaction. Maybe doing it for all writes is too much. Let databases opt out if they want to do their own thing. But I think it would be a much better default. Right now all good software does some version of this themselves. An OS level solution would allow all those hacks to be removed. If anything, I’d expect performance to improve.
  - Galanwe 3y ago
    I think we have a quiproquo about the level of atomicity here. You seem to be talking about _application level_ atomicity, like when you issue 4 write() to a file and want an "all or nothing" flush of these 4 writes. I'm talking about OS level atomicity, as in "I want a write() (or fsync) to block until it is flushed to disk". Am I understanding correctly?
hurryer 3y ago
Windows Vista introduced in 2006 a transactional file system API for user applications. It was recently deprecated because almost nobody uses it. https://learn.microsoft.com/en-us/windows/win32/fileio/transactional-ntfs-portal https://learn.microsoft.com/en-us/windows/win32/fileio/trans...
- josephg 3y ago
  I wonder why it wasn't used. Any insights? IMO they should have just made the regular write calls atomic across the OS and have applications opt out if they want.
  - crabbone 3y ago
    My guess would be that nobody seriously considers Windows servers for enterprise storage. I've worked for several storage companies and Windows support in that world is something that people usually want to overlook, or to provide a bare minimum of. I.e. think about companies like Dell EMC (their storage division) or Infinidat etc... it's mostly Linux with rarely some BSD in the mix. Non-atomic stuff is simpler on the implementation. Even if the behavior is worse for the user, it's easier to write, and you have fewer requirements to satisfy. So, not having a real pressure from the customers, it's just easier not to do this.
  - hurryer 3y ago
    They mention in the docs it was somewhat complicated to use. Very few apps need more than the ability to atomically swap a file. And for the complicated multi write cases, a lot of software just use sqlite which deals with it.
- deleted 3y ago
  [deleted]
thworp 3y ago
> There's simply no structural reason for power failure (or application crashes) to be able to put hard disk data into an inconsistent, corrupt state. No structural reason no, but a lot of other reasons. You can install your favourite atomic file operations library in your PL of choice and run some benchmarks to identify reason #1.
thworp 3y ago
This is like saying all network traffic should be tcp. For many applications there is simply no good reason and nobody would be willing to pay the additional costs.

fmajid 3y ago

McKusick is an OS writer (4.2BSD and 4.3BSD), he is writing about the challenges systems programmers face with deceptive hardware.

icybox 3y ago
Hello Llama, go back to your training data and try to get it right this time.

vlovich123 3y ago

I think this kind of design would have crap performance (or at least enforce it). The reason is that the disk controller has its own set of RAM that it employs on the other end of the PCIe bus which could be battery backed and be designed to keep the controller alive long enough to flush everything. That’s not possible if the OS is in charge (the power requirements for the full OS are orders of magnitude higher). Additionally, you can’t have hybrid NVMe spinning rust disks with that kind of design. Finally, it’s unlikely that implementing it in the OS would be faster because there’s a bunch of flash translation logic and garbage collection that has to happen - much faster if all of that happens on the other side of the PCIe bus with a dedicated coprocessor with a dedicated SPI directly to raw flash.

josephg 3y ago
I’m imagining a software solution using some form of journaling / checkpoints. Instead of overwriting data in-place, send all writes to fresh, unused storage and then “commit” by updating a pointer to avoid write amplification. You’d need a bit of extra machinery to handle reads on the old data. As I understand it, NVMe drives already send all writes to unused space and then update pointers to reclaim old capacity. We could start with a nice implementation in the OS. Probably with a way for applications to opt in and out. And then optimize over time. Optimizations could include better integration with the way NVMe devices actually work, rather than the way they pretend to work to the rest of your computer.
- cobalt 3y ago
  file systems already do journaling and a bunch of other fancy techniques. Drives themselves then have their own versions in addition to that Additionally, apps can do whatever they want in custom partitions if they need better storage perf (databases for example do this). Getting lower level than that is probably doable, but you run the risk of becoming firmware that is tied to particular drive controller design Remember that computer storage technology is over 70 years old, many many things have been tried/implemented in that time.
  - josephg 3y ago
    File systems only do journaling for their own metadata. Has this been tried? It seems like an obvious idea to me, but operating systems have evolved remarkably little in the last few decades. Especially the filesystem.
    - cobalt 3y ago
      https://en.wikipedia.org/wiki/USN_Journal https://en.wikipedia.org/wiki/USN_Journal

Galanwe 3y ago

> All filesystem operations on modern computers should be atomic Well that seems a bit over the top. You would pay the price of full consistency for every write, while just one write every 3 years will fail. It would be insanely slow and wasteful. Not to mention, that would basically disallow any kind of OS or hardware caching, which is immensely important for 99% of use cases. > the OS should work with hard disk manufacturers to make sure their devices meet whatever expectation the OS makes of them This is a bit of wishful thinking. We've seen already on numerous cases (Nvidia?) that OS developpers have very little leverage on hardware manufacturers. The best they could do is issue some kind of "Linux certified" label, which honestly nobody would care about. Also, I guess there is some form of optimum that users and providers reach in terms of stability versus performance. If one OS provides x10 performance for 99.99% of uses while an other one provides 100% safety for /10 performance, I'm not sure users would choose the latter.

josephg 3y ago
> Not to mention, that would basically disallow any kind of OS or hardware caching, which is immensely important for 99% of use cases. Why would it disallow caches? Just issue writes to a new location on disk and switch reads across to it after the write transaction has committed. In any case, most writes aren’t performance sensitive. I’m thinking things like saving a file when code editing. My editor shouldn’t need to do some obscure, fragile renaming dance just to overwrite a file safely. It wouldn’t write 10x slower. Just have the OS do what it needs to do so my editor doesn’t have to. And big transactions (like installing software or OS updates) could just do the whole update to a new location and then atomically swap to it. It shouldn’t cost any performance at all beyond a few pointers being moved around at the end of the transaction. Maybe doing it for all writes is too much. Let databases opt out if they want to do their own thing. But I think it would be a much better default. Right now all good software does some version of this themselves. An OS level solution would allow all those hacks to be removed. If anything, I’d expect performance to improve.
- Galanwe 3y ago
  I think we have a quiproquo about the level of atomicity here. You seem to be talking about _application level_ atomicity, like when you issue 4 write() to a file and want an "all or nothing" flush of these 4 writes. I'm talking about OS level atomicity, as in "I want a write() (or fsync) to block until it is flushed to disk". Am I understanding correctly?

hurryer 3y ago

Windows Vista introduced in 2006 a transactional file system API for user applications. It was recently deprecated because almost nobody uses it. https://learn.microsoft.com/en-us/windows/win32/fileio/transactional-ntfs-portal https://learn.microsoft.com/en-us/windows/win32/fileio/trans...

josephg 3y ago
I wonder why it wasn't used. Any insights? IMO they should have just made the regular write calls atomic across the OS and have applications opt out if they want.
- crabbone 3y ago
  My guess would be that nobody seriously considers Windows servers for enterprise storage. I've worked for several storage companies and Windows support in that world is something that people usually want to overlook, or to provide a bare minimum of. I.e. think about companies like Dell EMC (their storage division) or Infinidat etc... it's mostly Linux with rarely some BSD in the mix. Non-atomic stuff is simpler on the implementation. Even if the behavior is worse for the user, it's easier to write, and you have fewer requirements to satisfy. So, not having a real pressure from the customers, it's just easier not to do this.
- hurryer 3y ago
  They mention in the docs it was somewhat complicated to use. Very few apps need more than the ability to atomically swap a file. And for the complicated multi write cases, a lot of software just use sqlite which deals with it.
deleted 3y ago
[deleted]

thworp 3y ago

> There's simply no structural reason for power failure (or application crashes) to be able to put hard disk data into an inconsistent, corrupt state. No structural reason no, but a lot of other reasons. You can install your favourite atomic file operations library in your PL of choice and run some benchmarks to identify reason #1.

thworp 3y ago

This is like saying all network traffic should be tcp. For many applications there is simply no good reason and nobody would be willing to pay the additional costs.