3 ms·

After decades of issues with the storage layer and even some of the most popular programs written by top notch developers having bugs due to the problematic nat

by eis 4y ago

After decades of issues with the storage layer and even some of the most popular programs written by top notch developers having bugs due to the problematic nature of the APIs and filesystems involved I wish a completely new storage API would emerge. Something that exposes an asynchronous (and synchronous build upon it) API with ACID semantics. Filesystems are nothing more than specialized databases but they don't expose the necessary interface to use them as such. We need an API that is dead simple and hard to misuse with clearly defined semantics and guarantees but lets seasoned developers still exploit the hardware to its fullest with additional work. Hope dies last I guess :)

GordonS 4y ago
Windows used to have an almost unknown "transactional file system API" that sounds similar to what you're asking for. I think it was recently deprecated, but I don't know why, or what the history of this API is. Might be interesting to read up on it!
- eis 4y ago
  You are probably thinking of WinFS https://en.wikipedia.org/wiki/WinFS https://en.wikipedia.org/wiki/WinFS > In 2013 Bill Gates cited WinFS as his greatest disappointment at Microsoft and that the idea of WinFS was ahead of its time, which will re-emerge
  - GordonS 4y ago
    Nope, I was thinking of transactional NTFS, which AFAIK is still supported, but deprecated. I'd totally forgotten about WinFS - I remember it being touted years back, and it sounded great... and then it got cut before the next version of Windows was released. I was pretty disappointed, as at the time it really sounded like the future of file systems. Maybe just ahead of it's time.
    - hyc_symas 4y ago
      Yes, deprecated. I thought it had already been dropped but apparently not yet. https://docs.microsoft.com/en-us/windows/win32/fileio/about-transactional-ntfs https://docs.microsoft.com/en-us/windows/win32/fileio/about-...
- ectopod 4y ago
  Transactional NTFS: https://docs.microsoft.com/en-us/windows/win32/fileio/about-transactional-ntfs https://docs.microsoft.com/en-us/windows/win32/fileio/about-...
masklinn 4y ago
> Filesystems are nothing more than specialized databases but they don't expose the necessary interface to use them as such. Most FS are not even transactional, so they can't expose the necessary interfaces (hell, most individual FS calls are not guaranteed to be transactional). I assume you could build an application-level transactional system on ZFS, but I don't know that it exposes any such APIs either.
- eis 4y ago
  My point is that we need a new storage interface that is transactional. It's extremely hard to build a sound interface on the existing ones. And it would not be built on the existing FS because those are already built for the broken interfaces.
  - masklinn 4y ago
    And my point is that you’re working from the wrong end: you can’t have a transactional storage interface when you don’t have a widely available and deployed transactional storage. It’s easy to build a non-transactional interface atop a transactional storage, but if you don’t have the latter the former makes no sense.
    - eis 4y ago
      When you say transactional storage do you mean hardware support for transactions? I don't think that would be required. As long as the storage hardware interface (e.g. NVME) guarantees certain low level primitives like atomic sector writes. I guess what you'd want from hardware is similar to what you get with RAM. Atomic reads and writes of certain size plus read and write barriers. And then ditch the posix FS APIs, replace with a transactional interface that probably uses io_uring under the hood and build an FS that handles it in a correct and performant way.
      - xyzzy_plugh 4y ago
        Right, you're pretty much on the money: you want transactional storage to build upon. If NVME or flash interfaces worked reliably as you suggest, then you could do this today by just treating them as memory, but right now there is an fsync (or equivalent) somewhere in the call stack. And even then, hardware complexities are wildly abstracted to the point where a lot of storage devices have no guarantees in the face of e.g. power loss. Older hardware actually excels here! Tape can be used very, very reliably because the interface is simple and obvious in comparison to flash-based storage.
ghoward 4y ago
I agree with you. My vision for the operating system of the future is to build a microkernel that exclusively uses a modified ZFS that has a transactional API, that can run Linux drivers in userspace with zero or few changes, and that uses a uniform event API like Windows' handles or Plan 9's file descriptors. But that modified ZFS is perhaps the most important part, and I would want to make it so `fsync()` on that platform behaves as people would want it to: transactional such that if it succeeds, the data was written, and if it doesn't, that data was not written. The ability to run Linux drivers is so there isn't a chicken-and-egg problem with drivers. (It would also be nice to implement the POSIX API to solve that part of the chicken-and-egg problem.) The uniform event API is because the current OS API's are difficult to work with. OS's are based around resources and events. The resource API's are pretty good, but the event API's (select(), poll(), epoll(), io_uring, kqueue, WaitForMultipleObjects(), etc.) are still artificially constrained.
- xorcist 4y ago
  That's one of those things that sounds good in theory, but in practice has more to do with hardware than software semantics. Any end-to-end guarantees pretty much requires control over hardware, which of course would be very interesting to a lot of people, but isn't really what mainstream operating system concern themselves with. Already the block device hides too much of hardware, and any VFS layer will be even worse. Just because a write is done doesn't mean it's actually on the spinning rust. And even if it is, it could be done in several ways, maybe it was relocated, or parity hasn't been computed, or it's in the queue. Any attempt of abstracting different types of storage will pretty much have to resort to the least bad common denominator here. I'm not saying it's a bad idea, just that it's not the same problem that operating systems try to solve. But in the post-Optane world maybe block devices are too low level anyway and we'll finally see higher level storage systems.
  - ghoward 4y ago
    You're right, for the most part. We have to have hardware that doesn't lie when the data made it to disk, whatever that disk is. And we also need hardware that can do atomic updates to single sectors. But once we have those things, and people are demanding them more and more, a copy-on-write filesystem should be perfect for emulating transactions, using that single sector atomic update to "commit" the transaction.
anamax 4y ago
ReiserFS was headed that way before Reiser went to prison for murdering his wife. Other filesystems may also be headed that way.
jandrewrogers 4y ago
There is an impedance mismatch between what is required out of filesystems in terms of backward compatibility and the features/capabilities that would be useful for applications like databases. If you discard a requirement to look like a standard Linux filesystem to arbitrary applications, writing a purpose-built filesystem for a specific application often requires less code than trying to make off-the-shelf filesystems consistently deliver desired behavior and performance. And even then it is often difficult to guarantee that the off-the-shelf filesystem will really do what you expect in all cases. Installing a custom filesystem isn't always a practical option operationally but it is sometimes done for applications like high-scale data infrastructure because of the headaches it eliminates. I've built a few systems designed to be deployed either way. I am not sure there is a "one size fits all" solution for this. Alternative filesystem implementations tend to make very application-specific and hardware-aware design choices.

by eis 4y ago

GordonS 4y ago

Windows used to have an almost unknown "transactional file system API" that sounds similar to what you're asking for. I think it was recently deprecated, but I don't know why, or what the history of this API is. Might be interesting to read up on it!

eis 4y ago
You are probably thinking of WinFS https://en.wikipedia.org/wiki/WinFS https://en.wikipedia.org/wiki/WinFS > In 2013 Bill Gates cited WinFS as his greatest disappointment at Microsoft and that the idea of WinFS was ahead of its time, which will re-emerge
- GordonS 4y ago
  Nope, I was thinking of transactional NTFS, which AFAIK is still supported, but deprecated. I'd totally forgotten about WinFS - I remember it being touted years back, and it sounded great... and then it got cut before the next version of Windows was released. I was pretty disappointed, as at the time it really sounded like the future of file systems. Maybe just ahead of it's time.
  - hyc_symas 4y ago
    Yes, deprecated. I thought it had already been dropped but apparently not yet. https://docs.microsoft.com/en-us/windows/win32/fileio/about-transactional-ntfs https://docs.microsoft.com/en-us/windows/win32/fileio/about-...
ectopod 4y ago
Transactional NTFS: https://docs.microsoft.com/en-us/windows/win32/fileio/about-transactional-ntfs https://docs.microsoft.com/en-us/windows/win32/fileio/about-...

masklinn 4y ago

> Filesystems are nothing more than specialized databases but they don't expose the necessary interface to use them as such. Most FS are not even transactional, so they can't expose the necessary interfaces (hell, most individual FS calls are not guaranteed to be transactional). I assume you could build an application-level transactional system on ZFS, but I don't know that it exposes any such APIs either.

eis 4y ago
My point is that we need a new storage interface that is transactional. It's extremely hard to build a sound interface on the existing ones. And it would not be built on the existing FS because those are already built for the broken interfaces.
- masklinn 4y ago
  And my point is that you’re working from the wrong end: you can’t have a transactional storage interface when you don’t have a widely available and deployed transactional storage. It’s easy to build a non-transactional interface atop a transactional storage, but if you don’t have the latter the former makes no sense.
  - eis 4y ago
    When you say transactional storage do you mean hardware support for transactions? I don't think that would be required. As long as the storage hardware interface (e.g. NVME) guarantees certain low level primitives like atomic sector writes. I guess what you'd want from hardware is similar to what you get with RAM. Atomic reads and writes of certain size plus read and write barriers. And then ditch the posix FS APIs, replace with a transactional interface that probably uses io_uring under the hood and build an FS that handles it in a correct and performant way.
    - xyzzy_plugh 4y ago
      Right, you're pretty much on the money: you want transactional storage to build upon. If NVME or flash interfaces worked reliably as you suggest, then you could do this today by just treating them as memory, but right now there is an fsync (or equivalent) somewhere in the call stack. And even then, hardware complexities are wildly abstracted to the point where a lot of storage devices have no guarantees in the face of e.g. power loss. Older hardware actually excels here! Tape can be used very, very reliably because the interface is simple and obvious in comparison to flash-based storage.

ghoward 4y ago

I agree with you. My vision for the operating system of the future is to build a microkernel that exclusively uses a modified ZFS that has a transactional API, that can run Linux drivers in userspace with zero or few changes, and that uses a uniform event API like Windows' handles or Plan 9's file descriptors. But that modified ZFS is perhaps the most important part, and I would want to make it so `fsync()` on that platform behaves as people would want it to: transactional such that if it succeeds, the data was written, and if it doesn't, that data was not written. The ability to run Linux drivers is so there isn't a chicken-and-egg problem with drivers. (It would also be nice to implement the POSIX API to solve that part of the chicken-and-egg problem.) The uniform event API is because the current OS API's are difficult to work with. OS's are based around resources and events. The resource API's are pretty good, but the event API's (select(), poll(), epoll(), io_uring, kqueue, WaitForMultipleObjects(), etc.) are still artificially constrained.

xorcist 4y ago
That's one of those things that sounds good in theory, but in practice has more to do with hardware than software semantics. Any end-to-end guarantees pretty much requires control over hardware, which of course would be very interesting to a lot of people, but isn't really what mainstream operating system concern themselves with. Already the block device hides too much of hardware, and any VFS layer will be even worse. Just because a write is done doesn't mean it's actually on the spinning rust. And even if it is, it could be done in several ways, maybe it was relocated, or parity hasn't been computed, or it's in the queue. Any attempt of abstracting different types of storage will pretty much have to resort to the least bad common denominator here. I'm not saying it's a bad idea, just that it's not the same problem that operating systems try to solve. But in the post-Optane world maybe block devices are too low level anyway and we'll finally see higher level storage systems.
- ghoward 4y ago
  You're right, for the most part. We have to have hardware that doesn't lie when the data made it to disk, whatever that disk is. And we also need hardware that can do atomic updates to single sectors. But once we have those things, and people are demanding them more and more, a copy-on-write filesystem should be perfect for emulating transactions, using that single sector atomic update to "commit" the transaction.

anamax 4y ago

ReiserFS was headed that way before Reiser went to prison for murdering his wife. Other filesystems may also be headed that way.

jandrewrogers 4y ago

There is an impedance mismatch between what is required out of filesystems in terms of backward compatibility and the features/capabilities that would be useful for applications like databases. If you discard a requirement to look like a standard Linux filesystem to arbitrary applications, writing a purpose-built filesystem for a specific application often requires less code than trying to make off-the-shelf filesystems consistently deliver desired behavior and performance. And even then it is often difficult to guarantee that the off-the-shelf filesystem will really do what you expect in all cases. Installing a custom filesystem isn't always a practical option operationally but it is sometimes done for applications like high-scale data infrastructure because of the headaches it eliminates. I've built a few systems designed to be deployed either way. I am not sure there is a "one size fits all" solution for this. Alternative filesystem implementations tend to make very application-specific and hardware-aware design choices.