3 ms·

> - An IOCP-style API for non-blocking applications. This exists, it's called Linux AIO (distinct from Posix AIO). The problem is, when you use it, you have t

> - An IOCP-style API for non-blocking applications. This exists, it's called Linux AIO (distinct from Posix AIO). The problem is, when you use it, you have to reimplement caching (and buffering) in userspace instead of journaling – which is just as hard to get right and can just as easily – maybe even more easily – lead to corruption. (Postgres, as an example, relies on the OS to buffer writes.) > - Write() to be blocking / atomic by default. Don't return until data is safely committed. This is the wrong thing for 99% of use cases. You get terrible performance unless you batch things, which means that casual use ends up with severe performance issues. (IIRC this was actually a problem on Android devices – many apps were misusing SQLite by not using transactions, resulting in every single database update being a separate atomic write to disk. Not only did this kill performance, but it caused excessive flash wear.) > Modern filesystems already have journals - buggy reimplementions of journals in userland doesn't help anyone. Databases (among other systems) need features and control over the journal that a filesystem cannot provide. (Think MVCC, replication, etc.) Beside – someone has to write the filesystem, and that filesystem uses largely the same mechanisms in the kernel that are exposed to userspace. I agree fully that fsync ought to be deprecated for something with much more clearly-defined semantics. Both OS X and Linux have made attempts at this (F_FULLSYNC and sync_file_range, respectively), though clearly at least Linux still has some work to do. But – barring such unclear semantics – the general model of using fsync to guarantee ordering is not a complex one to understand, and matches most use cases well.

paulmd 9y ago
Congratulations, you've just described Oracle Database Filesystem. https://docs.oracle.com/cd/E11882_01/appdev.112/e18294/adlob_fs.htm#ADLOB45943 https://docs.oracle.com/cd/E11882_01/appdev.112/e18294/adlob...
deleted 9y ago
[deleted]
anarazel 9y ago
> (Postgres, as an example, relies on the OS to buffer writes.) We don't really buffer writes via the OS that much, most of them are buffered in postgres' internal buffer cache. We even force the kernel's hand to write out things earlier (otherwise there often are huge latency spikes). It's more reads where it's useful because it allows the OS to cache as much IO data as the current memory situation on the system makes sensible. > > Modern filesystems already have journals - buggy reimplementions of journals in userland doesn't help anyone. You end up with journals anyway. There's no way to be performant otherwise, because you'd do hundreds of tiny writes instead of a few bigger journal writes. Imagine if your 1-row / 3 index transaction would have to write out ~7 blocks.
- josephg 9y ago
  > You end up with journals anyway. There's no way to be performant otherwise, because you'd do hundreds of tiny writes instead of a few bigger journal writes. Imagine if your 1-row / 3 index transaction would have to write out ~7 blocks. We can make that fast by letting the OS / libc batch write transactions. Although if `write()` blocks until the batch commits & gets called in a loop we'd have a performance problem. Non-blocking APIs don't have that problem though. The guarantee I want is that every write moves the system between two states: (not written)->(written). It should be impossible to reach some third (partially written) state through disk failure or an inappropriate power loss.
  - vidarh 9y ago
    > We can make that fast by letting the OS / libc batch write transactions. Those are two very different things, potentially. If the write traverse a userspace-kernel boundary, there is no way to make that fast for small writes. If the batching happens entirely userspace, but just in a standard implementation, then sure. But people tend to severely underestimate how expensive syscalls are.
  - colanderman 9y ago
    > We can make that fast by letting the OS / libc batch write transactions. That's not the problem. (OSes already do this.) The problem is one of locality. (Data writes are spread out among many parts of the disk, causing many writes to be issued for any given transaction.) You get around this by (1) logging to a journal, then (2) batching many seconds worth of writes in the hopes that they touch fewer blocks than the sum of their parts (or, in the case of rotary media, that they can be ordered efficiently). You can't just "wait" for this to happen before responding to the client unless you like 5+ second service times. Which brings us back to the necessity of a journal, which, as I already stated, must be managed by the application because different applications require different features and control which the filesystem doesn't (and likely can't) expose.
josephg 9y ago
> > - Write() to be blocking / atomic by default. Don't return until data is safely committed. > This is the wrong thing for 99% of use cases. You get terrible performance unless you batch things, which means that casual use ends up with severe performance issues. The right answer would be to batch writes in the OS, but of course this would deadlock without non-blocking APIs. Maybe you're right - maybe write() should have the effectively non-blocking API it has now with fsync as a commit. But I still want my guarantee that if the system / hardware dies halfway through a write (or batch of writes) then data won't be left in a corrupt half-written state. Package installations and OS upgrades should be able to do the entirety of their work in a filesystem-level transaction. I shouldn't need to worry about my OS needing a reinstall because my laptop ran out of power halfway through an update. I just can't think of any sensible way to do this using the APIs we have today. > Beside – someone has to write the filesystem, and that filesystem uses largely the same mechanisms in the kernel that are exposed to userspace. But importantly not the same mechanisms. The OS has control over write ordering, and most block devices support read & write barriers. Filesystems lean heavily on these primitives for correctness - and for good reason; they're really useful primitives for building correct, performant systems. But for some reason they aren't exposed to userspace. Userspace applications are instead left scratching in the dirt with fsync. It would be much easier to build correct, performant databases in userspace if we had access to this stuff.
- colanderman 9y ago
  A filesystem-level transaction would be a good idea. Maybe some FSes (e.g. ZFS) already support such a thing. You can get this (somewhat awkwardly) for a single file with any FS that supports copy-on-write by making a (lazy) copy of the file, doing your updates there, and then renaming it to the original file name (which is guaranteed to be atomic on most FSes). With smaller files this is standard practice even on FSes without copy-on-write. > most block devices support read & write barriers No, they really don't. Some hardware devices might. I'm not aware of any SAN that does. (I've specifically asked this question of the developers of three.) > Filesystems lean heavily on these primitives for correctness In Linux, not since 2010 they don't [1], for exactly the reason that they're poorly implemented by devices. [1] https://lwn.net/Articles/400541/ https://lwn.net/Articles/400541/

paulmd 9y ago

Congratulations, you've just described Oracle Database Filesystem. https://docs.oracle.com/cd/E11882_01/appdev.112/e18294/adlob_fs.htm#ADLOB45943 https://docs.oracle.com/cd/E11882_01/appdev.112/e18294/adlob...

deleted 9y ago

[deleted]

anarazel 9y ago

> (Postgres, as an example, relies on the OS to buffer writes.) We don't really buffer writes via the OS that much, most of them are buffered in postgres' internal buffer cache. We even force the kernel's hand to write out things earlier (otherwise there often are huge latency spikes). It's more reads where it's useful because it allows the OS to cache as much IO data as the current memory situation on the system makes sensible. > > Modern filesystems already have journals - buggy reimplementions of journals in userland doesn't help anyone. You end up with journals anyway. There's no way to be performant otherwise, because you'd do hundreds of tiny writes instead of a few bigger journal writes. Imagine if your 1-row / 3 index transaction would have to write out ~7 blocks.

josephg 9y ago
> You end up with journals anyway. There's no way to be performant otherwise, because you'd do hundreds of tiny writes instead of a few bigger journal writes. Imagine if your 1-row / 3 index transaction would have to write out ~7 blocks. We can make that fast by letting the OS / libc batch write transactions. Although if `write()` blocks until the batch commits & gets called in a loop we'd have a performance problem. Non-blocking APIs don't have that problem though. The guarantee I want is that every write moves the system between two states: (not written)->(written). It should be impossible to reach some third (partially written) state through disk failure or an inappropriate power loss.
- vidarh 9y ago
  > We can make that fast by letting the OS / libc batch write transactions. Those are two very different things, potentially. If the write traverse a userspace-kernel boundary, there is no way to make that fast for small writes. If the batching happens entirely userspace, but just in a standard implementation, then sure. But people tend to severely underestimate how expensive syscalls are.
- colanderman 9y ago
  > We can make that fast by letting the OS / libc batch write transactions. That's not the problem. (OSes already do this.) The problem is one of locality. (Data writes are spread out among many parts of the disk, causing many writes to be issued for any given transaction.) You get around this by (1) logging to a journal, then (2) batching many seconds worth of writes in the hopes that they touch fewer blocks than the sum of their parts (or, in the case of rotary media, that they can be ordered efficiently). You can't just "wait" for this to happen before responding to the client unless you like 5+ second service times. Which brings us back to the necessity of a journal, which, as I already stated, must be managed by the application because different applications require different features and control which the filesystem doesn't (and likely can't) expose.

josephg 9y ago

> > - Write() to be blocking / atomic by default. Don't return until data is safely committed. > This is the wrong thing for 99% of use cases. You get terrible performance unless you batch things, which means that casual use ends up with severe performance issues. The right answer would be to batch writes in the OS, but of course this would deadlock without non-blocking APIs. Maybe you're right - maybe write() should have the effectively non-blocking API it has now with fsync as a commit. But I still want my guarantee that if the system / hardware dies halfway through a write (or batch of writes) then data won't be left in a corrupt half-written state. Package installations and OS upgrades should be able to do the entirety of their work in a filesystem-level transaction. I shouldn't need to worry about my OS needing a reinstall because my laptop ran out of power halfway through an update. I just can't think of any sensible way to do this using the APIs we have today. > Beside – someone has to write the filesystem, and that filesystem uses largely the same mechanisms in the kernel that are exposed to userspace. But importantly not the same mechanisms. The OS has control over write ordering, and most block devices support read & write barriers. Filesystems lean heavily on these primitives for correctness - and for good reason; they're really useful primitives for building correct, performant systems. But for some reason they aren't exposed to userspace. Userspace applications are instead left scratching in the dirt with fsync. It would be much easier to build correct, performant databases in userspace if we had access to this stuff.

colanderman 9y ago
A filesystem-level transaction would be a good idea. Maybe some FSes (e.g. ZFS) already support such a thing. You can get this (somewhat awkwardly) for a single file with any FS that supports copy-on-write by making a (lazy) copy of the file, doing your updates there, and then renaming it to the original file name (which is guaranteed to be atomic on most FSes). With smaller files this is standard practice even on FSes without copy-on-write. > most block devices support read & write barriers No, they really don't. Some hardware devices might. I'm not aware of any SAN that does. (I've specifically asked this question of the developers of three.) > Filesystems lean heavily on these primitives for correctness In Linux, not since 2010 they don't [1], for exactly the reason that they're poorly implemented by devices. [1] https://lwn.net/Articles/400541/ https://lwn.net/Articles/400541/