6 ms·
I've said it before and I'll say it again: Modern filesystems straight out expose the wrong API for 99% of applications. App developers almost never think of da
by josephg 9y ago
I've said it before and I'll say it again: Modern filesystems straight out expose the wrong API for 99% of applications. App developers almost never think of data as a stream of bytes. We think of data as a set of records. Values in the set change through atomic modification events.
The fundamental API primitives should be atomic changes. Atomic write (bytes), and atomic append. The funny thing about it is that POSIX already supports basically this API (datagrams) for both IPC and networking. It just doesn't support this API in the one place it would be most useful - the filesystem.
Ideally I want:
- Write() to be blocking / atomic by default. Don't return until data is safely committed.
- A transactional API: begin(fd); write(); write(); err = commit(fd). If any error happens, commit returns the error and none of the data is stored.
- An IOCP-style API for non-blocking applications. This is the API databases want to use, with the loop being <get network request>, <write data to filesystem>, <yield>, <get write completion event>, <send confirmation to client>.
- Deprecate fsync & friends. If you don't want to wait for the data to get committed, write in non-blocking mode and ignore the completion event.
Solving this problem in end-user applications is really hard - almost no applications implement atomic write on top of filesystems correctly. And they shouldn't have to - this should be the job of the filesystem. The filesystem can do this much more safely, with better guarantees, better performance and better error handling. Modern filesystems already have journals - buggy reimplementions of journals in userland doesn't help anyone. "Do not turn off console while game is saving" is an embarrassment to everyone.
- colanderman 9y ago> - An IOCP-style API for non-blocking applications. This exists, it's called Linux AIO (distinct from Posix AIO). The problem is, when you use it, you have to reimplement caching (and buffering) in userspace instead of journaling – which is just as hard to get right and can just as easily – maybe even more easily – lead to corruption. (Postgres, as an example, relies on the OS to buffer writes.) > - Write() to be blocking / atomic by default. Don't return until data is safely committed. This is the wrong thing for 99% of use cases. You get terrible performance unless you batch things, which means that casual use ends up with severe performance issues. (IIRC this was actually a problem on Android devices – many apps were misusing SQLite by not using transactions, resulting in every single database update being a separate atomic write to disk. Not only did this kill performance, but it caused excessive flash wear.) > Modern filesystems already have journals - buggy reimplementions of journals in userland doesn't help anyone. Databases (among other systems) need features and control over the journal that a filesystem cannot provide. (Think MVCC, replication, etc.) Beside – someone has to write the filesystem, and that filesystem uses largely the same mechanisms in the kernel that are exposed to userspace. I agree fully that fsync ought to be deprecated for something with much more clearly-defined semantics. Both OS X and Linux have made attempts at this (F_FULLSYNC and sync_file_range, respectively), though clearly at least Linux still has some work to do. But – barring such unclear semantics – the general model of using fsync to guarantee ordering is not a complex one to understand, and matches most use cases well.
- paulmd 9y agoCongratulations, you've just described Oracle Database Filesystem. https://docs.oracle.com/cd/E11882_01/appdev.112/e18294/adlob_fs.htm#ADLOB45943 https://docs.oracle.com/cd/E11882_01/appdev.112/e18294/adlob...
- deleted 9y ago[deleted]
- anarazel 9y ago> (Postgres, as an example, relies on the OS to buffer writes.) We don't really buffer writes via the OS that much, most of them are buffered in postgres' internal buffer cache. We even force the kernel's hand to write out things earlier (otherwise there often are huge latency spikes). It's more reads where it's useful because it allows the OS to cache as much IO data as the current memory situation on the system makes sensible. > > Modern filesystems already have journals - buggy reimplementions of journals in userland doesn't help anyone. You end up with journals anyway. There's no way to be performant otherwise, because you'd do hundreds of tiny writes instead of a few bigger journal writes. Imagine if your 1-row / 3 index transaction would have to write out ~7 blocks.
- josephg 9y ago> You end up with journals anyway. There's no way to be performant otherwise, because you'd do hundreds of tiny writes instead of a few bigger journal writes. Imagine if your 1-row / 3 index transaction would have to write out ~7 blocks. We can make that fast by letting the OS / libc batch write transactions. Although if `write()` blocks until the batch commits & gets called in a loop we'd have a performance problem. Non-blocking APIs don't have that problem though. The guarantee I want is that every write moves the system between two states: (not written)->(written). It should be impossible to reach some third (partially written) state through disk failure or an inappropriate power loss.
- vidarh 9y ago> We can make that fast by letting the OS / libc batch write transactions. Those are two very different things, potentially. If the write traverse a userspace-kernel boundary, there is no way to make that fast for small writes. If the batching happens entirely userspace, but just in a standard implementation, then sure. But people tend to severely underestimate how expensive syscalls are.
- gruez 9y ago>Ideally I want: everything you describe will murder IO performance because you're essentially implementing a database with MVCC. it makes much more sense to implement what you described as a VFS library.
- sethhochberg 9y agoZFS gets a lot of the way there - and indeed has lots of similarities to a database in its internals. (In fact, ZFS with `sync=always` set on the dataset would do exactly what the parent commenter is asking for without requiring a whole new API. If someone really cared about performance they could even make this perform well with a battery-backed DRAM device for their intent log... but by that point we're way outside of the discussion of a general-purpose filesystem. If you really need guaranteed consistency for every write _without_ crippling performance, ZFS can do it for you at a hardware cost. Writes get cached in RAM, batched, flushed in transaction groups, and the fast/durable SLOG protects against corruption or loss of power or device failure without harming performance much. $$$ though. You'd better have quite a bit of RAM, and need a proper enterprise-grade DRAM storage device - or probably two in a mirror if your data is this critical).
- josephg 9y agoMurder the IO performance of what, exactly? For what use case is fast non-atomic writes something we care about? Atomic-append is the IO operation we should be optimizing for! What murders IO performance is trying to reimplement a second write-ahead-log in userspace on top of fsync. Which is something modern databases have to actually do in order to meet their correctness requirements. This complexity and performance cost would be much lower with knowledge of the kernel & control over filesystem internals. And for every case of something that would have lower IO performance with atomic appends, I'm sure we can think of ways the kernel could make those atomic writes fast. Lets say you're copying a big file by wrapping a transaction around a read();write() loop. The OS could make the transaction extremely cheap by simply not changing the file's INODE list (or whatever) until the write has completed. In a filesystem like ZFS, using a transaction for this would let ZFS not update its filesystem checksums until the copy has finished, which would probably in aggregate result in improved performance.
- zie 9y agoThis is why SQLite advertises itself as a replacement for fopen(). It's crazy hard to get right, and SQLite did the work so we don't have to: "Think of SQLite not as a replacement for Oracle but as a replacement for fopen()" - https://sqlite.org/about.html https://sqlite.org/about.html Obviously this would be horrible for PG to use as a replacement for fopen() and other systems, but for many, many use-cases SQLite is a good replacement for fopen().
- copperx 9y agoI don't understand why PG would change fopen() to SQLite calls in Arc, if Arc is supossed to be a general purpose programming language in the first place. Are there any other languages doing that?
- harperlee 9y agoI think the gp is referring to pg as postgres, not Paul Graham. As in, "Obviously [it] would be horrible for [postgres to be used] as a replacement for fopen() and other systems, but for many, many use-cases SQLite is a good replacement for fopen()"
- kjeetgill 9y agoActually I think gp meant: "Obviously [SQLite] would be horrible for postgres to use as a replacement for fopen() and other systems, but for many, many use-cases SQLite is a good replacement for fopen()"
- quickben 9y ago> "ideally I want: - Write() to be blocking / atomic by default. Don't return until data is safely committed." How do you deal with user pulling out usb stick?
- cryptonector 9y agoNo no no, we need async I/O for files. Writing should not be synchronous, but you should be able to find out about completion of each write. Asynchrony is absolutely critical for performance. Filesystem I/O, and, really, disk/SSD I/O is really very much the same as network I/O nowadays. You might be traversing a SAN, for example, or the seek latency on HDD could be reasonable for an HDD but nowadays that's just way too high anyways. Transactional APIs are very difficult for filesystems. Barriers are enough for many apps. A barrier is also much easier to start using in existing apps. You'd call a system call to request a write barrier, and wait, preferably asynchronously, for completion notice (whereupon you'd know all writes done before the barrier must have reached stable storage).
- trentnelson 9y agoSo, basically, we need the NT-style, completion-oriented, inherently-asynchronous I/O model? ;-) (Amen!)
- davidgould 9y agoYou misspelled VMS as NT. And actually VMS borrowed the IO model from RSX-11. Async QIO style IO is all good clean fun, until someone writes something like: fml(...) { char buf[1024]; sys$qio(QIO_READ, buf, sizeof(buf)); return; } main() { fml(); do_stuff(); //surprise package delivered into stack here! }
- cryptonector 9y agoYes. Every system call that can block should take an event queue argument. If you want the system call to block then the C library's stub should wait for it for you.
- teddyh 9y ago> Modern filesystems straight out expose the wrong API for 99% of applications. App developers almost never think of data as a stream of bytes. We think of data as a set of records As I recall, older operating systems like VMS and MULTICS all did this, and it was all tremendously complex. Unix’ simple stream-of-bytes file abstraction was a reaction to this, and it worked so well that it became the prevailing model. Before doing again what didn’t work before, check up on why it failed the previous time.
- gaius 9y agoYour recollection is faulty; RMS was an absolute pleasure to work with. Fast, clean and efficient, and concurrent access was pain-free too. RMS supported streams as well as records. Others have mentioned SQLite as replacement for fopen()... that’s a fairly RMS-like experience on modern systems (I wonder if this is being downvoted because people think RMS means Stallman)
- jordanb 9y agoI think you guys are arguing worse is better versus the right thing. :) Unix made for very simple filesystem internals, with a crappy interface (worse is better). RMS was a very complicated filesystem with a nice interface (the right thing).
- davidgould 9y agoRMS was an absolute pleasure to work with. Fast, clean and efficient, and concurrent access was pain-free too. RMS supported streams as well as records. Hummmph, it was kind of a pleasure on a VAX, but on a PDP-11 under RSX or RSTS/E the libraries for it ate so much of the (16 bit) address space that it became a real challenge to fit a useful application into memory. Also, it only sort of supported streams, they were not really byte streams like Unix, but variable length record streams with CRLF marking the record delimiters. So porting Unix code to VMS was never quite as straightforward as one hoped.
- davidgould 9y agoUnix is older than VMS (source: programmed on VMS when it was new), so it is unlikely that the unix file abstraction is a reaction to VMS.
- quietbritishjim 9y agoMicrosoft introduced transactional file system (and registry) APIs in Windows Vista [1]. But it was so complex that no one ever used it, and now it is semi-officially deprecated. [1] https://en.wikipedia.org/wiki/Transactional_NTFS https://en.wikipedia.org/wiki/Transactional_NTFS
- branko_d 9y agoI used it, and I would disagree that it was "complex" to use. No more than non-transacted Win32 file APIs anyway. We did a nice P/Invoke wrapper around native Win32 functions in C#, exposing FileStream and such, that the rest of the code could then easily consume. I think the main complexity is that there are no "official" high-level wrappers the way there are for non-transacted APIs. But at the level of Win32, there is no significant difference.