4 ms·

This is fascinating, a bit in the same way that looking at accidents is interesting. A good synthesis is: filesystem API design is obviously a problem, given t

by ovi256 7y ago

This is fascinating, a bit in the same way that looking at accidents is interesting. A good synthesis is: filesystem API design is obviously a problem, given that people that specialize in using them can't do it correctly: "Pillai et al., OSDI’14 looked at a bunch of software that writes to files, including things we'd hope would write to files safely, like datbases and version control systems ... they found that every single piece of software they tested except for SQLite in one particular mode had at least one bug ... programmers who work on things like Leveldb, LBDM, etc., know more about filesystems than the vast majority of programmers ... they still can't use files safely every time"

pjc50 7y ago
> filesystem API design is obviously a problem It's an API that's so critical it's almost impossible to change or rewrite. The API itself has barely moved in 30 years, whether it's POSIX or Win32. Possibly the only widely adopted change in filesystem API has been "S3" and compatibles, which provide a completely different set of atomicity semantics as well as being network-native.
- derefr 7y ago
  > Possibly the only widely adopted change in filesystem API has been "S3" and compatibles, which provide a completely different set of atomicity semantics as well as being network-native. I would love if operating systems exposed a local object storage syscall API (with object versioning and all that good stuff.) It could be implemented on top of the filesystem for all I care, as long as it's an abstraction with safe object-storage semantics, enforced by the kernel and exposed to all processes without a need for library support. I believe that processes that wanted to "base" themselves entirely on object-storage would still need to touch the fileystem abstraction as well, mostly to allocate temporary on-disk "buffers" to gradually write to before submitting them to the object-store ABI as new object bodies. But: 1. Most such buffers would be small enough that you could get away with using an anonymous mmap(2) instead, keeping the file "on disk" in the page file rather than in the filesystem itself. 2. For objects you're writing to by streaming, with an unbounded eventual size, you could do the same trick Google Cloud Storage does: allocate a series of fixed-size "chunk" objects to receive the stream data, closing one and opening the next as each previous chunk gets "filled"; and then expose an API call to concatenate such chunk objects together on the object-storage kernel "backend" into single files (probably in O(1) time, because at a low level it's just concatenating disk extent lists.) 3. For other unbounded-size buffers, you could also have the object-store-kernel-daemon provide an API where it manages "large durable working copy" files for you, sort of "checking out" objects into file descriptors (probably using copy-on-write file clones on the backend), then "checking in" file descriptors to become new versions of those objects (maybe even "helpfully" avoiding doing so if the buffer hasn't been touched.)
  - BubRoss 7y ago
    What's the difference between an 'object' and a file?
    - pjc50 7y ago
      "Objects" in S3 are basically immutable. You can't overwrite the middle of an object or append to it - you can only overwrite the object as a a whole. This makes concurrent access much easier to reason about by making certain operations impossible. You couldn't build a database on an S3 object, for example. So you wouldn't want S3 as your only filesystem. But a combination of S3 semantics and block storage semantics covers the use cases of virtually all applications, with a bit of extra work to cover streaming writes.
- adrianN 7y ago
  The typical solution is to provide a new, safe API while supporting the broken stuff forever.
  - wvenable 7y ago
    Except if the API's have different semantics and you can't reconcile them. Then your new API can't be significantly different from your old API or your old API gets broken on the edge cases. That's the situation we are in with file systems API's now.
jandrewrogers 7y ago
I write database engines for Linux and the filesystem situation really is a train wreck. It isn't anyone's fault per se, design choices and standards were accreted over many decades that in isolation made sense in some context but which in aggregate have many poorly defined interactions and create conflicting requirements. And you can't change any of it easily because there are several decades of software built using the existing design. Guaranteeing precise, consistent behavior is nearly impossible with the standard filesystem API infrastructure and the implementation details change invisibly in important ways. This is evident in database engines if you look at the number of lines of code dedicated to storage. Working with raw block devices requires the fewest lines of code, working through the filesystem requires the most, and direct I/O (partial filesystem bypass) is somewhere in the middle. And even if you design for comparable guarantees across these models for working with storage, the consistency of behavior across Linux environments also significantly improves as you bypass more of the filesystem. Ironically, I now tend to borrow the low-level storage interface from database kernels, which abstracts the filesystem mess, for all code that needs to work with storage even if it isn't a database. It provides a saner interface with more consistent guarantees and often better performance. In my ideal world, someone properly designs a completely new filesystem API from scratch that sits alongside the legacy APIs that applications could start migrating to. But it would probably require adverse changes in the way the Linux kernel works for the legacy path, which means it will never happen.
- brandmeyer 7y ago
  Very few applications need random read/write access to files. Most of the time, you need to read an entire file in, or write an entire file out via the streaming access APIs. This core fact of typical usage is why I think so many application developers have naive expectations about filesystem behavior. Read-only random-access is well served by mmap() and pread(). For random writes access within a file, preadv2() and pwritev2() could be augmented with additional flags RWF_ACQUIRE and RWF_RELEASE. That's Linux-specific, but it could give database developers the ability to separate ordering with barriers from flushing with fsync. But perhaps I'm being naive. My assumption is that database developers are using flushes in order to get the barriers that they really want.

by ovi256 7y ago

pjc50 7y ago

> filesystem API design is obviously a problem It's an API that's so critical it's almost impossible to change or rewrite. The API itself has barely moved in 30 years, whether it's POSIX or Win32. Possibly the only widely adopted change in filesystem API has been "S3" and compatibles, which provide a completely different set of atomicity semantics as well as being network-native.

derefr 7y ago
> Possibly the only widely adopted change in filesystem API has been "S3" and compatibles, which provide a completely different set of atomicity semantics as well as being network-native. I would love if operating systems exposed a local object storage syscall API (with object versioning and all that good stuff.) It could be implemented on top of the filesystem for all I care, as long as it's an abstraction with safe object-storage semantics, enforced by the kernel and exposed to all processes without a need for library support. I believe that processes that wanted to "base" themselves entirely on object-storage would still need to touch the fileystem abstraction as well, mostly to allocate temporary on-disk "buffers" to gradually write to before submitting them to the object-store ABI as new object bodies. But: 1. Most such buffers would be small enough that you could get away with using an anonymous mmap(2) instead, keeping the file "on disk" in the page file rather than in the filesystem itself. 2. For objects you're writing to by streaming, with an unbounded eventual size, you could do the same trick Google Cloud Storage does: allocate a series of fixed-size "chunk" objects to receive the stream data, closing one and opening the next as each previous chunk gets "filled"; and then expose an API call to concatenate such chunk objects together on the object-storage kernel "backend" into single files (probably in O(1) time, because at a low level it's just concatenating disk extent lists.) 3. For other unbounded-size buffers, you could also have the object-store-kernel-daemon provide an API where it manages "large durable working copy" files for you, sort of "checking out" objects into file descriptors (probably using copy-on-write file clones on the backend), then "checking in" file descriptors to become new versions of those objects (maybe even "helpfully" avoiding doing so if the buffer hasn't been touched.)
- BubRoss 7y ago
  What's the difference between an 'object' and a file?
  - pjc50 7y ago
    "Objects" in S3 are basically immutable. You can't overwrite the middle of an object or append to it - you can only overwrite the object as a a whole. This makes concurrent access much easier to reason about by making certain operations impossible. You couldn't build a database on an S3 object, for example. So you wouldn't want S3 as your only filesystem. But a combination of S3 semantics and block storage semantics covers the use cases of virtually all applications, with a bit of extra work to cover streaming writes.
adrianN 7y ago
The typical solution is to provide a new, safe API while supporting the broken stuff forever.
- wvenable 7y ago
  Except if the API's have different semantics and you can't reconcile them. Then your new API can't be significantly different from your old API or your old API gets broken on the edge cases. That's the situation we are in with file systems API's now.

jandrewrogers 7y ago

I write database engines for Linux and the filesystem situation really is a train wreck. It isn't anyone's fault per se, design choices and standards were accreted over many decades that in isolation made sense in some context but which in aggregate have many poorly defined interactions and create conflicting requirements. And you can't change any of it easily because there are several decades of software built using the existing design. Guaranteeing precise, consistent behavior is nearly impossible with the standard filesystem API infrastructure and the implementation details change invisibly in important ways. This is evident in database engines if you look at the number of lines of code dedicated to storage. Working with raw block devices requires the fewest lines of code, working through the filesystem requires the most, and direct I/O (partial filesystem bypass) is somewhere in the middle. And even if you design for comparable guarantees across these models for working with storage, the consistency of behavior across Linux environments also significantly improves as you bypass more of the filesystem. Ironically, I now tend to borrow the low-level storage interface from database kernels, which abstracts the filesystem mess, for all code that needs to work with storage even if it isn't a database. It provides a saner interface with more consistent guarantees and often better performance. In my ideal world, someone properly designs a completely new filesystem API from scratch that sits alongside the legacy APIs that applications could start migrating to. But it would probably require adverse changes in the way the Linux kernel works for the legacy path, which means it will never happen.

brandmeyer 7y ago
Very few applications need random read/write access to files. Most of the time, you need to read an entire file in, or write an entire file out via the streaming access APIs. This core fact of typical usage is why I think so many application developers have naive expectations about filesystem behavior. Read-only random-access is well served by mmap() and pread(). For random writes access within a file, preadv2() and pwritev2() could be augmented with additional flags RWF_ACQUIRE and RWF_RELEASE. That's Linux-specific, but it could give database developers the ability to separate ordering with barriers from flushing with fsync. But perhaps I'm being naive. My assumption is that database developers are using flushes in order to get the barriers that they really want.