3 ms·
> Possibly the only widely adopted change in filesystem API has been "S3" and compatibles, which provide a completely different set of atomicity semantics as we
by derefr 7y ago
> Possibly the only widely adopted change in filesystem API has been "S3" and compatibles, which provide a completely different set of atomicity semantics as well as being network-native.
I would love if operating systems exposed a local object storage syscall API (with object versioning and all that good stuff.)
It could be implemented on top of the filesystem for all I care, as long as it's an abstraction with safe object-storage semantics, enforced by the kernel and exposed to all processes without a need for library support.
I believe that processes that wanted to "base" themselves entirely on object-storage would still need to touch the fileystem abstraction as well, mostly to allocate temporary on-disk "buffers" to gradually write to before submitting them to the object-store ABI as new object bodies. But:
1. Most such buffers would be small enough that you could get away with using an anonymous mmap(2) instead, keeping the file "on disk" in the page file rather than in the filesystem itself.
2. For objects you're writing to by streaming, with an unbounded eventual size, you could do the same trick Google Cloud Storage does: allocate a series of fixed-size "chunk" objects to receive the stream data, closing one and opening the next as each previous chunk gets "filled"; and then expose an API call to concatenate such chunk objects together on the object-storage kernel "backend" into single files (probably in O(1) time, because at a low level it's just concatenating disk extent lists.)
3. For other unbounded-size buffers, you could also have the object-store-kernel-daemon provide an API where it manages "large durable working copy" files for you, sort of "checking out" objects into file descriptors (probably using copy-on-write file clones on the backend), then "checking in" file descriptors to become new versions of those objects (maybe even "helpfully" avoiding doing so if the buffer hasn't been touched.)
- BubRoss 7y agoWhat's the difference between an 'object' and a file?
- pjc50 7y ago"Objects" in S3 are basically immutable. You can't overwrite the middle of an object or append to it - you can only overwrite the object as a a whole. This makes concurrent access much easier to reason about by making certain operations impossible. You couldn't build a database on an S3 object, for example. So you wouldn't want S3 as your only filesystem. But a combination of S3 semantics and block storage semantics covers the use cases of virtually all applications, with a bit of extra work to cover streaming writes.