4 ms·
I've often thought it would be useful (especially for software that manages container or archive file-formats, or for databases) to have exposed to the user a f
by derefr 7y ago
I've often thought it would be useful (especially for software that manages container or archive file-formats, or for databases) to have exposed to the user a file-system object that operates like a file, but which—instead of being exposed by OS as a seekable byte stream that can be appended/truncated on one end—is exposed by syscalls as a vector of arbitrary, non-uniformly-sized extents, where the user is expected to ask the OS to pre-allocate extent buffers (think mmap(2) with anonymous private disk pages), and then, separately, stuff those into this vector object (which would be an operation almost exactly like hard-linking an existing file into a directory, in terms of its time and space complexity.)
Of course, for most filesystems (and especially ones with sparse-file support, and especially ones with copy-on-write support), such an on-disk data structure is exactly what's already underlying the byte-stream abstraction. So this would just be a passthrough to allow people to directly manipulate that data structure. (In the process probably breaking certain preconditions the filesystem relies on, though, so it would need to track these "low-level extent vectors" as a separate filesystem object type. Programs would still be able to use regular file-abstraction syscalls on them, though.)
Interestingly, despite filesystems themselves not exposing the lower-level extent-vector abstractions, in some other systems that have file-stream-like abstractions, you can operate on "files" this way.
Postgres's BLOBs, for example, are seekable byte-streams, which also (at least theoretically) allow you to insert into the middle of them. (I say theoretically because it's not an implemented API, but it's not exactly hidden from the user, either. BLOBs just get broken out into records in a table representing their extents; you can rewrite the keys of said records in that table to do whatever low-level operations you like.)
Or, for another example, S3 and its competitors let you arbitrarily compose objects as “components” of other virtual objects, which then read back as the concatenation of the objects they contain. (Sadly you don’t get any other vector-manipulation ops than this, but if you keep references to the leaf extents around, that’s often enough to rebuild your extent-vectors any time they change.)
Also, of course, back on the real filesystem, you can just manage your "vector of parts of a file" as multiple files in a directory, and then abstract over that directory that by using a FUSE server which exposes a view where the files are one contiguous file.
- EvanAnderson 7y agoThat sounds a lot like RMS files in the VMS operating system.
- JoshTriplett 7y agoToday, though, that filesystem abstraction is block-aligned. So you could delete blocks from the beginning or middle of a file easily, but you couldn't delete bytes from the beginning or middle.
- derefr 7y agoRight—the difference is that, with a file object, the extents are all assumed to be full to capacity except for the very last one, with the information on the virtual size within the last extent being held in the file's inode. Whereas, with these vector-of-buffers objects, either each extent would need to reserve two bytes at the beginning to represent the virtual size of that extent, or the vector would be a vector of disk slices at byte granularity, rather than a vector of plain disk offsets at sector granularity. I can see the arguments for both. The former is extremely efficient, and would probably be the go-to for databases that want to be performant without needing to dedicate a raw block device to them. The latter approach, though, you can implement entirely "on top of" the filesystem (in the same way that the ext3 journal, or APFS containers, are implemented "on top of" their filesystems.) You could have a regular file representing the storage reservation (in full extents) for the vector-object; and then the vector object could be a tree (or something else efficient to rewrite) of byte-granular slices of offsets into that backing storage-file, with logic to ensure that such a reserved slice never straddles two extents, and a cache in the entry for a pointer to the actual extent. These would be perfect for, say, generating archive files. In fact, since these are such different use-cases, there's no reason to not just give users both.
- JoshTriplett 7y agoFor that matter, byte-oriented persistent memory would allow something byte-oriented and not block-oriented.