9 ms·
> I can imagine some performance concerns in the real world. A friend has pointed out that, for performance reasons, games already tend to sidestep the filesys
by Expurple 1y ago
> I can imagine some performance concerns in the real world.
A friend has pointed out that, for performance reasons, games already tend to sidestep the filesystem and bundle everything into container/archive files, like .pak [1]. It also reminded me of how compile times are often noticeably faster on Linux vs Windows, because its filesystem is better optimized for handling many small files.
Honestly, it's weird and makes me kind of sad. Filesystems are such an important, convenient and (mostly) standardized and portable abstraction. But to this day, it's often a bottleneck and too slow for some domains. It seems like there's a lot of missed opportunity here.
> when the "tree" is actually a sequential array of nameless elements, there's some extra overhead involved with writing and storing made-up file names, as well as sorting by name when reading. This could be remedied by some new API.
As I keep thinking about it, the fastest approach is still a sequential binary blob. It avoids indirection, fragmentation and needlessly storing separate metadata for each element. A VFS mount could still be implemented on top, for accessing elements through a filesystem interface (something like `my_array_file/0`, reporting the same medatata as the parent).
But if storing separate FS-level metadata for each element is actually desirable and indirection+fragmentation isn't a critical problem, we could use folders for arrays. As a partial optimization, we could introduce a special "ordered" folder type. It requires explicitly ordering the subfiles on write, so that later listing the children is sorted and fast by default.
Option 1. The ordering is a separate metadata in the folder. Subfiles still have unique names that are not related to the ordering. This could be useful for the app, or could be unnecessary. POSIX apps (that don't know about "sorted folders") would still re-sort the subfiles by name and could get a different ordering as the result.
Option 2. The subfiles don't have user-provided names. For POSIX-compatibility, the VFS supports "artificial" file names like 0000000, 0000001, 0000002 (the number of digits according to the FS limits, sorts as text correctly) or 0, 1, 2 (prettier, independent of the FS limits and portable, but doesn't sort correctly as text).
In some sense, this special folder type is against the spirit of the original article. It has special write restrictions that prevent treating every inode the same way (as an arbitratily writetable folder). But it's still in the spirit of the original in the sense of reusing the standard FS abstractions and features as much as possible.
[1] https://quakewiki.org/wiki/.pak https://quakewiki.org/wiki/.pak