3 ms·
For workloads with many small files, it usually is better to store many files in a single object. Filesystems with regular POSIX semantics, such as atomic direc
by objectivefs 3y ago
For workloads with many small files, it usually is better to store many files in a single object. Filesystems with regular POSIX semantics, such as atomic directory renames etc, also makes it easier to integrate with existing software. We have seen a lot of scientific computing usage of our filesystem (https://objectivefs.com https://objectivefs.com) and as you mentioned localized caching of the working set is key to great performance.
- carbocation 3y agoVery strongly agree with your point. In my case the real number of files is in the billions, and they are already aggregated into grouped files to reduce that overall size. But for a period of time I tried to operate on the individual unaggregated files, and that was totally untenable. (Also expensive due to the normally-negligible cost of fetching operations.)