If a file is large enough that it cannot be read into memory in a single pass, it does beg the question if it should be in a single file in the first place. If

If a file is large enough that it cannot be read into memory in a single pass, it does beg the question if it should be in a single file in the first place. If it is that big, there may end up being multiple simultaneous writers and readers accessing various parts, depending on the use case. Now you are getting into some pretty advanced territory and will eventually need to introduce a WAL log into your file format for consistency (or crude locks). Formats like HDF5 never bothered with this leaving it up to the user to ensure that writers and readers are not accessing the file simultaneously. A directory on a file system happens to have these capability already. If you absolutely need everything in a single enormous file perhaps consider using Sqlite.