4 ms·

Interesting, is there a blog around discussing this in detail ? If not would be kind enough to go more into detail.

nieve 6y ago
I don't think anyone posted about their particular system, but it's not unknown now. If you google "filesystem as a database" there are some relevant hits. One super simple and probably not ideal, but at least balanced version uses a hash of some primary key like customer row id as the file index, then partitions the items into directories with all permutations at each level (or only populated ones) based on successive parts of the hash. For example an item key that hashes to a32c4214585e9cb7a55474133a5fc986 would be located somewhere like this: a32c/4214/585e/9cb7/a554/74133a5fc986 a32c/ 4214/ 585e/ 9cb7/ a554/ 74133/a5fc986 The advantage of this kind of structure is that you never need to manually scan a directory since you know exactly what path you're trying to open. You still incur the OS lookup time for the inode-equivalent in the directory entry, but a deeper heirarchy keeps that faster. You can trade off time to traverse the heirarchy versus number of entries in the final directories by adjusting the length of the hash chunk you use at each level. Two characters will put vastly fewer entries at a given level, but vastly increase your directory depth. Basically if you're manually scanning the heirarchy for anything but a consistency check or garbage collection you've already lost.
- nieve 6y ago
  One important note: make sure you carefully consider using atomic renames and such for manipulating the files! Overwrite in place is a great way to end up with a corrupted item if something goes desperately wrong and you're not protected by COW or data journaling.
  - zaphar 6y ago
    Usually you write these sorts of things as append only with an optional garbage collect. You get a minimal sort of Atomicity with that.
- andreareina 6y ago
  That's how git stores its objects: 18:35 $ tree .git/objects/ .git/objects/ ├── 02 │ └── 9581d0c8ecb87cf1771afc0b4c2f1d9f7bfa82 ├── 3b │ └── 97b950623230bd218cef6aebd983eb826b2078 (...) ├── info └── pack ├── pack-b1fe2364423805afb6b1c03be0811c93b19dedc9.idx └── pack-b1fe2364423805afb6b1c03be0811c93b19dedc9.pack 10 directories, 10 files

I don't think anyone posted about their particular system, but it's not unknown now. If you google "filesystem as a database" there are some relevant hits. One super simple and probably not ideal, but at least balanced version uses a hash of some primary key like customer row id as the file index, then partitions the items into directories with all permutations at each level (or only populated ones) based on successive parts of the hash. For example an item key that hashes to a32c4214585e9cb7a55474133a5fc986 would be located somewhere like this: a32c/4214/585e/9cb7/a554/74133a5fc986 a32c/ 4214/ 585e/ 9cb7/ a554/ 74133/a5fc986 The advantage of this kind of structure is that you never need to manually scan a directory since you know exactly what path you're trying to open. You still incur the OS lookup time for the inode-equivalent in the directory entry, but a deeper heirarchy keeps that faster. You can trade off time to traverse the heirarchy versus number of entries in the final directories by adjusting the length of the hash chunk you use at each level. Two characters will put vastly fewer entries at a given level, but vastly increase your directory depth. Basically if you're manually scanning the heirarchy for anything but a consistency check or garbage collection you've already lost.

nieve 6y ago
One important note: make sure you carefully consider using atomic renames and such for manipulating the files! Overwrite in place is a great way to end up with a corrupted item if something goes desperately wrong and you're not protected by COW or data journaling.
- zaphar 6y ago
  Usually you write these sorts of things as append only with an optional garbage collect. You get a minimal sort of Atomicity with that.
andreareina 6y ago
That's how git stores its objects: 18:35 $ tree .git/objects/ .git/objects/ ├── 02 │ └── 9581d0c8ecb87cf1771afc0b4c2f1d9f7bfa82 ├── 3b │ └── 97b950623230bd218cef6aebd983eb826b2078 (...) ├── info └── pack ├── pack-b1fe2364423805afb6b1c03be0811c93b19dedc9.idx └── pack-b1fe2364423805afb6b1c03be0811c93b19dedc9.pack 10 directories, 10 files