3 ms·

Or at least using a multilayered approach i.e. all the files starting with A in the A subfolder. You still have to deal with that amount of files, but you don't

Or at least using a multilayered approach i.e. all the files starting with A in the A subfolder. You still have to deal with that amount of files, but you don't lose so much time dealing with filesystem/directory operations. And that might be better for accessing individual files outside the app instead of a big binary blob that you may have no clue on its format (or corruption/deletion of that single file). Thats why there are no universal solutions, there are many different use cases.

bob1029 4y ago
The multilayered approach is something we also do in one area, but at a certain point you are still playing dangerous games with the filesystem. The most recent innovation for us is to use a lot of smaller SQLite databases, each scoped to a specific customer, session or unit of work. This is still far superior to loose records on disk, but you also get clear separation and reasonable firewalls between system entities.
aargh_aargh 4y ago
I think the question should be why don't we have filesystems handle this implicitly using a tree structure behind the scenes. Are we so many layers above our hardware that it simply doesn't occur to us as the right solution?
- londons_explore 4y ago
  The unix filesystem API doesn't let you "give me a list of all files in this directory matching this pattern". Nor does it let you start iterating through a directory with getdents(), then pause, then come back to iterate the rest of the files later. You have to receive every file name in the order the OS wants to give them to you. That prevents applications really being able to make use of clever storage mechanisms - you can never find the most recent 3 files in a big directory with anything cheaper than a linear scan.
- dahauns 4y ago
  Well, NTFS works with B-Trees...so nope, it's not a silver bullet either, the bottleneck will just be somewhere else in the storage stack :) (But we have impressively speedy tools like Wiztree or Everything thanks to it.)
  - arantius 4y ago
    I've used WinDirStat but not Wiztree before -- Wiztree is so much faster!

bob1029 4y ago

The multilayered approach is something we also do in one area, but at a certain point you are still playing dangerous games with the filesystem. The most recent innovation for us is to use a lot of smaller SQLite databases, each scoped to a specific customer, session or unit of work. This is still far superior to loose records on disk, but you also get clear separation and reasonable firewalls between system entities.

aargh_aargh 4y ago

I think the question should be why don't we have filesystems handle this implicitly using a tree structure behind the scenes. Are we so many layers above our hardware that it simply doesn't occur to us as the right solution?

londons_explore 4y ago
The unix filesystem API doesn't let you "give me a list of all files in this directory matching this pattern". Nor does it let you start iterating through a directory with getdents(), then pause, then come back to iterate the rest of the files later. You have to receive every file name in the order the OS wants to give them to you. That prevents applications really being able to make use of clever storage mechanisms - you can never find the most recent 3 files in a big directory with anything cheaper than a linear scan.
dahauns 4y ago
Well, NTFS works with B-Trees...so nope, it's not a silver bullet either, the bottleneck will just be somewhere else in the storage stack :) (But we have impressively speedy tools like Wiztree or Everything thanks to it.)
- arantius 4y ago
  I've used WinDirStat but not Wiztree before -- Wiztree is so much faster!