3 ms·

What about on the other end? Why not have each hex digit be a directory along a path? Then you have very, very few files per directory at the cost of deeper hie

What about on the other end? Why not have each hex digit be a directory along a path? Then you have very, very few files per directory at the cost of deeper hierarchy. What's the practical downside?

thewakalix 6y ago
[epistemic status: amateur speculation] I’d expect it to take up slightly more disk space. More importantly, the more indirections, the longer it’ll take to get to the file.
pmiller2 6y ago
Directories are just another type of file. So, if you do this, and your filenames are n characters long, you'll end up needing to do n file accesses just to find the file you're looking for. Unless the underlying file system does something to make that particular access pattern fast, well... it's going to be stupidly slow after a certain point.
- loeg 6y ago
  > Unless the underlying file system does something to make that particular access pattern fast I don't know exactly how Linux does it. Windows hands off whole paths to the filesystem, so this idea is possible there. In FreeBSD, there is a generic routine (lookup(9)) that goes component by component, so at each step the filesystem is only asked to resolve a single component to a vnode. I think a clever filesystem implementation (in FreeBSD) could look at the remaining path and kick off asynchronous prefetch... but I am not aware of anything doing this.
  - eru 6y ago
    I've played around with fuse a bit. There are two modes you can implement for your filesystem: in the so-called 'high level' mode you get the whole path. In the 'low level' mode the filesystem asks you for one piece of the path at a time. The low level mode seemed faster in my tests, and I think it's also closer to how Linux kernel works internally?
ArneBab 6y ago
You then need to do more stat calls — one per hierarchy level — and you waste more inodes.
loeg 6y ago
More overhead for each extra layer. Kernel serialization/locking, random seeks (dirent and inode per level), maybe other concerns.
- eru 6y ago
  The random seeks aren't a given: your filesystem could do something clever with memory or disk layout to avoid most of them.
  - loeg 6y ago
    Anything particular in mind? You could probably get a good academic paper or patent out of this.

What about on the other end? Why not have each hex digit be a directory along a path? Then you have very, very few files per directory at the cost of deeper hierarchy. What's the practical downside?

thewakalix 6y ago

[epistemic status: amateur speculation] I’d expect it to take up slightly more disk space. More importantly, the more indirections, the longer it’ll take to get to the file.

pmiller2 6y ago

Directories are just another type of file. So, if you do this, and your filenames are n characters long, you'll end up needing to do n file accesses just to find the file you're looking for. Unless the underlying file system does something to make that particular access pattern fast, well... it's going to be stupidly slow after a certain point.

loeg 6y ago
> Unless the underlying file system does something to make that particular access pattern fast I don't know exactly how Linux does it. Windows hands off whole paths to the filesystem, so this idea is possible there. In FreeBSD, there is a generic routine (lookup(9)) that goes component by component, so at each step the filesystem is only asked to resolve a single component to a vnode. I think a clever filesystem implementation (in FreeBSD) could look at the remaining path and kick off asynchronous prefetch... but I am not aware of anything doing this.
- eru 6y ago
  I've played around with fuse a bit. There are two modes you can implement for your filesystem: in the so-called 'high level' mode you get the whole path. In the 'low level' mode the filesystem asks you for one piece of the path at a time. The low level mode seemed faster in my tests, and I think it's also closer to how Linux kernel works internally?

ArneBab 6y ago

You then need to do more stat calls — one per hierarchy level — and you waste more inodes.

loeg 6y ago

More overhead for each extra layer. Kernel serialization/locking, random seeks (dirent and inode per level), maybe other concerns.

eru 6y ago
The random seeks aren't a given: your filesystem could do something clever with memory or disk layout to avoid most of them.
- loeg 6y ago
  Anything particular in mind? You could probably get a good academic paper or patent out of this.