3 ms·
> to list a directory with millions of entries Fair, it'll likely break most tools written in the last 10-15 years. > (e.g., commits) that require a lot of wo
by TeMPOraL 3y ago
> to list a directory with millions of entries
Fair, it'll likely break most tools written in the last 10-15 years.
> (e.g., commits) that require a lot of work to find.
That's solvable with a cache. I'm surprised git doesn't seem to have one for those, at least going by how long it takes to generate a full "shorthand log" on a large repo.
- cryptonector 3y agoCaching is nice, but if it takes an hour when the cache is cold then it's not usable even if a directory with millions of entries posed no problems for any tools. Now for most projects that's not going to happen, but for something like the Linux kernel it just might. And there might be many caches to warm, and much to cache. For a large enough project and a computer that one might not think of as small that might just fail to work at all. This is why as Git grows up it's using Bloom filters to optimize things like git blame and git log on one file rather than adopting the fully relational model of Fossil. (Please read into this all the chagrin I'm feeling about that, because I'm quite the fan of Fossil's relational model.) PS, the single biggest problem with million-entry directories is the propensity of tools like ls(1) to want to sort the darned things, so one has to remember to use `ls -f`, or to at least use the C locale to get memset() collations instead of much much slower Unicode collations. Another problem is that the POSIX stat(2) family of functions combine reading metadata that could come from just the directory (e.g., a file's inode number) (and which contents has already been read) with reading metadata that requires [possibly much] extra I/O to get, so if you're doing `ls -l` on a million-entry directory you might as well go on vacation (but make sure to send the output to some file, cause your scrollback buffer just won't do).