4 ms·
I know the article is for Linux... I recently was involved with a Windows filesystem driver. Windows lowest-level usermode APIs support iterating through direc
by gwbas1c 5y ago
I know the article is for Linux...
I recently was involved with a Windows filesystem driver. Windows lowest-level usermode APIs support iterating through directory contents for infinitely sized directories. (You just need to make sure whatever wrapper your language provides doesn't wrap the result in a pre-allocated array.)
For example, in C#, the System.IO.Directory static methods that return IEnumerables will let you iterate over millions of files without issue, at least on Windows. They probably will do the same on Linux and Mac.
- thewakalix 5y agoIsn’t that comparing apples with oranges? Low-level APIs on both OSes can deal with very large directories — the question is whether the high-level APIs can. (For that matter, streaming isn’t a panacea if your goal is, as the article states, to determine the number of files in the directory.)
- gwbas1c 5y agoYes, the high-level APIs can. As I mentioned, "For example, in C#, the System.IO.Directory static methods that return IEnumerables will let you iterate over millions of files without issue, at least on Windows." In C#, at that point counting the number of files in a giant directory is trivial. (I was wrong about the method being static) Basically, "new DirectoryInfo(path).EnumerateFiles().Count()" is all you need. You're going to hit the most efficient APIs on Windows, so all you're limited by is hardware speed. (It's also worth checking on Linux.)