In general, filesystem code isn't optimized to handle millions of files in a single folder. It's generally optimized for thousands, and generally under 100,000

In general, filesystem code isn't optimized to handle millions of files in a single folder. It's generally optimized for thousands, and generally under 100,000 files per folder. (I used to work on an industry-leading sync product, Syncplicity) In general, the UI in Explorer and Finder isn't designed to handle more then a couple-hundred, or low thousand, files in a folder. The whole UI metaphors just fall apart when you have millions of files in a folder. There's plenty of other applications that can handle such situations better. Thus why optimize the filesystem? It's not a database, and if you need a database, you should use a database. SQLite is a wonderful database, and there are plenty more. But, getting back to the APIs: Even though the Windows kernel and lower-level APIs will let you enumerate files in a folder via paging and filters, there's nothing that forces an application to use paging or OS filters. It might not even be possible: The Windows kernel sends wildcard strings to the filesystem driver; but it doesn't send regexes. So, if you need to filter files in a directory by names that match a regex, you have to load all filenames into your process. A database like SQLite will have indexes and better filtering capabilities. Furthermore, filesystems generally are block storage. Each file takes up, on disk, the number of blocks that it needs, rounded up. For example, your 10k file will take up a whole 100k block, if 100k is your block size. Again, a database like SQLite, (or a zip file,) will be much more efficient.