4 ms·
Using file extensions (or having to open the file and inspect contents) to determine file type is a slow way to find files. If you have one of the newest hard d
by didgetmaster 3y ago
Using file extensions (or having to open the file and inspect contents) to determine file type is a slow way to find files. If you have one of the newest hard drives that is capable of storing 100M+ files on it (or a NAS made up of multiple, smaller drives); then it can take forever to find all files of a certain kind (e.g. photos, documents, etc.).
Just try to write a program that lists all your photos. You have to know all the various extensions that apply to still images (.jpg, .jpeg, .gif, .png, etc.) and do a string comparison against every file in the drive hierarchy. Might as well go to lunch waiting for it to finish if you have many millions of files to search.
This was the primary reason why I set out to build a better system that could scan through hundreds of millions of files and find all that matched a given subset in just a few seconds. https://www.Didgets.com https://www.Didgets.com
- fooker 3y agoYou seem to go out of your way to avoid mentioning 'index' when that's the usual way to solve this problem.
- didgetmaster 3y agoApparently, your experience with various indexing services has been different than mine. I found it is just too easy for the index to become 'out of sync' with the actual file system and has to be occasionally rebuilt. Even with just 1M files, that always took a long time; but with 100M+ files?? If the storage was removeable or if you booted occasionally with another OS (or even just a different version of the same OS) then I found it was near impossible for an indexing service to keep track of all the changes.
- fooker 3y agoI was curious how slow it was for 1M files and tried it out on XFS and btrfs partitions. It was basically instant without any explicit indexing on my end.
- eviks 3y agoBut Everything does this kind of search instantly on NTFS
- porridgeraisin 3y agoYeah, it uses some kind of journal to keep updated on changes etc