3 ms·
It's a general issue that occurs whenever you scale from small to large things. Generally speaking, things that are expected to be "small" are processed as who
by crazygringo 4y ago
It's a general issue that occurs whenever you scale from small to large things.
Generally speaking, things that are expected to be "small" are processed as whole units. Memory is allocated to store the whole thing, a function won't return until it's processed the whole thing, the whole thing gets copied multiple times, and any kind of sorting/lookups often just scan the whole thing because with small things that's fastest/simplest.
When you expect something to be large, you architect things totally differently -- you buffer/stream rather than allocate, you pass pointers rather than copying data, you use indexes/hashes for lookups rather than full scans, and so forth.
And generally, filesystem paradigms are designed around small-number assumptions for number of files in directories, and hierarchy depths, and filename lengths, etc. Because these are all things you interact with using "human-scale" tools like GUI file managers and 'ls' commands. Whereas file sizes and disk sizes use large-number assumptions, because a video file is easily 10 GB or much more.
- gwbas1c 4y agoHaving a shipped a Windows filesystem driver, I can tell you that Windows's lowest level filesystem APIs "you buffer/stream rather than allocate, you pass pointers rather than copying data, you use indexes/hashes for lookups rather than full scans, and so forth" Memory was a lot more limited when those APIs were designed! Then again, many programming languages expose simplified APIs for the exact reasons you describe. It's a lot easier to encapsulate all of the above when all you care about is getting an array of filenames in a directory, and then performing your own application logic to filter it further.