3 ms·
Standard practice to store a large number of files but in such a way that they don't end up in a single directory, is to create a crypto hash of your object nam
by oxplot 10y ago
Standard practice to store a large number of files but in such a way that they don't end up in a single directory, is to create a crypto hash of your object names and use the first n characters as the directory name. Properties of hash functions will ensure that the files are distributed evenly between all the directories.
git is an example of a program that uses this technique (among others) to store blobs. In case of git, the blob names are already SHA-1 hashes.
- majewsky 10y agoYou don't actually need a cryptographic hash here [1]. I'd rather use a fast hash like Murmur instead. [1] Exception: I don't know how prone Murmur is to hash collisions, so if collision-based attacks are part of your threat model, a randomly salted cryptographic hash might be appropriate.