3 ms·
I worked at a place where the directory was split based on a character or two from the start of a hash. They had millions of profile images but didn't want the
by jonwinstanley 4y ago
I worked at a place where the directory was split based on a character or two from the start of a hash.
They had millions of profile images but didn't want them all in one directory, so they hashed profile ID and used the first 2 letters as the name of a sub-directory. So you end up with sub-directories called aa, ab, ac, ad etc.
It's not perfect but I suppose the original creator had seen issues in the past when directories have too many files in them.
- pixl97 4y agoThings like Squid Cache/Proxy did (does?) this so you don't stick millions of files on a buys cache in the same directory and kill performance. Especially back in the old days when things were on spinning disks.
- recuter 4y agoSeems perfectly fine. I remember having to do this trick on s3 for similar reasons.
- layer8 4y agoIdeally, you want the number of letter combinations to roughly correspond to the square root of number of files you expect to store, to balance the number of directories with the number of files stored per directory. Assuming 26 letters, two letters allow 676 combinations, so the total expected number of files shouldn’t exceed ~457.000 (676 squared), or else each subdirectory runs the risk of containing more than 676 files on average (making it unbalanced with the number of directories). More realistically, you would use base32 or hex encoding. For example, if you want to store up to around 16 million files (2^24) and use a 64-bit hash with hex encoding (16 hex characters), you would use the first three hex characters (12 bits) for the directory name (since 2^12 is the square root of 2^24), and the remaining 13 hex characters for the file name. As a result, the 16 million files would be stored in 4096 directories with roughly 4000 files each.