5 ms·
At $oldjob, taking care of a busy and successful web estate that is now close to 25 years old, one of the ugliest and longest-standing warts was the "image stor
by c0l0 4y ago
At $oldjob, taking care of a busy and successful web estate that is now close to 25 years old, one of the ugliest and longest-standing warts was the "image store". That was a simple, flat directory on a single node, shared over NFS, which had accumulated more than 1.2 million (yes, 1_200_000) inodes/files in a single directory. No-one wanted (read: dared) to properly fix the code to rid it of this once-convenient assumption and (lack of) hierarchy, so I tried to work around the ever-growing pain by tuning the filesystem, the host's kernel (a buffed dentry cache size goes a long way at making this kind of abuse more tolerable, for instance), and the NFS server and clients involved to mitigate emerging problems as much as possible. In the end, a readdirplus()/getdents()-chain invoked over NFS only took a few seconds to finish. It's pretty amazing what Linux can cope with, if you make it have to :)
- desro 4y agoI'd be very interested in a more detailed write-up of what you did here -- sounds pretty impressive.
- tryauuum 4y agoI think we had something similar at my old place, millions of files in one dir, shared over samba. It worked ok because each file was always retrieved by the full path (filenames were stored in a database) so no readdir was required
- sixothree 4y agoAt the same time, nobody would bat an eye at 1 million rows in a database table. People don't really start to notice until you hit 100 million.
- bluedino 4y agohttps://www.sqlite.org/fasterthanfs.html https://www.sqlite.org/fasterthanfs.html
- morelisp 4y ago> nobody would bat an eye at 1 million rows in a database table They often would if they tried to access them like they usually do a filesystem, but for some reason people know a full table scan and N+1 queries are bad, but don't bat an eye at readdir and stat each entry in a directory (or worse, scandir+sort+stat).
- gregmac 4y agoI've dealt with a different situation before and handled it with sharding. It's pretty easy to retrofit in code, and it's not hard to do a one-time move of existing files to the new layout. Simply put, you make an algorithm like: let hash = md5(filename) let folderPath = path.join(root, hash.substr(0,1), hash.substr(0,2), hash.substr(0,3), filename) The original name can be used to derive the sharded path, so you don't need a lookup database or anything crazy like that. You end up storing files like: .../7/7f/7fa/somefile.png This scales to whatever depth you want: with 3 levels and 1.2 million files, you can expect about 1200000/16^3 = 293 files per directory, which is trivial. There's lots of other strategies, too, depending on your needs: # filename are already GUIDs or hashes: .../1/1f/1fd/1fd2dd27-d307-47b2-b37d-11903fd0f03d.png # date-based to make cleaning up old files easy: .../2023/01/19/app.log # hash by existing numeric identifier (like customerid) using modulus: .../customers/4/94/52194/customerfile.dat
- pc86 4y agoThis seems like a better solution that just tweaking Linux to make it take only "a few seconds" to pull up an image. I'm curious what downsides/traps there are in this approach vs. sticking with the single flat directory.
- bluedino 4y agoI have used that method many times with success. One side-effect is you can't go into a folder and type something like: `cp .doc` or `cp screenshots xxxxx`
- thesneakin 4y ago./**/*.doc Or `find -exec`
- bluedino 4y agoExcept that will be very slow when you have millions of files
- nine_k 4y agoIt's amazing how 1.2M entries in a directory is a pathologically slow case for a file system, but a trivial number of table rows for a database. Of course, a B-tree is somehow larger than whatever was used to store inode data in the (ext2?) filesystem 25 years ago.