3 ms·

Which file system is best optimized to deal with a huge number of tiny files?

by eviks 3y ago

liveoneggs 3y ago
None of them are very good in practice. Databases exist for just this sort of job.
- webstrand 3y ago
  But you're not supposed to store "large" blobs in databases, either?
  - pizzafeelsright 3y ago
    Define "large" because it keeps getting bigger. Systems I've built generally use a pointer within a database to a secondary location. I've also seen hundreds of megs stored in binary json blobs.
- snuxoll 3y ago
  For what it's worth, I've built an entire data ingest pipeline with PostgreSQL that stores tens of millions of PDFs in a giant database table and...it works fine. Now we are working on scooping up a bunch of small (1-2KB) files we want to index with metadata and chucking them into a SQLite file, because as fast as `grep` is searching through the text the constant open() and close() calls are much more impactful.
wongarsu 3y ago
You'd have to quantify what you mean by "best". Lowest storage overhead? Being able to deal with the most files without needing tuning (e.g. not running out of inodes)? Access Latency? For metadata or do you want the actual file? Iterating the directory or accessing random file names? And if you care about speed, are the files in a single folder or in a reasonable directory structure? Do you need the files to be able to change or is readonly enough? For readonly there are a couple good options, though arguably they are then archives. Many use sqlite because it's widely available and deals with this well. But most average file systems are pretty bad at at least one of those factors. It's probably easier to identify which file systems are bad for which reasons. For example on NTFS you can't store a file in less than 1kb, and large directories can be an issue because a directory entry contains a sequential list of files. In contrast ext3 introduced a hash table to quickly look up file metadata in huge directories. On the other hand, if your files are e.g. about 800B and spread into directories of reasonable file count NTFS might suddenly outperform ext4 on some metrics, because NTFS can store the first ~900 bytes of the file together with the metadata, ext4 only the first 60. Or if your huge number of tiny files contains lots of identical files a zfs with live deduplication might become interesting.
- eviks 3y ago
  I meant regular "consumer" file operations - opening a folder with a few mil files and getting basic data like names/sizes, copying a few million files from one folder to another, renaming, etc. So definitely not read-only, not identical, not all tiny 0b or even 800b files (maybe <1mb is still tiny?) , and storage overead is a much lower priority vs latency of access and speed of operations
  - crabbone 3y ago
    Why would a consumer have a billion files in a directory? Your question reminds me of a famous (some fifteen-twenty years ago) question from MSDN forums. A poor soul complained that when they put 10K buttons on a form in the designer view in MSVS it comes to a grinding halt. And that's impossible to use the studio anymore. To which a bunch of respondents quickly replied: "but why tho? why do you need so many buttons on the form?" -- From the discussion that followed it became apparent that the poster wanted to reproduce the minesweeper game with a 100x100 board. What I'm trying to say is: if you have a problem where you think you need a billion of files all in one directory, it's easier to reformulate the problem than to try to find a solution for it.
- aseipp 3y ago
  > In contrast ext3 introduced a hash table to quickly look up file metadata in huge directories. Ah, but it has its own consequences. ext4 doesn't dynamically reclaim inodes or garbage collect these directory hash tables, so if you create a directory and cycle a bunch of files through it, like millions and billions of files many times over, you can get into a case where you have sizable portions of your drive as free space, but you get ENOSPC errors because adding files to the directory tries to insert them into the hash table, which fails, because the hash table takes up so much space it causes the drive to be actually full. Oh, don't try actually doing directory operations without the index, either. Something like `du -sh` without that index will take upwards of 10+ minutes on a fast virtio SSD to account for like 10 gigabytes of files. I recently had a 1TB ext4 drive that got into this state where it had 150GB free but the directory hash indexes were fucking massive, because billions of files had gone through the filesystem and it was basically hosed because even turning off the index and then turning it back on does not drop the hash table index, it only literally turns off the index use in the read path and leaves the index as is. I couldn't find a way to drop and clear the hash indicies. Yeah. Apparently you can get into a similar state if you exhaust the inode count because, again, there is no dynamic inode reclamation. In this case, I had the `gecko-dev` Git repository, which is absolutely massive, and I was extracting many copies of it onto my drive as part of some testing automation. So that added up fast. It was on a separate drive from my main /home mount, at least. The lesson I have learned from this is to mostly just use XFS instead of ext4, I guess. Which I was already doing on all my servers, so a bit funny I learned that lesson only on my own personal machine.
jeffbee 3y ago
I know which one I wouldn't pick for five billion files, that's for sure. EXT4 prepays for some of the work at mkfs time, but it also has certain limits. XFS can create as many inodes as you want, if you have the space for it.
linsomniac 3y ago
A decade ago I was dealing with map tiles for the entire globe, stored each in it's own file, and reiserfs was dramatically better at dealing with all the small files associated with this: there are a lot of solid blue and solid green tiles in the globe. Reiser's idea of what a small file is was 1-2 orders of magnitude smaller than what ext3's idea of a small file was. More recently our map tiles are limited to a single US state, and we are using XFS, but because of the much smaller data set size I haven't really done any optimization beyond hard linking duplicate files. I inherited that choice, so I don't know what went into that.
- Minor49er 3y ago
  Why not just identify the solid tiles differently than ones that have a mix? That should reduce the number of tiles to store. Or is there some drawback to that approach?
ComputerGuru 3y ago
This was a design goal for reiserfs, fwiw.

Which file system is best optimized to deal with a huge number of tiny files?

by eviks 3y ago

Which file system is best optimized to deal with a huge number of tiny files?

liveoneggs 3y ago

None of them are very good in practice. Databases exist for just this sort of job.

webstrand 3y ago
But you're not supposed to store "large" blobs in databases, either?
- pizzafeelsright 3y ago
  Define "large" because it keeps getting bigger. Systems I've built generally use a pointer within a database to a secondary location. I've also seen hundreds of megs stored in binary json blobs.
snuxoll 3y ago
For what it's worth, I've built an entire data ingest pipeline with PostgreSQL that stores tens of millions of PDFs in a giant database table and...it works fine. Now we are working on scooping up a bunch of small (1-2KB) files we want to index with metadata and chucking them into a SQLite file, because as fast as `grep` is searching through the text the constant open() and close() calls are much more impactful.

wongarsu 3y ago

You'd have to quantify what you mean by "best". Lowest storage overhead? Being able to deal with the most files without needing tuning (e.g. not running out of inodes)? Access Latency? For metadata or do you want the actual file? Iterating the directory or accessing random file names? And if you care about speed, are the files in a single folder or in a reasonable directory structure? Do you need the files to be able to change or is readonly enough? For readonly there are a couple good options, though arguably they are then archives. Many use sqlite because it's widely available and deals with this well. But most average file systems are pretty bad at at least one of those factors. It's probably easier to identify which file systems are bad for which reasons. For example on NTFS you can't store a file in less than 1kb, and large directories can be an issue because a directory entry contains a sequential list of files. In contrast ext3 introduced a hash table to quickly look up file metadata in huge directories. On the other hand, if your files are e.g. about 800B and spread into directories of reasonable file count NTFS might suddenly outperform ext4 on some metrics, because NTFS can store the first ~900 bytes of the file together with the metadata, ext4 only the first 60. Or if your huge number of tiny files contains lots of identical files a zfs with live deduplication might become interesting.

eviks 3y ago
I meant regular "consumer" file operations - opening a folder with a few mil files and getting basic data like names/sizes, copying a few million files from one folder to another, renaming, etc. So definitely not read-only, not identical, not all tiny 0b or even 800b files (maybe <1mb is still tiny?) , and storage overead is a much lower priority vs latency of access and speed of operations
- crabbone 3y ago
  Why would a consumer have a billion files in a directory? Your question reminds me of a famous (some fifteen-twenty years ago) question from MSDN forums. A poor soul complained that when they put 10K buttons on a form in the designer view in MSVS it comes to a grinding halt. And that's impossible to use the studio anymore. To which a bunch of respondents quickly replied: "but why tho? why do you need so many buttons on the form?" -- From the discussion that followed it became apparent that the poster wanted to reproduce the minesweeper game with a 100x100 board. What I'm trying to say is: if you have a problem where you think you need a billion of files all in one directory, it's easier to reformulate the problem than to try to find a solution for it.
aseipp 3y ago
> In contrast ext3 introduced a hash table to quickly look up file metadata in huge directories. Ah, but it has its own consequences. ext4 doesn't dynamically reclaim inodes or garbage collect these directory hash tables, so if you create a directory and cycle a bunch of files through it, like millions and billions of files many times over, you can get into a case where you have sizable portions of your drive as free space, but you get ENOSPC errors because adding files to the directory tries to insert them into the hash table, which fails, because the hash table takes up so much space it causes the drive to be actually full. Oh, don't try actually doing directory operations without the index, either. Something like `du -sh` without that index will take upwards of 10+ minutes on a fast virtio SSD to account for like 10 gigabytes of files. I recently had a 1TB ext4 drive that got into this state where it had 150GB free but the directory hash indexes were fucking massive, because billions of files had gone through the filesystem and it was basically hosed because even turning off the index and then turning it back on does not drop the hash table index, it only literally turns off the index use in the read path and leaves the index as is. I couldn't find a way to drop and clear the hash indicies. Yeah. Apparently you can get into a similar state if you exhaust the inode count because, again, there is no dynamic inode reclamation. In this case, I had the `gecko-dev` Git repository, which is absolutely massive, and I was extracting many copies of it onto my drive as part of some testing automation. So that added up fast. It was on a separate drive from my main /home mount, at least. The lesson I have learned from this is to mostly just use XFS instead of ext4, I guess. Which I was already doing on all my servers, so a bit funny I learned that lesson only on my own personal machine.

jeffbee 3y ago

I know which one I wouldn't pick for five billion files, that's for sure. EXT4 prepays for some of the work at mkfs time, but it also has certain limits. XFS can create as many inodes as you want, if you have the space for it.

linsomniac 3y ago

A decade ago I was dealing with map tiles for the entire globe, stored each in it's own file, and reiserfs was dramatically better at dealing with all the small files associated with this: there are a lot of solid blue and solid green tiles in the globe. Reiser's idea of what a small file is was 1-2 orders of magnitude smaller than what ext3's idea of a small file was. More recently our map tiles are limited to a single US state, and we are using XFS, but because of the much smaller data set size I haven't really done any optimization beyond hard linking duplicate files. I inherited that choice, so I don't know what went into that.

Minor49er 3y ago
Why not just identify the solid tiles differently than ones that have a mix? That should reduce the number of tiles to store. Or is there some drawback to that approach?

ComputerGuru 3y ago

This was a design goal for reiserfs, fwiw.