5 ms·

> Having thousands of files in a single directory is usually not very file system-friendly, and definitely not recommended. I wonder, is this specific to the s

> Having thousands of files in a single directory is usually not very file system-friendly, and definitely not recommended. I wonder, is this specific to the situation when I use "ls", or is it in general? I recently worked on a project where I need to store many small files on ext4, although these files are not read/written by human. I came across suggestions to group files into subdirectories and not put them all in one directory. Is there evidence that it's actually worth it on a modern filesystem? EDIT: by "many" I mean 10+ million files.

tyingq 8y ago
At the very least, it would be more polite to anyone that might need to support, troubleshoot, remove the files with shell tools after removing your software, etc. I suspect it's also still a performance issue with many popular filesystems. See https://lwn.net/Articles/606995/ https://lwn.net/Articles/606995/ for more info.
mcbits 8y ago
For future reference, next time you need to store millions of related small files, it might be worth checking if SQLite is an option instead of using the filesystem: https://www.sqlite.org/fasterthanfs.html https://www.sqlite.org/fasterthanfs.html Performance aside, I wish NodeJS had a way to pack all its little turds into one SQLite file, out of sight. (Admittedly I hadn't thought to look for such a thing until now... Preliminary results are negatory.)
- civility 8y ago
  You should benchmark SQLite before recommending it like this (for future reference). I think it's a very elegant library, and I admire the development philosophy that went into it. However, every time I've tried to use it for something that needs to be fast it ended up being a painful mistake.
  - striking 8y ago
    Could you show us your methodology for your benchmark? We don't know under which circumstances you experienced degraded performance. I've seen a lot of people just missing an index or something, and their DB then ran fine, so that's why I ask.
    - civility 8y ago
      I don't work for that company any more, and I couldn't have legally showed the code even if I did. It's not an index thing though. Try inserting 10 million rows into a simple table. No foreign constraints or anything. Some of our "documents" had 300 million rows. Handle the primary key however you want (let the DB do it, or generate your own). Use whatever SQLite API calls you want (precompiled statements, whatever). In a case like this, adding other indexes can only slow it down. There are a few options to disable integrity and other safety constraints. They help speed things up a little, but it's all painfully slow in comparison to a writing simple CSV file. The same is true on reading. You can parse an entire CSV file more quickly than "SELECT *" from the table. I've tried several time to use SQLite as a document file format. It performs worse than CSV and JSON. It performs much worse than just an array of binary records. The devil is in the details, and assuming each file is not a single row/record, I wouldn't be surprised if 10+ million files in a structured directory tree performs better too.
      - mcbits 8y ago
        Well, I certainly agree about benchmarking versus other options, especially if I/O proves to be a bottleneck. The other option above was not a single CSV or JSON-structured file, but millions of small files stored on the filesystem (with its requisite indexes, permissions, access logs, etc). And the comparison is not with the file contents being splayed out into a relational structure in SQLite, but just one flat key-blob table for storage and retrieval. It's possible that a multi-gigabyte CSV file would be faster still, depending on actual access patterns and how much effort you want to devote to this bespoke CSV database.
        civility 8y ago
loeg 8y ago
What is a modern filesystem and what are its performance characteristics? If you don't want to evaluate these things ahead of time, erring on the side of using subdirectories is safer — it's unlikely to be much worse even on filesystems with fast huge directory traversal and lookup, and it's a lot better on conventional filesystems.
xorcist 8y ago
Modern file systems with indexed metadata has improved that situation considerably. There is always a limit however. If your application stores a million files today, maybe it will grow to a billion? File systems aren't the end of the story however. There's always the need to do backups or copy files for some other reason. Some tool somewhere will find the need to readdir and will not cope well with it. So it is still be a good idea to partition files in directories when you have many of them, and millions of files are still many.
- CaliforniaKarl 8y ago
  Here's another little "story from the trenches". Many of our users use MATLAB. Yes, they should probably use Python, or Octave, but they're using MATLAB. We use SLURM as our job scheduler. It supports "job arrays", which allows a user to submit a large number of jobs (up to 1,000) with a single "batch script", which is a shell script that gets run on a compute node, in a cgroup with the requested amount of memory and CPU. A job array simply runs this on many (for us, up to 1,000) cgroups, spread out across nodes; the batch script is able to check environment variables to see its "array index", which the user uses to split up work. Once people found this, they took advantage of our site-wide MATLAB license, launching hundreds or thousands of MATLAB instances at around the same time. MATLAB likes to create a 'prefdir' inside `~/.matlab/RXXXXy` ("R2017b", for example). Thousands of MATLABs, all making directories, inside the same common directory, on network-mounted storage. Our Isilon cluster was floored. We found the environment variable which controlled where the prefdir is placed. We changed it to a path in $SCRATCH (the user's scratch space), which is on Lustre. Our Luster environment held up for a while, but was similarly floored. In all of these cases, the issue appears to be many clients trying to take a short lock on the `~/.matlab/RXXXXy` directory, to make their subdirectory. "It worked fine for one, or for a hundred, so it must be OK?"
  - gnufx 8y ago
    I'm surprised that the number of concurrent array job elements isn't more limited, partly for that reason. I'd also expect the resource manager to make a per-job (element) TMPDIR on local disk and to get jobs to use that. Where array elements are accessing the same data, it's typically good to unroll the array loop and stage data to local storage for multiple operations on them. I wrote support for that with BLAST jobs in mind, whose mmap was particularly problematic.
jlg23 8y ago
That depends on how you access them. If you only ever iterate over all files readdir(3) in a tight loop is as fast as it can get; but if you need lots of access to specific files, you will waste a lot of time reading directory entries just to discard them... If there is even a remote chance you will use standard Unix tools on your files as well, I'd go for the subdirectories.

tyingq 8y ago

At the very least, it would be more polite to anyone that might need to support, troubleshoot, remove the files with shell tools after removing your software, etc. I suspect it's also still a performance issue with many popular filesystems. See https://lwn.net/Articles/606995/ https://lwn.net/Articles/606995/ for more info.

mcbits 8y ago

For future reference, next time you need to store millions of related small files, it might be worth checking if SQLite is an option instead of using the filesystem: https://www.sqlite.org/fasterthanfs.html https://www.sqlite.org/fasterthanfs.html Performance aside, I wish NodeJS had a way to pack all its little turds into one SQLite file, out of sight. (Admittedly I hadn't thought to look for such a thing until now... Preliminary results are negatory.)

civility 8y ago
You should benchmark SQLite before recommending it like this (for future reference). I think it's a very elegant library, and I admire the development philosophy that went into it. However, every time I've tried to use it for something that needs to be fast it ended up being a painful mistake.
- striking 8y ago
  Could you show us your methodology for your benchmark? We don't know under which circumstances you experienced degraded performance. I've seen a lot of people just missing an index or something, and their DB then ran fine, so that's why I ask.
  - civility 8y ago
    I don't work for that company any more, and I couldn't have legally showed the code even if I did. It's not an index thing though. Try inserting 10 million rows into a simple table. No foreign constraints or anything. Some of our "documents" had 300 million rows. Handle the primary key however you want (let the DB do it, or generate your own). Use whatever SQLite API calls you want (precompiled statements, whatever). In a case like this, adding other indexes can only slow it down. There are a few options to disable integrity and other safety constraints. They help speed things up a little, but it's all painfully slow in comparison to a writing simple CSV file. The same is true on reading. You can parse an entire CSV file more quickly than "SELECT *" from the table. I've tried several time to use SQLite as a document file format. It performs worse than CSV and JSON. It performs much worse than just an array of binary records. The devil is in the details, and assuming each file is not a single row/record, I wouldn't be surprised if 10+ million files in a structured directory tree performs better too.
    - mcbits 8y ago
      Well, I certainly agree about benchmarking versus other options, especially if I/O proves to be a bottleneck. The other option above was not a single CSV or JSON-structured file, but millions of small files stored on the filesystem (with its requisite indexes, permissions, access logs, etc). And the comparison is not with the file contents being splayed out into a relational structure in SQLite, but just one flat key-blob table for storage and retrieval. It's possible that a multi-gigabyte CSV file would be faster still, depending on actual access patterns and how much effort you want to devote to this bespoke CSV database.
      - civility 8y ago

loeg 8y ago

What is a modern filesystem and what are its performance characteristics? If you don't want to evaluate these things ahead of time, erring on the side of using subdirectories is safer — it's unlikely to be much worse even on filesystems with fast huge directory traversal and lookup, and it's a lot better on conventional filesystems.

xorcist 8y ago

Modern file systems with indexed metadata has improved that situation considerably. There is always a limit however. If your application stores a million files today, maybe it will grow to a billion? File systems aren't the end of the story however. There's always the need to do backups or copy files for some other reason. Some tool somewhere will find the need to readdir and will not cope well with it. So it is still be a good idea to partition files in directories when you have many of them, and millions of files are still many.

CaliforniaKarl 8y ago
Here's another little "story from the trenches". Many of our users use MATLAB. Yes, they should probably use Python, or Octave, but they're using MATLAB. We use SLURM as our job scheduler. It supports "job arrays", which allows a user to submit a large number of jobs (up to 1,000) with a single "batch script", which is a shell script that gets run on a compute node, in a cgroup with the requested amount of memory and CPU. A job array simply runs this on many (for us, up to 1,000) cgroups, spread out across nodes; the batch script is able to check environment variables to see its "array index", which the user uses to split up work. Once people found this, they took advantage of our site-wide MATLAB license, launching hundreds or thousands of MATLAB instances at around the same time. MATLAB likes to create a 'prefdir' inside `~/.matlab/RXXXXy` ("R2017b", for example). Thousands of MATLABs, all making directories, inside the same common directory, on network-mounted storage. Our Isilon cluster was floored. We found the environment variable which controlled where the prefdir is placed. We changed it to a path in $SCRATCH (the user's scratch space), which is on Lustre. Our Luster environment held up for a while, but was similarly floored. In all of these cases, the issue appears to be many clients trying to take a short lock on the `~/.matlab/RXXXXy` directory, to make their subdirectory. "It worked fine for one, or for a hundred, so it must be OK?"
- gnufx 8y ago
  I'm surprised that the number of concurrent array job elements isn't more limited, partly for that reason. I'd also expect the resource manager to make a per-job (element) TMPDIR on local disk and to get jobs to use that. Where array elements are accessing the same data, it's typically good to unroll the array loop and stage data to local storage for multiple operations on them. I wrote support for that with BLAST jobs in mind, whose mmap was particularly problematic.

jlg23 8y ago

That depends on how you access them. If you only ever iterate over all files readdir(3) in a tight loop is as fast as it can get; but if you need lots of access to specific files, you will waste a lot of time reading directory entries just to discard them... If there is even a remote chance you will use standard Unix tools on your files as well, I'd go for the subdirectories.