3 ms·

Modern file systems with indexed metadata has improved that situation considerably. There is always a limit however. If your application stores a million files

Modern file systems with indexed metadata has improved that situation considerably. There is always a limit however. If your application stores a million files today, maybe it will grow to a billion? File systems aren't the end of the story however. There's always the need to do backups or copy files for some other reason. Some tool somewhere will find the need to readdir and will not cope well with it. So it is still be a good idea to partition files in directories when you have many of them, and millions of files are still many.

CaliforniaKarl 8y ago
Here's another little "story from the trenches". Many of our users use MATLAB. Yes, they should probably use Python, or Octave, but they're using MATLAB. We use SLURM as our job scheduler. It supports "job arrays", which allows a user to submit a large number of jobs (up to 1,000) with a single "batch script", which is a shell script that gets run on a compute node, in a cgroup with the requested amount of memory and CPU. A job array simply runs this on many (for us, up to 1,000) cgroups, spread out across nodes; the batch script is able to check environment variables to see its "array index", which the user uses to split up work. Once people found this, they took advantage of our site-wide MATLAB license, launching hundreds or thousands of MATLAB instances at around the same time. MATLAB likes to create a 'prefdir' inside `~/.matlab/RXXXXy` ("R2017b", for example). Thousands of MATLABs, all making directories, inside the same common directory, on network-mounted storage. Our Isilon cluster was floored. We found the environment variable which controlled where the prefdir is placed. We changed it to a path in $SCRATCH (the user's scratch space), which is on Lustre. Our Luster environment held up for a while, but was similarly floored. In all of these cases, the issue appears to be many clients trying to take a short lock on the `~/.matlab/RXXXXy` directory, to make their subdirectory. "It worked fine for one, or for a hundred, so it must be OK?"
- gnufx 8y ago
  I'm surprised that the number of concurrent array job elements isn't more limited, partly for that reason. I'd also expect the resource manager to make a per-job (element) TMPDIR on local disk and to get jobs to use that. Where array elements are accessing the same data, it's typically good to unroll the array loop and stage data to local storage for multiple operations on them. I wrote support for that with BLAST jobs in mind, whose mmap was particularly problematic.

CaliforniaKarl 8y ago

Here's another little "story from the trenches". Many of our users use MATLAB. Yes, they should probably use Python, or Octave, but they're using MATLAB. We use SLURM as our job scheduler. It supports "job arrays", which allows a user to submit a large number of jobs (up to 1,000) with a single "batch script", which is a shell script that gets run on a compute node, in a cgroup with the requested amount of memory and CPU. A job array simply runs this on many (for us, up to 1,000) cgroups, spread out across nodes; the batch script is able to check environment variables to see its "array index", which the user uses to split up work. Once people found this, they took advantage of our site-wide MATLAB license, launching hundreds or thousands of MATLAB instances at around the same time. MATLAB likes to create a 'prefdir' inside `~/.matlab/RXXXXy` ("R2017b", for example). Thousands of MATLABs, all making directories, inside the same common directory, on network-mounted storage. Our Isilon cluster was floored. We found the environment variable which controlled where the prefdir is placed. We changed it to a path in $SCRATCH (the user's scratch space), which is on Lustre. Our Luster environment held up for a while, but was similarly floored. In all of these cases, the issue appears to be many clients trying to take a short lock on the `~/.matlab/RXXXXy` directory, to make their subdirectory. "It worked fine for one, or for a hundred, so it must be OK?"

gnufx 8y ago
I'm surprised that the number of concurrent array job elements isn't more limited, partly for that reason. I'd also expect the resource manager to make a per-job (element) TMPDIR on local disk and to get jobs to use that. Where array elements are accessing the same data, it's typically good to unroll the array loop and stage data to local storage for multiple operations on them. I wrote support for that with BLAST jobs in mind, whose mmap was particularly problematic.