3 ms·

It does seem certainly seem atypical, and I'd be curious if anyone has ever encountered something like this in the real world. It seems unlikely because 1. It

It does seem certainly seem atypical, and I'd be curious if anyone has ever encountered something like this in the real world. It seems unlikely because 1. It takes so long to create that many files in the first place. In this case 26 hours of doing nothing but file creation. 2. If the files contained any data at all, you would need a lot of disk space. If each file was 64k, you'd need 640TB of disk. It seems more likely you'd run out of disk space before you reached a billion files. 3. Someone developing such a system where that many files could be created would probably realize that it's a poor design and figure out a way to mitigate it before it ever happened. Nevertheless, it's fun to think about.

lanthade 3y ago
Back in the late 2000’s I was a windows clustering SME for a major tech company on account with a major financial company. One project I was tasked with was migrating a system which had a huge number of small files. I don’t recall the number. IIRC it took a couple weeks to do the initial file copy. Then the task was to sync the changes during the change window which was 8 hours. Every time I tried to sync the changes it took well more than 8 hours. I did find a tool that could do the job but of course no one wanted to pay for it. Fortunately for me I left that position before the migration and someone else had to figure it out. So yeah, I’ve encountered something similar in the real world. It was the result of poor software design choices but by the time I got there that path was chosen and I had to deal with the results.
vlovich123 3y ago
10k file creations per second does seem kind of slow no? A typical SSD today can do many millions of operations per second…
yjftsjthsd-h 3y ago
> and I'd be curious if anyone has ever encountered something like this in the real world. Not that many, no, but I can tell you from professional experience that real life Linux systems get really weird when you have merely tens of thousands of files in a directory, and it would be kind of nice if that were fixed.
- yencabulator 3y ago
  That used to be true, but ext4 htrees fixed most of the Linux issues in 2002. These days, most of the stupid resides firmly in the userspace consumers, making assumptions like "readdir a whole directory into an in-memory list". https://en.wikipedia.org/wiki/HTree https://en.wikipedia.org/wiki/HTree
  - yjftsjthsd-h 3y ago
    Of all the times for me to not write out GNU/Linux, you're telling me this is the one time the GNU part actually is the important part?:) (Though in fairness, ex. ls did become... kind of functional when run as `ls -f`)

lanthade 3y ago

Back in the late 2000’s I was a windows clustering SME for a major tech company on account with a major financial company. One project I was tasked with was migrating a system which had a huge number of small files. I don’t recall the number. IIRC it took a couple weeks to do the initial file copy. Then the task was to sync the changes during the change window which was 8 hours. Every time I tried to sync the changes it took well more than 8 hours. I did find a tool that could do the job but of course no one wanted to pay for it. Fortunately for me I left that position before the migration and someone else had to figure it out. So yeah, I’ve encountered something similar in the real world. It was the result of poor software design choices but by the time I got there that path was chosen and I had to deal with the results.

vlovich123 3y ago

10k file creations per second does seem kind of slow no? A typical SSD today can do many millions of operations per second…

yjftsjthsd-h 3y ago

> and I'd be curious if anyone has ever encountered something like this in the real world. Not that many, no, but I can tell you from professional experience that real life Linux systems get really weird when you have merely tens of thousands of files in a directory, and it would be kind of nice if that were fixed.

yencabulator 3y ago
That used to be true, but ext4 htrees fixed most of the Linux issues in 2002. These days, most of the stupid resides firmly in the userspace consumers, making assumptions like "readdir a whole directory into an in-memory list". https://en.wikipedia.org/wiki/HTree https://en.wikipedia.org/wiki/HTree
- yjftsjthsd-h 3y ago
  Of all the times for me to not write out GNU/Linux, you're telling me this is the one time the GNU part actually is the important part?:) (Though in fairness, ex. ls did become... kind of functional when run as `ls -f`)