4 ms·
I see it speculated, downthread, that this is a response to modern web-dev and node (?) creating millions of files, etc. I can’t comment on that but I do know
by rsync 4y ago
I see it speculated, downthread, that this is a response to modern web-dev and node (?) creating millions of files, etc.
I can’t comment on that but I do know that modern, encrypted, archive tools such as duplicity and borg and restic “chunk” your source files into thousands (potentially millions) of little files.
We see tens of billions of files on our zpools and have “normal” customers breaking 1B … and the cause is typically duplicity or borg.
- jeffbee 4y agoAt a previous company the biggest cause of files in a particular context was that "oh my zsh" was installed for each of thousands of engineers by default. It often seems as though "modern framework" just means it is an abusive piece of junk.
- Dylan16807 4y agoBillions seems weird for normal customers. Duplicity appears to put its chunks into 25MB files, I know for sure that restic puts them into 4MB or more recently 16MB files, and Borg looks like it puts them into 500MB files.
- rsync 4y agoNot average, but normal. What I mean is, they just have a plain old rsync.net account and pay no premium nor incur any penalty for using all of those inodes. You are correct: the average customer does not use billions of inodes.
- Dylan16807 4y agoWell I interpreted "normal" to mean "does not have many petabytes of data", and if those tools are reaching a billion files with smaller amounts of data then I'm very confused about how it's happening.