2 ms·
If technical goes to scaling and effects related to scaling, maybe they just decided to let the people with >5m inodes incur the technical penalty, which is alm
by ggm 4y ago
If technical goes to scaling and effects related to scaling, maybe they just decided to let the people with >5m inodes incur the technical penalty, which is almost always speed?
it was always a cost issue, in some sense. If they designed the entire FS model behind drive to have inode limits per entity, then there's a small technology issue bumping that number maybe. Quota systems can be quite painful, in days gone past the Robert Elz Quota model on UFS demanded reboots for some changes, which at scale in a google sized entity is a rolling surface of state change.
- vlovich123 4y agoTheir explanation of the 5M limit is because of inode limitations when they materialize the content via the companion client?
- ggm 4y ago5m is an odd number. 24 bit numberfield would get you 16m uniques. I can't see how they needed to steal bits to get to 23 or 22. So there must be some other factor at play here apart from 'we used 3 bytes in the gRPC model' Perhaps the Windows exposed API assumed somewhere around 5m objects, and they care about the compatibility greatly.
- perbu 4y agoI don't think the number is based on any technical limitation. It could be a number based on usage. Say 99.9% of the users use less than 5M files. So my pruning 0.1% of the user base they are getting rid of half the number of files, saving X millions of dollars in cost. Nobody wants to cater to the difficult users that are doing stuff like this.
- hnfong 4y agoThat's interesting, because if your argument is valid, they wouldn't have to roll back the change.
- ggm 4y agoif filescount() > 5m then.. had to be removed. There is no quick release in a google scale monrepo and QA process.
- Tommstein 4y agoAfter that pruning, you now have a new 0.1% of users using more files than the other 99.9%. Might as well prune that 0.1% of file hogs too. Rinse and repeat until you have no users left.
- Cthulhu_ 4y agoMaybe they just keep a safe margin so they can add their own metadata inodes or something. Or it's arbitrary.
- vlovich123 4y agoSure. But all that would mean is that you can only have 5M files materialized locally. I don’t see why drive itself would be limited at 5M. Very strange for this to have any real technical limit.