3 ms·
It's not uncommon with distributed storage to err one way or another, because you have to store large amounts of data and metadata. Designing a system that does
by TallGuyShort 11y ago
It's not uncommon with distributed storage to err one way or another, because you have to store large amounts of data and metadata. Designing a system that does both extremely well is non-trivial. You can optimize to store a few very large files or many very small files, but doing both while scaling reliably is harder than you might think.
- DannoHung 11y agoNaively it seems weird since metadata is just data, right? So if you can store a large amount of data in a file, you should be able to store the metadata But I don't know shit about filesystem design, soooooo...
- TallGuyShort 11y agoWell the metadata I'm referring to is the metadata required to locate a file: mappings from directory structure, filename, possibly permissions / ownership to the blocks and replicas distributed across a cluster. If you get to the point where you need start locating metadata because it might also be arbtirarily distributed you now need meta-metadata. And I'm sure you can see how complex that's going to get :)