2 ms·
Do they do need to even do that? Couldn't that be engineered into the free space on other physical drives and change as needed? (I do a version of that on a ver
by gist 11y ago
Do they do need to even do that? Couldn't that be engineered into the free space on other physical drives and change as needed? (I do a version of that on a very small scale and noting I am not expert in this area it was just a common sense approach.) Meaning it wouldn't have to be "doubled" or anything close to that.
- nostrademons 11y agoThey do, but not quite in such a simplistic way. Basically, there's one service that manages the individual drives, and then another service (Colossus) that sits on top of it and provides the abstraction of an infinite, highly-available, perfectly robust filesystem. So services like YouTube just specify a path, data, an owner, and a replication policy and then hand it off to Colossus to find free space on one of the zillions of disks. There's a side mechanism for quota & budgeting. It's unclear from the article whether YouTube's 1P/day is pre-replication or post-replication. I'd just assumed post-replication; it doesn't really change the conclusion in a material way. (Rather than the answer being "1 person", it becomes "2-3 people".) And yes, you do need 2x (or more commonly, 3x) the disk space. You can use error-correcting codes to correct single-bit errors (Colossus uses Reed-Solomon), but that won't help you if there's a fiber cut and a whole datacenter goes offline.
- gist 11y agoHah! Thanks for making me feel somewhat smart. Reminds me of when I was a kid and I said to my mom about cancer "why don't they just cut it out?".