6 ms·
They do, but not quite in such a simplistic way. Basically, there's one service that manages the individual drives, and then another service (Colossus) that si
by nostrademons 11y ago
They do, but not quite in such a simplistic way. Basically, there's one service that manages the individual drives, and then another service (Colossus) that sits on top of it and provides the abstraction of an infinite, highly-available, perfectly robust filesystem. So services like YouTube just specify a path, data, an owner, and a replication policy and then hand it off to Colossus to find free space on one of the zillions of disks. There's a side mechanism for quota & budgeting.
It's unclear from the article whether YouTube's 1P/day is pre-replication or post-replication. I'd just assumed post-replication; it doesn't really change the conclusion in a material way. (Rather than the answer being "1 person", it becomes "2-3 people".)
And yes, you do need 2x (or more commonly, 3x) the disk space. You can use error-correcting codes to correct single-bit errors (Colossus uses Reed-Solomon), but that won't help you if there's a fiber cut and a whole datacenter goes offline.
- gist 11y agoHah! Thanks for making me feel somewhat smart. Reminds me of when I was a kid and I said to my mom about cancer "why don't they just cut it out?".