4 ms·
I think you forget the redundancy. You probably need to double it to 256 drives per day.
by imaginenore 11y ago
I think you forget the redundancy. You probably need to double it to 256 drives per day.
- gist 11y agoDo they do need to even do that? Couldn't that be engineered into the free space on other physical drives and change as needed? (I do a version of that on a very small scale and noting I am not expert in this area it was just a common sense approach.) Meaning it wouldn't have to be "doubled" or anything close to that.
- nostrademons 11y agoThey do, but not quite in such a simplistic way. Basically, there's one service that manages the individual drives, and then another service (Colossus) that sits on top of it and provides the abstraction of an infinite, highly-available, perfectly robust filesystem. So services like YouTube just specify a path, data, an owner, and a replication policy and then hand it off to Colossus to find free space on one of the zillions of disks. There's a side mechanism for quota & budgeting. It's unclear from the article whether YouTube's 1P/day is pre-replication or post-replication. I'd just assumed post-replication; it doesn't really change the conclusion in a material way. (Rather than the answer being "1 person", it becomes "2-3 people".) And yes, you do need 2x (or more commonly, 3x) the disk space. You can use error-correcting codes to correct single-bit errors (Colossus uses Reed-Solomon), but that won't help you if there's a fiber cut and a whole datacenter goes offline.
- gist 11y agoHah! Thanks for making me feel somewhat smart. Reminds me of when I was a kid and I said to my mom about cancer "why don't they just cut it out?".