3 ms·

This is an impressive product, and I apologize, but I'm gonna go on a bit of a rant about the PR language. I hate the phrase "Our system has over twelve 9s of

This is an impressive product, and I apologize, but I'm gonna go on a bit of a rant about the PR language. I hate the phrase "Our system has over twelve 9s of durability." Amazon was the first motherfucker to claim this, but the other cloud storage folks are also culpable, but at least they mostly had the modesty to add some weasel words like "designed for" and didn't just straight up claim there was less than a 1 in a trillion chance of a durability failure. You don't have twelve 9s of durability. Your collection of copies of data on the hard drives do, assuming they exist in a vacuum and nothing bad happens to them except the normal sorts of things that cause hard drive failures that are nice and completely independent. But it completely ignores all other sources of problem, and those are so many orders of magnitude more common that you might as well claim "God-given, perfect durability" because it'd be just as accurate.

laluser 3y ago
Of course, these systems are always designed for that. Just like every system is designed for a certain amount of availability. Also, even in a vacuum, things would degrade over time due to bit rot, etc. That's why the article mentions protections with verifiers and due to other things such accidental deletions due to potential bugs.
sujayakar 3y ago
magic pocket's tech lead (disclaimer: my cofounder at convex) has a whole talk on this concept of "durability theater" [1]! the tldr is that those numbers of 9s are just table stakes. no system should ever lose data due to routine disk failures. so then, as you mention, there's another whole art to mitigating those other sources of problems. [1] https://www.facebook.com/atscaleevents/videos/1741691622770601/ https://www.facebook.com/atscaleevents/videos/17416916227706...
- james_cowling 3y ago
  We should also clarify that Sujay and I are part of the old old team :) The current team have been doing an awesome job since then! (a bunch of the early MP folks work at Convex now)
james_cowling 3y ago
Yeah we have a few talks about this and a chapter about this very issue in https://www.oreilly.com/library/view/seeking-sre/9781491978856/ch17.html https://www.oreilly.com/library/view/seeking-sre/97814919788.... Totally agree that in a well designed system the sources of data loss are certainly not disk failures. As far as I know Magic Pocket has had 100% durability, but that's obviously beside the point.
- james_cowling 3y ago
  "It’s fairly easy to design a system with astronomically high durability numbers. 24 nines is a mean time to failure of 1,000,000,000,000,000,000,000,000 years. When your MTTF dwarfs the age of the universe then it might be time to reevaluate your priorities. Should we trust these numbers though? Of course not, because the secret truth is that adherence to theoretical durability estimates is missing the point. They tell you how likely you are to lose data due to routine disk failure, but routine disk failure is easy to model for and protect against. If you lose data due to routine disk failure you’re probably doing something wrong." https://medium.com/@jamesacowling/how-many-nines-is-my-storage-system-7d16e852d56d https://medium.com/@jamesacowling/how-many-nines-is-my-stora...

laluser 3y ago

Of course, these systems are always designed for that. Just like every system is designed for a certain amount of availability. Also, even in a vacuum, things would degrade over time due to bit rot, etc. That's why the article mentions protections with verifiers and due to other things such accidental deletions due to potential bugs.

sujayakar 3y ago

magic pocket's tech lead (disclaimer: my cofounder at convex) has a whole talk on this concept of "durability theater" [1]! the tldr is that those numbers of 9s are just table stakes. no system should ever lose data due to routine disk failures. so then, as you mention, there's another whole art to mitigating those other sources of problems. [1] https://www.facebook.com/atscaleevents/videos/1741691622770601/ https://www.facebook.com/atscaleevents/videos/17416916227706...

james_cowling 3y ago
We should also clarify that Sujay and I are part of the old old team :) The current team have been doing an awesome job since then! (a bunch of the early MP folks work at Convex now)

james_cowling 3y ago

Yeah we have a few talks about this and a chapter about this very issue in https://www.oreilly.com/library/view/seeking-sre/9781491978856/ch17.html https://www.oreilly.com/library/view/seeking-sre/97814919788.... Totally agree that in a well designed system the sources of data loss are certainly not disk failures. As far as I know Magic Pocket has had 100% durability, but that's obviously beside the point.

james_cowling 3y ago
"It’s fairly easy to design a system with astronomically high durability numbers. 24 nines is a mean time to failure of 1,000,000,000,000,000,000,000,000 years. When your MTTF dwarfs the age of the universe then it might be time to reevaluate your priorities. Should we trust these numbers though? Of course not, because the secret truth is that adherence to theoretical durability estimates is missing the point. They tell you how likely you are to lose data due to routine disk failure, but routine disk failure is easy to model for and protect against. If you lose data due to routine disk failure you’re probably doing something wrong." https://medium.com/@jamesacowling/how-many-nines-is-my-storage-system-7d16e852d56d https://medium.com/@jamesacowling/how-many-nines-is-my-stora...