4 ms·
This is just one of those solutions that was beautiful and simple and why didn't we think of that. The goal of distributed consensus can be viewed as something
by crdrost 8y ago
This is just one of those solutions that was beautiful and simple and why didn't we think of that. The goal of distributed consensus can be viewed as something to do with creating a transitive order A < B < C < D out of individual ordering judgments: so I might know A < B and C < D, someone else knows B < C, eventually everybody can know the full sequence of items. Making this sequence contain metaïnformation about the entities that it contains has been a well-known trick for just about forever, usually done by using that metadata to elect a “leader” who handles the writes personally, for speed's sake.
But the idea of switching from a single leader to a quorum and then using the metadata to commit to overlapping quorum groups in order to preserve this transitive property between orderings was just really really slick.
About three years ago there was an early presentation about the backend https://www.youtube.com/watch?v=-TbRxwcux3c https://www.youtube.com/watch?v=-TbRxwcux3c which made it just seem like the 4/6 thing just had to do with a normal replicated database or so, it seemed like a fixed topology to do with their availability zones—I didn't realize that they were also dealing with some sort of consensus overhead to enable swapping new nodes in or out.
There was also a somewhat marketingy presentation about this technology in the deep dive at https://www.youtube.com/watch?v=U42mC_iKSBg https://www.youtube.com/watch?v=U42mC_iKSBg which I confess kinda went over my head at the time, so these two blog articles are a wonderful condensing of the heart of the service for me. None of the companies I've worked for has made much use of Amazon but it's a really clever idea that certainly deserves to kick around in the back of folks' heads.