3 ms·
> Ultimately I was hoping that antirez and other contributors might realize why their proposal for a custom replication protocol was unsafe nine months ago, and
by rdtsc 13y ago
> Ultimately I was hoping that antirez and other contributors might realize why their proposal for a custom replication protocol was unsafe nine months ago, and abandon it in favor of an established algorithm with a formal model and a peer-reviewed proof, but that hasn’t happened yet. Redis continues to accrete homegrown consensus and replication algorithms without even a cursory nod to formal analysis.
That is kind of my feel. Redis is an outstanding product with a beautiful code base. This replication feature has been tough though. It is kind of due to external factors as I've mentioned in the previous post. Everyone and their cousin are talking about distributed databases, everyone likes CAP, CRDTs, Vector Clocks, Raft, Zookeeper and so on. It is hard to come up and say "Here I have made this custom replication protocol". Everyone stares and asks, "Hey where is your whitepaper or your partition tolerance tests?". 5-7 years ago, there would be only nods and approvals. The other aspect is this is about a database, so it is potentially toying and touching user's valuable data. If that gets lost either by a bug, mis-communication in docs, bad default, anything, it will not be taken lightly.
In the end I think it is fine to have it as what it is, with the warnings and disclaimers that data could be lost and avoiding papering over or hiding issues.
As an extra side note, simply put partition tolerance is hard. Net-splits are the devil of the distributed world. Some claim it doesn't exist or doesn't happens often. Others fear and tremble its name is mentioned. When it does happen it means having to resolve conflicts, throwing away user data, stopping killing your availability to stop some from accepting writes in order to provide consistency. This is a tough test (that Aphyr runs) and not very many databases fair well in it. But it is good these things are discussed.
- DanielRibeiro 13y agoThe author, Kyle Kingsbury (aka aphyr[1]), has done an amazing experimental analysis of several distributed databases/datastores recently[2]. I hope his experiments and writings not only shine some light into how these systems actually behave in practice, but also influence the underlying projects to improve. [1] https://github.com/aphyr https://github.com/aphyr [2] http://www.infoq.com/presentations/partitioning-comparison http://www.infoq.com/presentations/partitioning-comparison
- jasonwatkinspdx 13y agoI think the blog post is valuable, but I'd point out that aphyr is also not publishing formal proofs or utilizing the verification he suggests is necessary. I don't think this is intentional, but they should realize and own that they are borrowing authority of formal method they are not demonstrating and then criticizing others for the same lack of demonstration.
- aphyr 13y agoWell this one's proof by construction, so I don't feel the need to go particularly deep into the math. If you prefer a more rigorous approach take a look at the TLA proofs earlier in the Jepsen series; think I linked one on Redis+async replication in the article.
- jasonwatkinspdx 13y agoI read and enjoyed them, thanks. I'd just like people to be in a more collaborative vs combative mode, and I think a big part of that is not overly relying on rhetoric or overreaching what you've actually demonstrated. Using formalism on a blog is commendable, but it's also not the same thing as a properly reviewed paper. People should keep straight the levels of dialog here and value them each for the light they shed. This has been mostly a productive exchange despite a few "yer doin' it wrong bro" attitude folks, and I just want to throw my hat in to strongly advocate for keeping it that way.
- davidw 13y agoAs a way of checking, when a phrase is in character for "Comic Book Guy", you know it's probably a bit too much on the mean-spirited side of things.
- justinsb 13y agoIsn't the simulation sufficient? A proof that something works requires logic, but a single counter-example suffices to show that something doesn't work.
- 13y ago