3 ms·

This is a really good writeup of a very complicated topic. Thank you. This kind of thing there's a lot that needs to be kept in mind, I've never done a live da

This is a really good writeup of a very complicated topic. Thank you. This kind of thing there's a lot that needs to be kept in mind, I've never done a live database migration before to this scale. I'm a beginner with distributed systems but I find them highly interesting. I feel that along of the work that was done in this article should have been the responsibility of a Kubernetes operator. My dream is multimaster databases that mean we don't need to have so much pain in doing database migrations. I wrote an eventually consistent Python socket server that asynchonously streams messages to its replicas. I use a client side timestamp to provide global ordering. In theory if two servers had differing system clocks, they could cause the causality of messages to be the wrong way round in a strict sense. But ALL replicas shall agree on the ordering, server agreement among replicas that are caught up is deterministic. If the servers are not caught up, a server shall serve a stale value or a value that only that replica knows about. The database essentially uses event sourcing and takes the last write to determine victory. Here's a diagram of a log where the cluster bounced between values, it's a very big diagram you'll have to zoom in. You can see states that were forks of the true cluster agreed value. https://github.com/samsquire/eventually-consistent-mesh/blob/main/diagrams/0.png https://github.com/samsquire/eventually-consistent-mesh/blob... I wrote a Jepsen test that because it is not linerarizable, it fails. My script abandons the C(onsistency) of CAP and maintains (A)vailability and (P)artition tolerance. It assumes the network shall heal and the replication shall resume. In PACELC theory in During normal operation it prefers Availability during Partitions and Else Latency in normal operations. How do you build a system around a database where values can no longer be valid at any time? If I kick off a process in the database with a value that was only reported by one replica that was partitioned away from the majority of the network, what do I do? My immediate thoughts are to create dependency relationships to values that are on the agreed path, data that is based on a failed branch is not valid. https://github.com/samsquire/eventually-consistent-mesh https://github.com/samsquire/eventually-consistent-mesh

mjb 3y ago
> How do you build a system around a database where values can no longer be valid at any time? This is a difficult problem, and there aren't nice general solutions. If you read the original Amazon Dynamo paper, for example, there's a good discussion of why eventual consistency is OK for their applications (https://www.allthingsdistributed.com/files/amazon-dynamo-sosp2007.pdf https://www.allthingsdistributed.com/files/amazon-dynamo-sos...), and more in Werner Vogel's CACM article from two years later (https://dl.acm.org/doi/10.1145/1435417.1435432 https://dl.acm.org/doi/10.1145/1435417.1435432). There are conceptual frameworks (like CALM https://arxiv.org/pdf/1901.01930.pdf https://arxiv.org/pdf/1901.01930.pdf) that provide a way to reason through the behavior of systems under eventual consistency (and provide a path to data structures with behavior that's easier to reason about than last-writer-wins). > My script abandons the C(onsistency) of CAP and maintains (A)vailability and (P)artition tolerance. I'd recommend you check out "Rethinking Eventual Consistency" (https://www.microsoft.com/en-us/research/publication/rethinking-eventual-consistency/ https://www.microsoft.com/en-us/research/publication/rethink...). PACELC is a better tool than CAP for thinking about this stuff, but Figure 1 in that paper is even better. Specifically, there is no need to abandon either availability or consistency in the majority partition. The confusion here comes from CAP's unhelpful use of "total availability", which isn't a system property that most datacenter applications care about. Mobile, IoT, and other frequently-disconnected applications do care about it.

mjb 3y ago

> How do you build a system around a database where values can no longer be valid at any time? This is a difficult problem, and there aren't nice general solutions. If you read the original Amazon Dynamo paper, for example, there's a good discussion of why eventual consistency is OK for their applications (https://www.allthingsdistributed.com/files/amazon-dynamo-sosp2007.pdf https://www.allthingsdistributed.com/files/amazon-dynamo-sos...), and more in Werner Vogel's CACM article from two years later (https://dl.acm.org/doi/10.1145/1435417.1435432 https://dl.acm.org/doi/10.1145/1435417.1435432). There are conceptual frameworks (like CALM https://arxiv.org/pdf/1901.01930.pdf https://arxiv.org/pdf/1901.01930.pdf) that provide a way to reason through the behavior of systems under eventual consistency (and provide a path to data structures with behavior that's easier to reason about than last-writer-wins). > My script abandons the C(onsistency) of CAP and maintains (A)vailability and (P)artition tolerance. I'd recommend you check out "Rethinking Eventual Consistency" (https://www.microsoft.com/en-us/research/publication/rethinking-eventual-consistency/ https://www.microsoft.com/en-us/research/publication/rethink...). PACELC is a better tool than CAP for thinking about this stuff, but Figure 1 in that paper is even better. Specifically, there is no need to abandon either availability or consistency in the majority partition. The confusion here comes from CAP's unhelpful use of "total availability", which isn't a system property that most datacenter applications care about. Mobile, IoT, and other frequently-disconnected applications do care about it.