5 ms·

Hi, here to answer question number 4. Short answer: Our configuration data is most similar to git. Any machine can be used as an administrative node via the We

Hi, here to answer question number 4. Short answer: Our configuration data is most similar to git. Any machine can be used as an administrative node via the WebUI or the CLI. It will make changes to the metadata which then get pushed to the other nodes. If 2 nodes make conflicting changes you get a conflict which the system will help you to merge. Long Answer Cluster configuration is stored in semilattices which are a neat mathematical structure with a few very desirable properties. Semilattices support have a join operator. For our cluster metadata joining is the means by which metadata is updated. When one server connects to another the two swap metadata and each joins the other's metadata into his own. In essence learning what the other knows. There are two properties in particular of the joining that are nice. First off joining is commutative. This means machines can exchange data in whatever order they want and get the same result at the end. Secondly they're indempotent. That means machines can resend their data without fear. The value doesn't change if the same value is joined in twice. These help us with a lot of the worries of distributed systems.

szopa 14y ago
Interesting. Do you have any way of checking that the change has actually propagated through the system before starting to act on it? Is the system consistent at all times? If I understand correctly, the client can connect to any instance and its request will get routed appropriately. Let's assume that you take a master offline and promote one of the replicas to be a new master. Won't that lead to a window in which (from the point of view of different instances) there are two masters at the same time and some writes are sent to the wrong instance? EDIT: One solution for such things is to use something like Zookeeper (or some other system whose documentation mentions "Paxos" ;)). Have you considered that? How does what you are doing compare with that?
- coffeemug 14y ago
  Joe may be responding to this soon, but in the meantime I'll chime in. There is no way to verify the propagation reliably without either introducing strong performance inefficiencies (e.g. two phase commit protocol), or divergence (paxos, semi lattices, etc.) In our implementation we're using immediately consistent algorithms for data, but eventually consistent algorithms for cluster metadata. This means that if there is a metadata conflict, the user is presented with an issue (via the web ui or CLI) that they have to resolve. We'll also be adding automated resolution soon. We basically have something very similar to zookeper baked into rethinkdb. We wrote it internally from scratch to better suit the needs of our architecture.
yxhuvud 14y ago
Now that you mention it, it would be very nice to have a database suited for configuration that behaved like git in that branching, restoring old states, reverting selected commits was built in, while at the same time supporting ACID features and replication? Does anyone know if something like that exist?

szopa 14y ago

Interesting. Do you have any way of checking that the change has actually propagated through the system before starting to act on it? Is the system consistent at all times? If I understand correctly, the client can connect to any instance and its request will get routed appropriately. Let's assume that you take a master offline and promote one of the replicas to be a new master. Won't that lead to a window in which (from the point of view of different instances) there are two masters at the same time and some writes are sent to the wrong instance? EDIT: One solution for such things is to use something like Zookeeper (or some other system whose documentation mentions "Paxos" ;)). Have you considered that? How does what you are doing compare with that?

coffeemug 14y ago
Joe may be responding to this soon, but in the meantime I'll chime in. There is no way to verify the propagation reliably without either introducing strong performance inefficiencies (e.g. two phase commit protocol), or divergence (paxos, semi lattices, etc.) In our implementation we're using immediately consistent algorithms for data, but eventually consistent algorithms for cluster metadata. This means that if there is a metadata conflict, the user is presented with an issue (via the web ui or CLI) that they have to resolve. We'll also be adding automated resolution soon. We basically have something very similar to zookeper baked into rethinkdb. We wrote it internally from scratch to better suit the needs of our architecture.

yxhuvud 14y ago

Now that you mention it, it would be very nice to have a database suited for configuration that behaved like git in that branching, restoring old states, reverting selected commits was built in, while at the same time supporting ACID features and replication? Does anyone know if something like that exist?