3 ms·

Those places could still all be closer to the client. If you have 50 nodes around the country and you require every write to be acknowledged by 3 nodes for cons

by remram 4y ago

Those places could still all be closer to the client. If you have 50 nodes around the country and you require every write to be acknowledged by 3 nodes for consistency, you can probably find 3 nodes very close to the client and therefore commit writes faster than if they all came to a central location.

eatonphil 4y ago
Everything has to be directed through a leader though. So it's not a matter of how close the client is to the cluster but how far the cluster's nodes are from eachother (since anyone could be the leader). No?
- remram 4y ago
  In practice you don't run the consensus over the whole cluster for every shard of data. This would be inefficient in terms of data storage (every node stores a copy of everything) and latency. For every shard you pick a few nodes, and you run consensus between those. This is the way Elasticsearch, CockroachDB, ... work. So if a specific shard is getting writes from a specific region, you can easily move that shard to nodes in that region. In contrast, etcd runs consensus over the whole cluster. So it is very safe, very slow, and meant for small datasets. In many ways, it is not a database, but a metadata store for distributed systems.
  - eatonphil 4y ago
    Yup that makes sense. It wasn't clear from the article though how their system works and other sqlite+raft implementations do the simpler version of one raft cluster for the entire database, I think.
  - anonymousDan 4y ago
    This is not entirely accurate. If you want to forego the ability to support cross shard transactions then sure you can run consensus independently on each shard. But I don't think that is what Spanner/CockRoachDB do. In addition, even if you want to limit transactions to a a single shard, the shard replicas need to be in different datacenters/AZs if you want to remain available when a DC/AZ goes down.
    - deleted 4y ago
      [deleted]
adamgordonbell 4y ago
you require every write to be acknowledged by 3 nodes for consistency How would that work? Someone else could get a different 3 nodes and write the same key, so they wouldn't read my write before writing their own? This isn't my area but I would assume the data has to be sharded ahead of time, not just write to any 3 out of 50 nodes that are closest to me.

by remram 4y ago

eatonphil 4y ago

Everything has to be directed through a leader though. So it's not a matter of how close the client is to the cluster but how far the cluster's nodes are from eachother (since anyone could be the leader). No?

remram 4y ago
In practice you don't run the consensus over the whole cluster for every shard of data. This would be inefficient in terms of data storage (every node stores a copy of everything) and latency. For every shard you pick a few nodes, and you run consensus between those. This is the way Elasticsearch, CockroachDB, ... work. So if a specific shard is getting writes from a specific region, you can easily move that shard to nodes in that region. In contrast, etcd runs consensus over the whole cluster. So it is very safe, very slow, and meant for small datasets. In many ways, it is not a database, but a metadata store for distributed systems.
- eatonphil 4y ago
  Yup that makes sense. It wasn't clear from the article though how their system works and other sqlite+raft implementations do the simpler version of one raft cluster for the entire database, I think.
- anonymousDan 4y ago
  This is not entirely accurate. If you want to forego the ability to support cross shard transactions then sure you can run consensus independently on each shard. But I don't think that is what Spanner/CockRoachDB do. In addition, even if you want to limit transactions to a a single shard, the shard replicas need to be in different datacenters/AZs if you want to remain available when a DC/AZ goes down.
  - deleted 4y ago
    [deleted]

adamgordonbell 4y ago

you require every write to be acknowledged by 3 nodes for consistency How would that work? Someone else could get a different 3 nodes and write the same key, so they wouldn't read my write before writing their own? This isn't my area but I would assume the data has to be sharded ahead of time, not just write to any 3 out of 50 nodes that are closest to me.