4 ms·

So you don't benefit in that simple approach from adding more disks. Not sure about this DB, but that's my understanding of consensus as well. The more nodes,

So you don't benefit in that simple approach from adding more disks. Not sure about this DB, but that's my understanding of consensus as well. The more nodes, the less likely any one going down harms things but the harder it is to reach consensus. You have to write to more places. It becomes slower not faster.

eatonphil 4y ago
> The more nodes, the less likely any one going down harms things but the harder it is to reach consensus Flexible quorums deals with this in the usual read/write case. But yeah leader election then becomes the bottleneck. So if the cluster is unstable overall performance gets even worse as you scale number of nodes. https://fpaxos.github.io/ https://fpaxos.github.io/
remram 4y ago
Those places could still all be closer to the client. If you have 50 nodes around the country and you require every write to be acknowledged by 3 nodes for consistency, you can probably find 3 nodes very close to the client and therefore commit writes faster than if they all came to a central location.
- eatonphil 4y ago
  Everything has to be directed through a leader though. So it's not a matter of how close the client is to the cluster but how far the cluster's nodes are from eachother (since anyone could be the leader). No?
  - remram 4y ago
    In practice you don't run the consensus over the whole cluster for every shard of data. This would be inefficient in terms of data storage (every node stores a copy of everything) and latency. For every shard you pick a few nodes, and you run consensus between those. This is the way Elasticsearch, CockroachDB, ... work. So if a specific shard is getting writes from a specific region, you can easily move that shard to nodes in that region. In contrast, etcd runs consensus over the whole cluster. So it is very safe, very slow, and meant for small datasets. In many ways, it is not a database, but a metadata store for distributed systems.
    - eatonphil 4y ago
      Yup that makes sense. It wasn't clear from the article though how their system works and other sqlite+raft implementations do the simpler version of one raft cluster for the entire database, I think.
    - anonymousDan 4y ago
      This is not entirely accurate. If you want to forego the ability to support cross shard transactions then sure you can run consensus independently on each shard. But I don't think that is what Spanner/CockRoachDB do. In addition, even if you want to limit transactions to a a single shard, the shard replicas need to be in different datacenters/AZs if you want to remain available when a DC/AZ goes down.
      - deleted 4y ago
        [deleted]
- adamgordonbell 4y ago
  you require every write to be acknowledged by 3 nodes for consistency How would that work? Someone else could get a different 3 nodes and write the same key, so they wouldn't read my write before writing their own? This isn't my area but I would assume the data has to be sharded ahead of time, not just write to any 3 out of 50 nodes that are closest to me.
bastawhiz 4y ago
If you have fifty nodes, you don't want all fifty working on every database. Each db should only be associated with with nodes to have a healthy quorum. But this is fine: you probably only have 3-5 regions. Each region could have 1-2 nodes supporting each db. It's almost certainly the case that you don't have dozens of regions/PoPs (and if you do, the material difference in latency between one PoP and another is probably ones of milliseconds for most users).

eatonphil 4y ago

> The more nodes, the less likely any one going down harms things but the harder it is to reach consensus Flexible quorums deals with this in the usual read/write case. But yeah leader election then becomes the bottleneck. So if the cluster is unstable overall performance gets even worse as you scale number of nodes. https://fpaxos.github.io/ https://fpaxos.github.io/

remram 4y ago

Those places could still all be closer to the client. If you have 50 nodes around the country and you require every write to be acknowledged by 3 nodes for consistency, you can probably find 3 nodes very close to the client and therefore commit writes faster than if they all came to a central location.

eatonphil 4y ago
Everything has to be directed through a leader though. So it's not a matter of how close the client is to the cluster but how far the cluster's nodes are from eachother (since anyone could be the leader). No?
- remram 4y ago
  In practice you don't run the consensus over the whole cluster for every shard of data. This would be inefficient in terms of data storage (every node stores a copy of everything) and latency. For every shard you pick a few nodes, and you run consensus between those. This is the way Elasticsearch, CockroachDB, ... work. So if a specific shard is getting writes from a specific region, you can easily move that shard to nodes in that region. In contrast, etcd runs consensus over the whole cluster. So it is very safe, very slow, and meant for small datasets. In many ways, it is not a database, but a metadata store for distributed systems.
  - eatonphil 4y ago
    Yup that makes sense. It wasn't clear from the article though how their system works and other sqlite+raft implementations do the simpler version of one raft cluster for the entire database, I think.
  - anonymousDan 4y ago
    This is not entirely accurate. If you want to forego the ability to support cross shard transactions then sure you can run consensus independently on each shard. But I don't think that is what Spanner/CockRoachDB do. In addition, even if you want to limit transactions to a a single shard, the shard replicas need to be in different datacenters/AZs if you want to remain available when a DC/AZ goes down.
    - deleted 4y ago
      [deleted]
adamgordonbell 4y ago
you require every write to be acknowledged by 3 nodes for consistency How would that work? Someone else could get a different 3 nodes and write the same key, so they wouldn't read my write before writing their own? This isn't my area but I would assume the data has to be sharded ahead of time, not just write to any 3 out of 50 nodes that are closest to me.

bastawhiz 4y ago

If you have fifty nodes, you don't want all fifty working on every database. Each db should only be associated with with nodes to have a healthy quorum. But this is fine: you probably only have 3-5 regions. Each region could have 1-2 nodes supporting each db. It's almost certainly the case that you don't have dozens of regions/PoPs (and if you do, the material difference in latency between one PoP and another is probably ones of milliseconds for most users).