4 ms·

I'm not an expert and every distributed database is different. My intuition about strapping a consensus library to a database is that you're going to have way h

I'm not an expert and every distributed database is different. My intuition about strapping a consensus library to a database is that you're going to have way higher latency because every operation, writes and reads, must go through consensus. (IIRC some databases like Cockroach mitigate this by having many mini Raft clusters rather than one single one [which IIRC is how rqlite works anyway].) The advantage you undeniably get is durability/high availability. > Scalability: Distributed databases can be scaled horizontally by adding more nodes to the network, making handling larger amounts of data and higher volumes of requests easier. This is only possible if you have a data partitioning scheme, right? But data partitioning is sort of a different problem from consensus/HA, right? If you do the simplest version of a consensus + database implementation, every disk on every instance of the consensus cluster is basically identical. So you don't benefit in that simple approach from adding more disks. It potentially just makes write latency worse unless you do flexible quorums so the quorum size is independent of the number of nodes in the cluster? > Localized access: With a distributed database, data can be stored closer to the users who need it, which can improve response times and reduce network latency. This would also only be true if you relax consistency requirements, right? If you need to be at the highest level of consistency you must send reads through consensus too otherwise if you read from the node nearest you it may be out of date. It would be cool to see some numbers about the difference between latency of reads/writes in sqlitecloud and sqlite on a single node! It would help my intuition anyway.

adamgordonbell 4y ago
So you don't benefit in that simple approach from adding more disks. Not sure about this DB, but that's my understanding of consensus as well. The more nodes, the less likely any one going down harms things but the harder it is to reach consensus. You have to write to more places. It becomes slower not faster.
- eatonphil 4y ago
  > The more nodes, the less likely any one going down harms things but the harder it is to reach consensus Flexible quorums deals with this in the usual read/write case. But yeah leader election then becomes the bottleneck. So if the cluster is unstable overall performance gets even worse as you scale number of nodes. https://fpaxos.github.io/ https://fpaxos.github.io/
- remram 4y ago
  Those places could still all be closer to the client. If you have 50 nodes around the country and you require every write to be acknowledged by 3 nodes for consistency, you can probably find 3 nodes very close to the client and therefore commit writes faster than if they all came to a central location.
  - eatonphil 4y ago
    Everything has to be directed through a leader though. So it's not a matter of how close the client is to the cluster but how far the cluster's nodes are from eachother (since anyone could be the leader). No?
    - remram 4y ago
      In practice you don't run the consensus over the whole cluster for every shard of data. This would be inefficient in terms of data storage (every node stores a copy of everything) and latency. For every shard you pick a few nodes, and you run consensus between those. This is the way Elasticsearch, CockroachDB, ... work. So if a specific shard is getting writes from a specific region, you can easily move that shard to nodes in that region. In contrast, etcd runs consensus over the whole cluster. So it is very safe, very slow, and meant for small datasets. In many ways, it is not a database, but a metadata store for distributed systems.
      - eatonphil 4y ago
        Yup that makes sense. It wasn't clear from the article though how their system works and other sqlite+raft implementations do the simpler version of one raft cluster for the entire database, I think.
      - anonymousDan 4y ago
        This is not entirely accurate. If you want to forego the ability to support cross shard transactions then sure you can run consensus independently on each shard. But I don't think that is what Spanner/CockRoachDB do. In addition, even if you want to limit transactions to a a single shard, the shard replicas need to be in different datacenters/AZs if you want to remain available when a DC/AZ goes down.
- bastawhiz 4y ago
  If you have fifty nodes, you don't want all fifty working on every database. Each db should only be associated with with nodes to have a healthy quorum. But this is fine: you probably only have 3-5 regions. Each region could have 1-2 nodes supporting each db. It's almost certainly the case that you don't have dozens of regions/PoPs (and if you do, the material difference in latency between one PoP and another is probably ones of milliseconds for most users).
aaronblohowiak 4y ago
> My intuition about strapping a consensus library to a database is that you're going to have way higher latency because every operation, writes and reads, must go through consensus nah, you have leases (a time-limited and often renewable lock). so like your leader gets elected through consensus and then for the next N miliseconds, you have a garauntee that nobody else can be leader. this means your reads dont need to go through consensus. (hand-waving a lot of the details here.)
mikesun 4y ago
I think there's some confusion at what distributed databases like Spanner, Cockroach, etc do: - A given row of data is configured to have N replicas, e.g. 3, 5, etc, but N doesn't have to equal the actual size the database cluster. e.g. you can have a database cluster of 21 nodes with N being 3. N represents the "Raft cluster size" and the database is composed of many "Raft clusters" that are responsible for segments of the data space. You can horizontally scale the size of the database cluster by adding more nodes and thus more "Raft clusters" - Raft/Paxos are used to ensure linearizable writes to those replicas through a leader - Multi row/partition transactions are handled through a different kind consensus called atomic commit/2 phase commit that's orthogonal to Raft/Paxos consensus: https://timilearning.com/posts/mit-6.824/lecture-12-distributed-transactions/#two-phase-commit-and-raft https://timilearning.com/posts/mit-6.824/lecture-12-distribu... - Systems like Spanner, Cockroach have higher write latencies because replicas in Raft cluster have to achieve quorum consensus, but reads latency can mostly not be affected by always reading from the leader of a cluster via things like leader leases.
mountainreason 4y ago
I think the idea with localized access is that reads coming in after the initial commit can hit a nearby db instance, rather than the single global instance. I agree with you about horizontal scalability in general though, the data isn't being partitioned. I guess read heavy loads can be scaled.

adamgordonbell 4y ago

So you don't benefit in that simple approach from adding more disks. Not sure about this DB, but that's my understanding of consensus as well. The more nodes, the less likely any one going down harms things but the harder it is to reach consensus. You have to write to more places. It becomes slower not faster.

eatonphil 4y ago
> The more nodes, the less likely any one going down harms things but the harder it is to reach consensus Flexible quorums deals with this in the usual read/write case. But yeah leader election then becomes the bottleneck. So if the cluster is unstable overall performance gets even worse as you scale number of nodes. https://fpaxos.github.io/ https://fpaxos.github.io/
remram 4y ago
Those places could still all be closer to the client. If you have 50 nodes around the country and you require every write to be acknowledged by 3 nodes for consistency, you can probably find 3 nodes very close to the client and therefore commit writes faster than if they all came to a central location.
- eatonphil 4y ago
  Everything has to be directed through a leader though. So it's not a matter of how close the client is to the cluster but how far the cluster's nodes are from eachother (since anyone could be the leader). No?
  - remram 4y ago
    In practice you don't run the consensus over the whole cluster for every shard of data. This would be inefficient in terms of data storage (every node stores a copy of everything) and latency. For every shard you pick a few nodes, and you run consensus between those. This is the way Elasticsearch, CockroachDB, ... work. So if a specific shard is getting writes from a specific region, you can easily move that shard to nodes in that region. In contrast, etcd runs consensus over the whole cluster. So it is very safe, very slow, and meant for small datasets. In many ways, it is not a database, but a metadata store for distributed systems.
    - eatonphil 4y ago
      Yup that makes sense. It wasn't clear from the article though how their system works and other sqlite+raft implementations do the simpler version of one raft cluster for the entire database, I think.
    - anonymousDan 4y ago
      This is not entirely accurate. If you want to forego the ability to support cross shard transactions then sure you can run consensus independently on each shard. But I don't think that is what Spanner/CockRoachDB do. In addition, even if you want to limit transactions to a a single shard, the shard replicas need to be in different datacenters/AZs if you want to remain available when a DC/AZ goes down.
bastawhiz 4y ago
If you have fifty nodes, you don't want all fifty working on every database. Each db should only be associated with with nodes to have a healthy quorum. But this is fine: you probably only have 3-5 regions. Each region could have 1-2 nodes supporting each db. It's almost certainly the case that you don't have dozens of regions/PoPs (and if you do, the material difference in latency between one PoP and another is probably ones of milliseconds for most users).

aaronblohowiak 4y ago

> My intuition about strapping a consensus library to a database is that you're going to have way higher latency because every operation, writes and reads, must go through consensus nah, you have leases (a time-limited and often renewable lock). so like your leader gets elected through consensus and then for the next N miliseconds, you have a garauntee that nobody else can be leader. this means your reads dont need to go through consensus. (hand-waving a lot of the details here.)

mikesun 4y ago

I think there's some confusion at what distributed databases like Spanner, Cockroach, etc do: - A given row of data is configured to have N replicas, e.g. 3, 5, etc, but N doesn't have to equal the actual size the database cluster. e.g. you can have a database cluster of 21 nodes with N being 3. N represents the "Raft cluster size" and the database is composed of many "Raft clusters" that are responsible for segments of the data space. You can horizontally scale the size of the database cluster by adding more nodes and thus more "Raft clusters" - Raft/Paxos are used to ensure linearizable writes to those replicas through a leader - Multi row/partition transactions are handled through a different kind consensus called atomic commit/2 phase commit that's orthogonal to Raft/Paxos consensus: https://timilearning.com/posts/mit-6.824/lecture-12-distributed-transactions/#two-phase-commit-and-raft https://timilearning.com/posts/mit-6.824/lecture-12-distribu... - Systems like Spanner, Cockroach have higher write latencies because replicas in Raft cluster have to achieve quorum consensus, but reads latency can mostly not be affected by always reading from the leader of a cluster via things like leader leases.

mountainreason 4y ago

I think the idea with localized access is that reads coming in after the initial commit can hit a nearby db instance, rather than the single global instance. I agree with you about horizontal scalability in general though, the data isn't being partitioned. I guess read heavy loads can be scaled.