3 ms·

So do the throughput decrease when you add more nodes? Or am I reading it wrong? This is a general problem with scaling horizontally, that the overhead can kill

by z3t4 5y ago

So do the throughput decrease when you add more nodes? Or am I reading it wrong? This is a general problem with scaling horizontally, that the overhead can kill the performance, and single core/node performance is sacrificed.

pas 5y ago
Likely you are reading it correctly. What you gain in possible availability (because now you have more nodes, which give some redundancy with regards to node failure) you have to pay the work necessary for constantly running the consistency protocol from your performance budget. How much you have to sacrifice, and how linear the scaling is, of course are important quality metrics of distributed systems. In multi-writer optimistic-locking sync-at-commit systems (eg Galera) in case of no conflict, it's possible to have the multi-node throughput exceed the throughput of the single-node version.
- AtlasBarfed 5y ago
  So this is a CP system?
  - pas 5y ago
    Kinf of. But it's not really Consistent in the CAP sense, as it only provides snapshot isolation, not linearizability, plus the A in CAP is kind of useless. (CAP is an amazing theoretical milestone nonetheless, just too narrow for real life situations. Basically it's a simple no-go theorem.) In practice even this ToyDB is likely able to serve requests in a degraded state (probably as long as the Raft leader's timer does not expire, and if there's a quorum of nodes they can reelect a leader). And it seems that if a node falls out of sync it will automatically rejoin and try to replay the logs. (As long as they are available of course.) https://github.com/erikgrinaker/toydb/blob/master/docs/architecture.md#raft-tradeoffs https://github.com/erikgrinaker/toydb/blob/master/docs/archi... https://www.youtube.com/watch?v=hUd_9FENShA https://www.youtube.com/watch?v=hUd_9FENShA https://blog.acolyer.org/2014/11/07/highly-available-transactions-virtues-and-limitations/ https://blog.acolyer.org/2014/11/07/highly-available-transac...

by z3t4 5y ago

pas 5y ago

Likely you are reading it correctly. What you gain in possible availability (because now you have more nodes, which give some redundancy with regards to node failure) you have to pay the work necessary for constantly running the consistency protocol from your performance budget. How much you have to sacrifice, and how linear the scaling is, of course are important quality metrics of distributed systems. In multi-writer optimistic-locking sync-at-commit systems (eg Galera) in case of no conflict, it's possible to have the multi-node throughput exceed the throughput of the single-node version.

AtlasBarfed 5y ago
So this is a CP system?
- pas 5y ago
  Kinf of. But it's not really Consistent in the CAP sense, as it only provides snapshot isolation, not linearizability, plus the A in CAP is kind of useless. (CAP is an amazing theoretical milestone nonetheless, just too narrow for real life situations. Basically it's a simple no-go theorem.) In practice even this ToyDB is likely able to serve requests in a degraded state (probably as long as the Raft leader's timer does not expire, and if there's a quorum of nodes they can reelect a leader). And it seems that if a node falls out of sync it will automatically rejoin and try to replay the logs. (As long as they are available of course.) https://github.com/erikgrinaker/toydb/blob/master/docs/architecture.md#raft-tradeoffs https://github.com/erikgrinaker/toydb/blob/master/docs/archi... https://www.youtube.com/watch?v=hUd_9FENShA https://www.youtube.com/watch?v=hUd_9FENShA https://blog.acolyer.org/2014/11/07/highly-available-transactions-virtues-and-limitations/ https://blog.acolyer.org/2014/11/07/highly-available-transac...