3 ms·

> Totally ordered request processing requires waiting for global consensus at each step, and thus fundamentally limits the throughput of each replica-set. real

> Totally ordered request processing requires waiting for global consensus at each step, and thus fundamentally limits the throughput of each replica-set. really? such global consensus increases latency (network round trip plus fsync write), with a fully batched and pipelined concurrent design, when CPU cycles are being saturated in those benchmarks, why throughput is fundamentally limited by such increased latency?

zzzcpan 9y ago
Let's say we have a primitive system with total ordering and single static coordinator per each replica-set. And we want to update a global counter from every node. Now since counter is global it lives on a single replica-set and every node has to communicate with this one coordinator. Be it 4 nodes or 4000, still one coordinator. If we drop total ordering and use something like gossip protocol to communicate between nodes, we can merge this counter everywhere in the system as it propagates, eliminating all unnecessary communications and distributing communications in the network. So, yeah, global consensus fundamentally limits both throughput and latency.
jmull 9y ago
That's a really good question... I think the answer is something like this (just spitballing here, not trying to make a strong claim): At some point in a system, you may need a response to one request in order to generate the next request. At this point latency affects throughput. Also, apparently, their "lattice composition" technique let's them push concurrency to a lower level, which would allow them to avoid the overhead of doing concurrency at a higher level. What I mean is, if you have to process items one-at-a-time in a replica, then to process more items at a time, you need more replicas. But each replica has overhead -- the details depend on what technology we're talking about, but there's overhead to each replica... e.g., to utilize 4 cores on your server you might host four replicas on that server... but now you have to maintain four memory spaces as well, which takes cycles, even though all you wanted was to make use of all your cores to process items. And it might not be four cores, but 18 or 36 or perhaps 1000s. At some point, the cost of the overhead is greater than amount of benefit.

zzzcpan 9y ago

Let's say we have a primitive system with total ordering and single static coordinator per each replica-set. And we want to update a global counter from every node. Now since counter is global it lives on a single replica-set and every node has to communicate with this one coordinator. Be it 4 nodes or 4000, still one coordinator. If we drop total ordering and use something like gossip protocol to communicate between nodes, we can merge this counter everywhere in the system as it propagates, eliminating all unnecessary communications and distributing communications in the network. So, yeah, global consensus fundamentally limits both throughput and latency.

jmull 9y ago

That's a really good question... I think the answer is something like this (just spitballing here, not trying to make a strong claim): At some point in a system, you may need a response to one request in order to generate the next request. At this point latency affects throughput. Also, apparently, their "lattice composition" technique let's them push concurrency to a lower level, which would allow them to avoid the overhead of doing concurrency at a higher level. What I mean is, if you have to process items one-at-a-time in a replica, then to process more items at a time, you need more replicas. But each replica has overhead -- the details depend on what technology we're talking about, but there's overhead to each replica... e.g., to utilize 4 cores on your server you might host four replicas on that server... but now you have to maintain four memory spaces as well, which takes cycles, even though all you wanted was to make use of all your cores to process items. And it might not be four cores, but 18 or 36 or perhaps 1000s. At some point, the cost of the overhead is greater than amount of benefit.