2 ms·

VoltDB Engineer here. It seems like the authors made some unfortunate choices in the configuration and usage of VoltDB. A limited number of synchronous loads a

by jhugg 14y ago

VoltDB Engineer here. It seems like the authors made some unfortunate choices in the configuration and usage of VoltDB. A limited number of synchronous loads are always going to present scalability problems and will do a poor job measureing the throughput achievable. Even if it takes 500usecs to round-trip a transaction from the client, that means each synchronous client can only do 2000 per second. To scale linearly to 12 nodes might require hundreds of these synchronous clients. VoltDB is commonly used with many parallel synchronous clients, such as web front-ends, or asynchronous workloads, such as event feeds. Both of these workload scale very well, so long as there is enough parallelism. They also used global, consistent, multi-partition transactions for the YCSB scan. I'm not sure if that is necessary, and it would also limit scalability. I can't speak too directly to the work because the code and configuration doesn't seem to be public. Perhaps I'm mistaken. It would have been nice if they had reached out to us at VoltDB to verify their configuration and client. They also could have contacted Andy Pavlo at the H-Store who has some experience with YCSB on H-Store. Still, we can always learn from something like this. One thing we're working on in VoltDB is better performance for workloads that aren't designed to be parallel. This means more performance for synchronous clients and for clients that do global consistent reads. Another area we could improve is more prominent diagnostics to show you why your cluster isn't running faster. That information is all available in our system tables and management tools, but we could probably boil it down to a single status field telling you whether your cluster is starved on client requests, intercluster-transaction-agreement or slow procedures. In our experience, it's usually the first one. I'll go file a ticket now.

jbellis 14y ago
Hmm, there has to be more to it than that. VoltDB throughput went down scaling from 1 node to 4 in every workload but RSW. This is not a sign of a system that would handle many more requests if only there were more client parallelism.
- jhugg 14y ago
  There is an explanation for this. VoltDB latency is currently sub-millisecond for single-node deployments, but often several times higher for clusters. This is due to the current system of global-ordering-agreement that adds some latency. So for synchronous workloads, you'll see a big drop moving from one to two nodes. With enough parallelism, you will see close to double the throughput. Still, we usually look at scalability starting at 2-3 nodes, because a distributed system with one node is a different animal. This is something we address in our docs, but people naively testing VoltDB run into it more than we'd like. So for the next major release, we've re-worked how we do global ordering and can now achieve sub-millisecond latency on larger clusters in internal testing. Note that there's no change to our serializable consistency to achieve lower latency. So this new scheme has a huge impact on synchronous workloads, but the scalability with enough parallelism has been close to linear all along.
- arielweisberg 14y ago
  Another Volt developer here. Short answer, Volt establishes a global order across transactions. Transactions can't be executed immediately because their position in the global order isn't known when they initially arrive. The exchange of ordering information is driven by in-flight transactions. In the absence of sufficient in-flight transactions to drive the ordering process heartbeats are sent every 5 milliseconds resulting in increased latency and lower throughput as the system waits on heartbeats. 2-4x the concurrency should have shown different results. We're finishing a different transaction initiation system that doesn't produce a global order so that people can do this kind of benchmark and get the expected result. You can switch the beta version on in 2.8. I also wonder if they had clock skew. The global ordering process is always delayed by the amount of clock skew you have. With NTP properly configured this is 10s of microseconds, but with a typical out of the box NTP config it will be milliseconds unless you have a lot of uptime. I don't really see this as an excuse and it is why we are eating the pain of a transaction initiation rewrite so people can use Volt the traditional way with small numbers of synchronous client threads.

by jhugg 14y ago

jbellis 14y ago

Hmm, there has to be more to it than that. VoltDB throughput went down scaling from 1 node to 4 in every workload but RSW. This is not a sign of a system that would handle many more requests if only there were more client parallelism.

jhugg 14y ago
There is an explanation for this. VoltDB latency is currently sub-millisecond for single-node deployments, but often several times higher for clusters. This is due to the current system of global-ordering-agreement that adds some latency. So for synchronous workloads, you'll see a big drop moving from one to two nodes. With enough parallelism, you will see close to double the throughput. Still, we usually look at scalability starting at 2-3 nodes, because a distributed system with one node is a different animal. This is something we address in our docs, but people naively testing VoltDB run into it more than we'd like. So for the next major release, we've re-worked how we do global ordering and can now achieve sub-millisecond latency on larger clusters in internal testing. Note that there's no change to our serializable consistency to achieve lower latency. So this new scheme has a huge impact on synchronous workloads, but the scalability with enough parallelism has been close to linear all along.
arielweisberg 14y ago
Another Volt developer here. Short answer, Volt establishes a global order across transactions. Transactions can't be executed immediately because their position in the global order isn't known when they initially arrive. The exchange of ordering information is driven by in-flight transactions. In the absence of sufficient in-flight transactions to drive the ordering process heartbeats are sent every 5 milliseconds resulting in increased latency and lower throughput as the system waits on heartbeats. 2-4x the concurrency should have shown different results. We're finishing a different transaction initiation system that doesn't produce a global order so that people can do this kind of benchmark and get the expected result. You can switch the beta version on in 2.8. I also wonder if they had clock skew. The global ordering process is always delayed by the amount of clock skew you have. With NTP properly configured this is 10s of microseconds, but with a typical out of the box NTP config it will be milliseconds unless you have a lot of uptime. I don't really see this as an excuse and it is why we are eating the pain of a transaction initiation rewrite so people can use Volt the traditional way with small numbers of synchronous client threads.