6 ms·

John here from VoltDB. We really enjoyed working with Kyle on this project. We’ve got some content on our website related to this work if you’re hungry for more

by jhugg 10y ago

John here from VoltDB. We really enjoyed working with Kyle on this project. We’ve got some content on our website related to this work if you’re hungry for more, including a blog post, a FAQ on transactions and consistency and more detail on the issues Kyle found in VoltDB 6.3 that have been fixed in 6.4. It can all be found here: https://voltdb.com/jepsen https://voltdb.com/jepsen There are also a number of us here to answer any questions about this Jepsen work or about VoltDB generally.

jasonjackson 10y ago
I hadn't heard of VoltDB until now, but have been following jepsen for a while. I wish more project did what you did. Congrats on the increased reliability.
hobs 10y ago
All software has bugs, not everyone pays independent and respected members of the community to root them out. Kudos.
swordswinger12 10y ago
Why roll your own consensus algorithm?
- jhugg 10y ago
  Historical reasons mostly. First, Raft wasn't a thing when we started. Second, the consensus algorithm we use today is used for cluster membership, schema changes and other agreement purposes, but not for the main transactional path. However, in versions before 3.0 it was. At the time, we wanted to make really different tradeoffs than something like Paxos. Namely, we wanted to be able to run with two copies, rather than the three that Paxos/Raft require. We also wanted crazy throughput and low performance variation, which are hard things to get with Paxos/Raft. The main tradeoff is that our system doesn't work well across high-latency links, which is ok with us. Ultimately we added a second layer protocol for transactions in 3.0, and this made the transaction path not dependent on clock skew. This layer actually looks a lot like Raft, but does differ in some key ways. We may even someday switch to Raft for the core agreement stuff, but there's a lot of work to be done for that to happen. This link has more info about how transactions work in VoltDB: https://voltdb.com/sites/default/files/tn-transactions.pdf https://voltdb.com/sites/default/files/tn-transactions.pdf
  - wmccullough 10y ago
    This is a great explanation. You can see the thought process and care that went into the design.
  - sseveran 10y ago
    Did you make formal specification of your algorithms in something like TLA+?
    - jhugg 10y ago
      That would be cool. Kyle Kingsbury started something like this (in Clojure) as part of this project (https://github.com/jepsen-io/voltdb/tree/master/replication-model https://github.com/jepsen-io/voltdb/tree/master/replication-...), but it's a large task and would probably take more than a few days. We have done some initial work on TLA+, but it may be a while to finish anything. If someone wants to help, there's probably a publishable paper in it.
      - sseveran 10y ago
        Cool. I am always a little wary of distributed algorithms that don't go through that level of validation. But I recognize that few commercial firms do that, although AWS is a stand out. I had the same criticism of ZooKeeper for a long time. Glad to hear you have at least considered it.
heavenlyhash 10y ago
You and your team's resolve and success in tacking these bugs is awesome. It's interesting to read about some of the relatively "unusual" choices VoltDB has made as well. Like this one: > network partitions cause minority nodes to shut down permanently; operator intervention is required to restore full service ...combined with requiring majorities of current rather than original clusters to continue. This is, for human operational interactions, really interesting. There's a big history of self-healing systems getting a little carried away and masking or sometimes even triggering bigger issues. Making signoff to rejoin nodes a required operation seems like it makes a whole category of "flapping" problems a non-issue. Kudos for making these "unusual" choices; this is a great horizon to explore :)
- jhugg 10y ago
  We have some users who have automated rejoin. This might actually make sense in the cloud, where you just spin up another pristine instance. Personally, when a node fails, I still want to try to understand why before I take the next operational step. Thanks for your words, BTW.
baq 10y ago
why would i use voltdb? serious question. this is the first time i hear about voltdb. i mean, i can use a postgres or mssql cluster for SQL needs and/or i can use riak/... for kv and/or cassandra/... for column indexed storage and/or ... etc. why should i look at voltdb?
- jhugg 10y ago
  Well, one thing we can say definitively today is that VoltDB offers strong serializability, when almost no other clustered systems do, and when they do, they are slow. But my more typical answer is the combination of throughput and transactional logic. No other system does both as well as VoltDB. This comes up a lot in policy enforcement, fraud detection, online gaming, ad-tech, billing-support and more. Here are some blog posts that might help: https://voltdb.com/blog/winning-now-and-future-where-voltdb-shines https://voltdb.com/blog/winning-now-and-future-where-voltdb-... https://voltdb.com/blog/apps-need-acid https://voltdb.com/blog/apps-need-acid https://voltdb.com/blog/call-center-example-integrating-processing-and-state-make-streaming-problems-simple-solve https://voltdb.com/blog/call-center-example-integrating-proc... This video's not bad: https://voltdb.com/resources/video/transactional-streaming-if-you-can-compute-it-you-can-probably-stream-it https://voltdb.com/resources/video/transactional-streaming-i...
- ruraljuror 10y ago
  If you like audio, Stonebreaker's interview on Software Engineering Radio is opinionated, excellent, and I think it will answer that question: http://www.se-radio.net/2013/12/episode-199-michael-stonebraker/ http://www.se-radio.net/2013/12/episode-199-michael-stonebra...
willvarfar 10y ago
Why does VoltDB require that transactions are written as stored procedures? Why can't a sequence of statements between a begin and end be transparently treated as though they were the body of a stored procedure?
- jhugg 10y ago
  You can actually send a bunch of SQL statements together, separated by semicolons to VoltDB and it will be treated as a transaction. In practice, this isn't always all that useful without a real stored procedure language. Supporting other ways to make procedures is something we've always planned and at some point we will get to it. I personally like javascript and lua.
cdcarter 10y ago
It seems like several of the issues aphyr found were due to optimizations the product was making. Fixing them in a point release suggests they weren't huge changes with massive ramifications. I don't know what the key benchmarks for software like VoltDB is. Are these operations which are (relatively) rare in the normal deployments of VoltDB? Did these optimizations make more of a difference in previous versions of VoltDB?
- jhugg 10y ago
  The stale and uncommitted reads was an optimization. It seemed like a harmless optimization to make at the time, and we were wrong. v6.4 allows you to pick strong serializability (default) or the old behavior at startup. For 100% read workloads, the impact to maximum throughput can be significant. That's a pretty uncommon workload for us though. It's likely a single-digit percentage problem at 50% reads. Two nice things about VoltDB users: 1) They often are very write heavy, 99% read-write transactions isn't uncommon. 2) Few are using anywhere near the maximum throughput for their machines. Most size their clusters for appropriate storage and redundancy, not throughput. The lost write issues weren't optimizations, just implementation bugs. Sigh.

by jhugg 10y ago

jasonjackson 10y ago

I hadn't heard of VoltDB until now, but have been following jepsen for a while. I wish more project did what you did. Congrats on the increased reliability.

hobs 10y ago

All software has bugs, not everyone pays independent and respected members of the community to root them out. Kudos.

swordswinger12 10y ago

Why roll your own consensus algorithm?

jhugg 10y ago
Historical reasons mostly. First, Raft wasn't a thing when we started. Second, the consensus algorithm we use today is used for cluster membership, schema changes and other agreement purposes, but not for the main transactional path. However, in versions before 3.0 it was. At the time, we wanted to make really different tradeoffs than something like Paxos. Namely, we wanted to be able to run with two copies, rather than the three that Paxos/Raft require. We also wanted crazy throughput and low performance variation, which are hard things to get with Paxos/Raft. The main tradeoff is that our system doesn't work well across high-latency links, which is ok with us. Ultimately we added a second layer protocol for transactions in 3.0, and this made the transaction path not dependent on clock skew. This layer actually looks a lot like Raft, but does differ in some key ways. We may even someday switch to Raft for the core agreement stuff, but there's a lot of work to be done for that to happen. This link has more info about how transactions work in VoltDB: https://voltdb.com/sites/default/files/tn-transactions.pdf https://voltdb.com/sites/default/files/tn-transactions.pdf
- wmccullough 10y ago
  This is a great explanation. You can see the thought process and care that went into the design.
- sseveran 10y ago
  Did you make formal specification of your algorithms in something like TLA+?
  - jhugg 10y ago
    That would be cool. Kyle Kingsbury started something like this (in Clojure) as part of this project (https://github.com/jepsen-io/voltdb/tree/master/replication-model https://github.com/jepsen-io/voltdb/tree/master/replication-...), but it's a large task and would probably take more than a few days. We have done some initial work on TLA+, but it may be a while to finish anything. If someone wants to help, there's probably a publishable paper in it.
    - sseveran 10y ago
      Cool. I am always a little wary of distributed algorithms that don't go through that level of validation. But I recognize that few commercial firms do that, although AWS is a stand out. I had the same criticism of ZooKeeper for a long time. Glad to hear you have at least considered it.

heavenlyhash 10y ago

You and your team's resolve and success in tacking these bugs is awesome. It's interesting to read about some of the relatively "unusual" choices VoltDB has made as well. Like this one: > network partitions cause minority nodes to shut down permanently; operator intervention is required to restore full service ...combined with requiring majorities of current rather than original clusters to continue. This is, for human operational interactions, really interesting. There's a big history of self-healing systems getting a little carried away and masking or sometimes even triggering bigger issues. Making signoff to rejoin nodes a required operation seems like it makes a whole category of "flapping" problems a non-issue. Kudos for making these "unusual" choices; this is a great horizon to explore :)

jhugg 10y ago
We have some users who have automated rejoin. This might actually make sense in the cloud, where you just spin up another pristine instance. Personally, when a node fails, I still want to try to understand why before I take the next operational step. Thanks for your words, BTW.

baq 10y ago

why would i use voltdb? serious question. this is the first time i hear about voltdb. i mean, i can use a postgres or mssql cluster for SQL needs and/or i can use riak/... for kv and/or cassandra/... for column indexed storage and/or ... etc. why should i look at voltdb?

jhugg 10y ago
Well, one thing we can say definitively today is that VoltDB offers strong serializability, when almost no other clustered systems do, and when they do, they are slow. But my more typical answer is the combination of throughput and transactional logic. No other system does both as well as VoltDB. This comes up a lot in policy enforcement, fraud detection, online gaming, ad-tech, billing-support and more. Here are some blog posts that might help: https://voltdb.com/blog/winning-now-and-future-where-voltdb-shines https://voltdb.com/blog/winning-now-and-future-where-voltdb-... https://voltdb.com/blog/apps-need-acid https://voltdb.com/blog/apps-need-acid https://voltdb.com/blog/call-center-example-integrating-processing-and-state-make-streaming-problems-simple-solve https://voltdb.com/blog/call-center-example-integrating-proc... This video's not bad: https://voltdb.com/resources/video/transactional-streaming-if-you-can-compute-it-you-can-probably-stream-it https://voltdb.com/resources/video/transactional-streaming-i...
ruraljuror 10y ago
If you like audio, Stonebreaker's interview on Software Engineering Radio is opinionated, excellent, and I think it will answer that question: http://www.se-radio.net/2013/12/episode-199-michael-stonebraker/ http://www.se-radio.net/2013/12/episode-199-michael-stonebra...

willvarfar 10y ago

Why does VoltDB require that transactions are written as stored procedures? Why can't a sequence of statements between a begin and end be transparently treated as though they were the body of a stored procedure?

jhugg 10y ago
You can actually send a bunch of SQL statements together, separated by semicolons to VoltDB and it will be treated as a transaction. In practice, this isn't always all that useful without a real stored procedure language. Supporting other ways to make procedures is something we've always planned and at some point we will get to it. I personally like javascript and lua.

cdcarter 10y ago

It seems like several of the issues aphyr found were due to optimizations the product was making. Fixing them in a point release suggests they weren't huge changes with massive ramifications. I don't know what the key benchmarks for software like VoltDB is. Are these operations which are (relatively) rare in the normal deployments of VoltDB? Did these optimizations make more of a difference in previous versions of VoltDB?

jhugg 10y ago
The stale and uncommitted reads was an optimization. It seemed like a harmless optimization to make at the time, and we were wrong. v6.4 allows you to pick strong serializability (default) or the old behavior at startup. For 100% read workloads, the impact to maximum throughput can be significant. That's a pretty uncommon workload for us though. It's likely a single-digit percentage problem at 50% reads. Two nice things about VoltDB users: 1) They often are very write heavy, 99% read-write transactions isn't uncommon. 2) Few are using anywhere near the maximum throughput for their machines. Most size their clusters for appropriate storage and redundancy, not throughput. The lost write issues weren't optimizations, just implementation bugs. Sigh.