7 ms·

Jepsen: YugaByte DB 1.1.9

115 points by aphyr 8y ago

shin_lao 8y ago
If they rely on clocks, why don't they use PTP? Am I missing something?
- aphyr 8y ago
  I think the YB team members are probably best equipped to talk about this, but I can note that while some databases do build their own clock synchronization protocol, many prefer to let the OS handle clocks. For one thing, clock sync is surprisingly tricky to do well, so it makes sense to write daemons that do it well once and be able to re-use them in lots of contexts. There's also the question of HW support: in theory, datacenter and hardware providers could do better than pure-software time synchronization by, say, offering dedicated physical links to a local atomic + GPS clock ensemble. AWS TimeSync is a step in this direction, and I wouldn't be surprised if we see more accurate clocks in the future. There are still tons of caveats with this idea--Linux and most database software ain't realtime, for starters--but you can imagine a world in which clock errors are sufficiently bounded and infrequent that they no longer represent the most urgent threat to safety. That's ultimately a quantitative risk assessment. My suspicion is that DB vendors like YugaByte and CockroachDB are making a strategic bet that although clocks right now are pretty terrible, they won't be that way forever. I'd like to see more rigorous measurement on this front, because while I've got plenty of anecdotes, I don't think we have a broad statistical picture of how bad typical clocks are, and whether they're improving.
  - shin_lao 8y ago
    My comment was that PTP is a much better protocol than NTP, and in the doc they only talk about NTP: https://docs.yugabyte.com/latest/deploy/checklist/#clock-synchronization https://docs.yugabyte.com/latest/deploy/checklist/#clock-syn... https://en.wikipedia.org/wiki/Precision_Time_Protocol https://en.wikipedia.org/wiki/Precision_Time_Protocol
    - aphyr 8y ago
      Yeah, you can deploy any type of NTP-alike time synchronization you like. That could be ntpd, chrony, hypervisor-guest synchronization tools, etc etc.
    - rkarthik007 8y ago
      Hi @shin_lao As @aphyr had mentioned, any NTP-alike system would work. We can update the docs to mention PTP, we do work with AWS Time Sync as well (which uses Chrony).
    - sllabres 8y ago
      https://en.wikipedia.org/wiki/The_White_Rabbit_Project https://en.wikipedia.org/wiki/The_White_Rabbit_Project
  - williamallthing 8y ago
    I'm not an expert, but don’t all modern databases suffer from these issues? I thought they all depended on clock sync like Google Spanner.
    - aphyr 8y ago
      In short, no: many transactional databases don't rely on clocks for safety. I'm going to speak in broad terms here--there's a lot of nuance and special cases that we can dig into, but I'd like to keep this accessible: You can use CRDTs, and other commutative data structures, to obtain totally-available replicated objects across wide area networks. Systems like Riak do this. CRDTs can't express some types of computation safely, though! For instance, you can't do something like a minimum-balance constraint, ensuring that an account always contains $25 or more, if you allow both deposits and withdrawals, in a commutative system. Why? Because order matters! Deposit, withdraw is different than withdraw, deposit, in terms of their intermediate states. For order, you can use a consensus mechanism, like ZAB (Zookeeper), Paxos (Riak SC, Cassandra LWT), or Raft (etcd, consul) to replicate arbitrary state machines without any clock dependence at all. These systems require at least one round trip to establish consensus, and their guarantees only apply within the consensus system itself. What if you have multiple consensus groups? Say, one per shard? Then you need a protocol to coordinate transactions on top of that. You can execute an atomic commit protocol for cross-shard transactions, perhaps using a consensus system. Or you can use a protocol like Calvin to obtain serializability (or stronger) across shards without relying on clocks. That's what FaunaDB does. That adds a round-trip, but if you're clever, you may only have to pay that round-trip cost between different datacenters once. Another tactic is to exploit well-synchronized clocks to obtain consistent views across independent consensus groups. You can use this technique to (theoretically) reduce the number of round trips a transaction costs, and there are different ways to balance whether you pay increased latency on read or write transactions. Spanner, CockroachDB, and YugaByte DB all take this approach, with different tradeoffs. Spanner is backed by custom hardware and carefully designed software, to obtain tight bounds on clock error. CockroachDB and YugaByte DB leave that problem to you, the operator. Often, a database uses a stronger replication mechanism inside a datacenter, but when it comes to replicating between datacenters, backs off to a weaker strategy which doesn't offer the same safety invariants.
the_duke 8y ago
First time I heard about them. Seems to be another distributed SQL (aka 'newsql') alternative to TiDB and CockroachDB. Based on RocksDB (like Cockroach) with a custom distributed key/val layer and and additional SQL layer on top. PostgreSQL protocol compatible. OS with Apache license. Seems interesting. (when ignoring the "planet scale SQL" marketing speak... [1]) [1] https://www.yugabyte.com/planet-scale-sql/ https://www.yugabyte.com/planet-scale-sql/
- marknadal 8y ago
  As an expert in the DB space, I'm extraordinarily cynical. But their willingness to license it as truly open, Apache-style, instantly is a big win-over. I'm a competitor, but I can tell these guys/gals are genuine in their efforts. We need more people, teams, and DBs like YugaByteDB in the world. Thank you for your efforts.
  - the_duke 8y ago
    The question is, how many of these very similar databases can the market support? The field is getting crowded, and the database market is already quite competitive as it is, without these new competitors. There just are not that many use cases where a larger Postgres/MySQL instance with one or two replicas is insufficient. From a user perspective, I'd much rather have one or two successful companies where I can be reasonably certain that the product will be maintained in 5 years than too much competition.
    - marknadal 8y ago
      This is why the license matters. Even if the commercial/AGPL/GPL ones "win" for a few years, they won't be able to compete with these more Open licensed DBs when they catch up. So at any given moment, too many DBs may be annoying, but for the long term and long game it is important there is this type of competition & research going on. Although, I absolutely agree, when it comes to Master-Slave based systems (I've been very vocal in criticizing them) that market is drying up to some very limited use cases (banking, etc.). 99%+ of use cases will be Strong Eventual Consistency and CRDT with distributed or decentralized/P2P tools. Some really old, yet very relevant, thoughts on this subject: https://hackernoon.com/the-implications-of-rethinkdb-and-parse-shutdowns-c076460058f7 https://hackernoon.com/the-implications-of-rethinkdb-and-par...
truth_seeker 8y ago
Is it also tested against ScyllaDB ? ScyllaDB could be up to 10x performant than Cassandra as backend storage.
- aphyr 8y ago
  Jepsen is not a performance test; we verify safety. I haven't looked at ScyllaDB personally, but you can read about Scylla's own work testing their database here [1], and see some of the issues they found here [2]. [1]: https://www.scylladb.com/2016/02/11/jepsen-testing/ https://www.scylladb.com/2016/02/11/jepsen-testing/ [2]: https://github.com/scylladb/scylla/issues?utf8=%E2%9C%93&q=is%3Aissue+jepsen+ https://github.com/scylladb/scylla/issues?utf8=%E2%9C%93&q=i...
  - truth_seeker 8y ago
    I never said Jespen is about performance. My comment was only about considering ScyllaDB as backend choice. Thanks for the links anyway.
    - sidch 8y ago
      YugaByte product manager here. The YCQL API which passes Jepsen has its roots in Cassandra Query Language but does not use Cassandra as its backend store. It’s backend store is DocDB, which is a Google Spanner-inspired distributed document store.
- manigandham 8y ago
  Yugabyte doesn't use Cassandra, it's a custom-built database using RocksDB as a key/value layer and with an internal document-store representation. It offers access to the data in multiple interfaces: Redis, Cassandra CQL and now PostgreSQL.
  - truth_seeker 8y ago
    Thanks for clarification.
spullara 8y ago
Since it doesn't support serializable transactions I'm not sure why FoundationDB would be mentioned as a comparison in the write up. The operations it does support seem to set the bar pretty low as to what to test. edit: good reply by the founder of YugaByte but for some reason the comment is dead. I have noticed that when founders don't have an account on here and then something comes up where they need to reply their comments are often deaded.
- mbautin 8y ago
  We have run a significant number of useful/practical tests via Jepsen, that only need snapshot isolation level. The tests that were run included a single-key counter test, set tests with and without a secondary index, a "long fork" test ensuring that the order of operations is the same when observed by different clients, and a bank test verifying that the total balance of multiple accounts stays the same when cross-shard transactions transfer funds between pairs of accounts. These tests were run under a variety of failure modes, including different types of network partitions and clock skew. Also, snapshot isolation covers a very large spectrum of practical uses cases, including secondary indexes, for building real-world applications. Having said that, we have recently added support for serializable isolation level to YugaByte DB, and we will be adding tests to the Jepsen suite for that in near term. Regards, Mikhail (a co-founder at YugaByte)
- gigatexal 8y ago
  We use cockroachDB in production and before that we were on MySQL and as of yet we don’t have a specific usecase where we use serializable transactions. Snapshot isolation or even read committed is just fine. So I don’t think it’s absolutely necessary
  - gigatexal 8y ago
    To be clear there’s no way around serializable transactions in cockroachDB. We have had to adapt our monolith to it (we’re thinking of ways to make it more nimble by breaking out services etc). But the point I was making was that we had MySQL for a while and never ran into issues with its isolation levels until it stopped scaling. Instead of vitess or some other MySQL system we went with cockroach after finding vitess didn’t fit us — too complicated and too many moving parts. CockroachDB just works. Also moving to k8s adds complexity too for a monolith built and run on VMs. But so far so good. Cockroach runs fast and is performant given production queries. And ops is happy because it self heals.
robterrell 8y ago
Not a comment on YugeByte, but... I love it when a new Jepsen report get released. Kyle Kingsbury has single-handedly raised the bar on an entire industry. (Well, not single-handedly anymore, but still.)
- Jupe 8y ago
  Couldn't agree more. There are 3 sources of information regarding database serializability/linearizability: 1. Marketing material (mostly useless) 2. Individual projects/post-mortems (50/50 here; some just mis-use the technology from the get-go, others have valid feedback, but it's tough to determine when either applies) 3. Jepsen Tests (which is more like independently verifiable science) Sure, you can decide that your social-media solution has no need for consistency (or even durability!) - but in my experience, most solutions don't have that flexibility.
danburkert 8y ago
Does YugaByte still use the Raft and HybridTime implementations from Apache Kudu? If so, how relevant are these results for Kudu?
- acubel23 8y ago
  Yes, it does. At the core, the raft implementation is still based on kudu's. But, these areas have been worked on actively so the implementations might has diverged a little. May be worth looking through the individual issues to see what applies and what doesn't: https://github.com/YugaByte/yugabyte-db/projects/11 https://github.com/YugaByte/yugabyte-db/projects/11
  - deleted 8y ago
    [deleted]
- kmuthukk 8y ago
  I wanted to add a few details to the previous reply. While the Raft/HybridTime implementation has its roots in Apache Kudu the results will NOT be quite applicable to Kudu. Aside from the fact that the code base has evolved/diverged over the 3+ years, there are key/relevant areas (ones very relevant to these Jepsen tests) where YugaByte DB has added capabilities or follows a different design than Kudu. For example: -- Leader Leases: YugaByte DB doesn't use Raft consensus for reads. Instead, we have implemented "leader leases" to ensure safety in allowing reads to be served from a tablet's Raft leader. -- Distributed/Multi-Shard Transactions: YugaByte DB uses a home grown (https://docs.yugabyte.com/latest/architecture/transactions/transactional-io-path/ https://docs.yugabyte.com/latest/architecture/transactions/t...) protocol based on two-phase commit across multiple Raft groups. Capabilities like secondary indexes, multi-row updates use multi-shard transactions. -- Allowing online/dynamic Raft membership changes so that tablets can be moved (such as for load-balancing to new nodes). regards Kannan (Co-founder @ YugaByte)
  - mpercy 8y ago
    FWIW, we implemented dynamic consensus membership change in Kudu way back in 2015 (https://github.com/apache/kudu/commit/535dae https://github.com/apache/kudu/commit/535dae) but presumably that was after the fork. We still haven't implemented leader leases or distributed transactions in Kudu though due to prioritizing other features. It's very cool that you have implemented those consistency features.

115 points by aphyr 8y ago

shin_lao 8y ago

If they rely on clocks, why don't they use PTP? Am I missing something?

aphyr 8y ago
I think the YB team members are probably best equipped to talk about this, but I can note that while some databases do build their own clock synchronization protocol, many prefer to let the OS handle clocks. For one thing, clock sync is surprisingly tricky to do well, so it makes sense to write daemons that do it well once and be able to re-use them in lots of contexts. There's also the question of HW support: in theory, datacenter and hardware providers could do better than pure-software time synchronization by, say, offering dedicated physical links to a local atomic + GPS clock ensemble. AWS TimeSync is a step in this direction, and I wouldn't be surprised if we see more accurate clocks in the future. There are still tons of caveats with this idea--Linux and most database software ain't realtime, for starters--but you can imagine a world in which clock errors are sufficiently bounded and infrequent that they no longer represent the most urgent threat to safety. That's ultimately a quantitative risk assessment. My suspicion is that DB vendors like YugaByte and CockroachDB are making a strategic bet that although clocks right now are pretty terrible, they won't be that way forever. I'd like to see more rigorous measurement on this front, because while I've got plenty of anecdotes, I don't think we have a broad statistical picture of how bad typical clocks are, and whether they're improving.
- shin_lao 8y ago
  My comment was that PTP is a much better protocol than NTP, and in the doc they only talk about NTP: https://docs.yugabyte.com/latest/deploy/checklist/#clock-synchronization https://docs.yugabyte.com/latest/deploy/checklist/#clock-syn... https://en.wikipedia.org/wiki/Precision_Time_Protocol https://en.wikipedia.org/wiki/Precision_Time_Protocol
  - aphyr 8y ago
    Yeah, you can deploy any type of NTP-alike time synchronization you like. That could be ntpd, chrony, hypervisor-guest synchronization tools, etc etc.
  - rkarthik007 8y ago
    Hi @shin_lao As @aphyr had mentioned, any NTP-alike system would work. We can update the docs to mention PTP, we do work with AWS Time Sync as well (which uses Chrony).
  - sllabres 8y ago
    https://en.wikipedia.org/wiki/The_White_Rabbit_Project https://en.wikipedia.org/wiki/The_White_Rabbit_Project
- williamallthing 8y ago
  I'm not an expert, but don’t all modern databases suffer from these issues? I thought they all depended on clock sync like Google Spanner.
  - aphyr 8y ago
    In short, no: many transactional databases don't rely on clocks for safety. I'm going to speak in broad terms here--there's a lot of nuance and special cases that we can dig into, but I'd like to keep this accessible: You can use CRDTs, and other commutative data structures, to obtain totally-available replicated objects across wide area networks. Systems like Riak do this. CRDTs can't express some types of computation safely, though! For instance, you can't do something like a minimum-balance constraint, ensuring that an account always contains $25 or more, if you allow both deposits and withdrawals, in a commutative system. Why? Because order matters! Deposit, withdraw is different than withdraw, deposit, in terms of their intermediate states. For order, you can use a consensus mechanism, like ZAB (Zookeeper), Paxos (Riak SC, Cassandra LWT), or Raft (etcd, consul) to replicate arbitrary state machines without any clock dependence at all. These systems require at least one round trip to establish consensus, and their guarantees only apply within the consensus system itself. What if you have multiple consensus groups? Say, one per shard? Then you need a protocol to coordinate transactions on top of that. You can execute an atomic commit protocol for cross-shard transactions, perhaps using a consensus system. Or you can use a protocol like Calvin to obtain serializability (or stronger) across shards without relying on clocks. That's what FaunaDB does. That adds a round-trip, but if you're clever, you may only have to pay that round-trip cost between different datacenters once. Another tactic is to exploit well-synchronized clocks to obtain consistent views across independent consensus groups. You can use this technique to (theoretically) reduce the number of round trips a transaction costs, and there are different ways to balance whether you pay increased latency on read or write transactions. Spanner, CockroachDB, and YugaByte DB all take this approach, with different tradeoffs. Spanner is backed by custom hardware and carefully designed software, to obtain tight bounds on clock error. CockroachDB and YugaByte DB leave that problem to you, the operator. Often, a database uses a stronger replication mechanism inside a datacenter, but when it comes to replicating between datacenters, backs off to a weaker strategy which doesn't offer the same safety invariants.

the_duke 8y ago

First time I heard about them. Seems to be another distributed SQL (aka 'newsql') alternative to TiDB and CockroachDB. Based on RocksDB (like Cockroach) with a custom distributed key/val layer and and additional SQL layer on top. PostgreSQL protocol compatible. OS with Apache license. Seems interesting. (when ignoring the "planet scale SQL" marketing speak... [1]) [1] https://www.yugabyte.com/planet-scale-sql/ https://www.yugabyte.com/planet-scale-sql/

marknadal 8y ago
As an expert in the DB space, I'm extraordinarily cynical. But their willingness to license it as truly open, Apache-style, instantly is a big win-over. I'm a competitor, but I can tell these guys/gals are genuine in their efforts. We need more people, teams, and DBs like YugaByteDB in the world. Thank you for your efforts.
- the_duke 8y ago
  The question is, how many of these very similar databases can the market support? The field is getting crowded, and the database market is already quite competitive as it is, without these new competitors. There just are not that many use cases where a larger Postgres/MySQL instance with one or two replicas is insufficient. From a user perspective, I'd much rather have one or two successful companies where I can be reasonably certain that the product will be maintained in 5 years than too much competition.
  - marknadal 8y ago
    This is why the license matters. Even if the commercial/AGPL/GPL ones "win" for a few years, they won't be able to compete with these more Open licensed DBs when they catch up. So at any given moment, too many DBs may be annoying, but for the long term and long game it is important there is this type of competition & research going on. Although, I absolutely agree, when it comes to Master-Slave based systems (I've been very vocal in criticizing them) that market is drying up to some very limited use cases (banking, etc.). 99%+ of use cases will be Strong Eventual Consistency and CRDT with distributed or decentralized/P2P tools. Some really old, yet very relevant, thoughts on this subject: https://hackernoon.com/the-implications-of-rethinkdb-and-parse-shutdowns-c076460058f7 https://hackernoon.com/the-implications-of-rethinkdb-and-par...

truth_seeker 8y ago

Is it also tested against ScyllaDB ? ScyllaDB could be up to 10x performant than Cassandra as backend storage.

aphyr 8y ago
Jepsen is not a performance test; we verify safety. I haven't looked at ScyllaDB personally, but you can read about Scylla's own work testing their database here [1], and see some of the issues they found here [2]. [1]: https://www.scylladb.com/2016/02/11/jepsen-testing/ https://www.scylladb.com/2016/02/11/jepsen-testing/ [2]: https://github.com/scylladb/scylla/issues?utf8=%E2%9C%93&q=is%3Aissue+jepsen+ https://github.com/scylladb/scylla/issues?utf8=%E2%9C%93&q=i...
- truth_seeker 8y ago
  I never said Jespen is about performance. My comment was only about considering ScyllaDB as backend choice. Thanks for the links anyway.
  - sidch 8y ago
    YugaByte product manager here. The YCQL API which passes Jepsen has its roots in Cassandra Query Language but does not use Cassandra as its backend store. It’s backend store is DocDB, which is a Google Spanner-inspired distributed document store.
manigandham 8y ago
Yugabyte doesn't use Cassandra, it's a custom-built database using RocksDB as a key/value layer and with an internal document-store representation. It offers access to the data in multiple interfaces: Redis, Cassandra CQL and now PostgreSQL.
- truth_seeker 8y ago
  Thanks for clarification.

spullara 8y ago

Since it doesn't support serializable transactions I'm not sure why FoundationDB would be mentioned as a comparison in the write up. The operations it does support seem to set the bar pretty low as to what to test. edit: good reply by the founder of YugaByte but for some reason the comment is dead. I have noticed that when founders don't have an account on here and then something comes up where they need to reply their comments are often deaded.

mbautin 8y ago
We have run a significant number of useful/practical tests via Jepsen, that only need snapshot isolation level. The tests that were run included a single-key counter test, set tests with and without a secondary index, a "long fork" test ensuring that the order of operations is the same when observed by different clients, and a bank test verifying that the total balance of multiple accounts stays the same when cross-shard transactions transfer funds between pairs of accounts. These tests were run under a variety of failure modes, including different types of network partitions and clock skew. Also, snapshot isolation covers a very large spectrum of practical uses cases, including secondary indexes, for building real-world applications. Having said that, we have recently added support for serializable isolation level to YugaByte DB, and we will be adding tests to the Jepsen suite for that in near term. Regards, Mikhail (a co-founder at YugaByte)
gigatexal 8y ago
We use cockroachDB in production and before that we were on MySQL and as of yet we don’t have a specific usecase where we use serializable transactions. Snapshot isolation or even read committed is just fine. So I don’t think it’s absolutely necessary
- gigatexal 8y ago
  To be clear there’s no way around serializable transactions in cockroachDB. We have had to adapt our monolith to it (we’re thinking of ways to make it more nimble by breaking out services etc). But the point I was making was that we had MySQL for a while and never ran into issues with its isolation levels until it stopped scaling. Instead of vitess or some other MySQL system we went with cockroach after finding vitess didn’t fit us — too complicated and too many moving parts. CockroachDB just works. Also moving to k8s adds complexity too for a monolith built and run on VMs. But so far so good. Cockroach runs fast and is performant given production queries. And ops is happy because it self heals.

robterrell 8y ago

Not a comment on YugeByte, but... I love it when a new Jepsen report get released. Kyle Kingsbury has single-handedly raised the bar on an entire industry. (Well, not single-handedly anymore, but still.)

Jupe 8y ago
Couldn't agree more. There are 3 sources of information regarding database serializability/linearizability: 1. Marketing material (mostly useless) 2. Individual projects/post-mortems (50/50 here; some just mis-use the technology from the get-go, others have valid feedback, but it's tough to determine when either applies) 3. Jepsen Tests (which is more like independently verifiable science) Sure, you can decide that your social-media solution has no need for consistency (or even durability!) - but in my experience, most solutions don't have that flexibility.

danburkert 8y ago

Does YugaByte still use the Raft and HybridTime implementations from Apache Kudu? If so, how relevant are these results for Kudu?

acubel23 8y ago
Yes, it does. At the core, the raft implementation is still based on kudu's. But, these areas have been worked on actively so the implementations might has diverged a little. May be worth looking through the individual issues to see what applies and what doesn't: https://github.com/YugaByte/yugabyte-db/projects/11 https://github.com/YugaByte/yugabyte-db/projects/11
- deleted 8y ago
  [deleted]
kmuthukk 8y ago
I wanted to add a few details to the previous reply. While the Raft/HybridTime implementation has its roots in Apache Kudu the results will NOT be quite applicable to Kudu. Aside from the fact that the code base has evolved/diverged over the 3+ years, there are key/relevant areas (ones very relevant to these Jepsen tests) where YugaByte DB has added capabilities or follows a different design than Kudu. For example: -- Leader Leases: YugaByte DB doesn't use Raft consensus for reads. Instead, we have implemented "leader leases" to ensure safety in allowing reads to be served from a tablet's Raft leader. -- Distributed/Multi-Shard Transactions: YugaByte DB uses a home grown (https://docs.yugabyte.com/latest/architecture/transactions/transactional-io-path/ https://docs.yugabyte.com/latest/architecture/transactions/t...) protocol based on two-phase commit across multiple Raft groups. Capabilities like secondary indexes, multi-row updates use multi-shard transactions. -- Allowing online/dynamic Raft membership changes so that tablets can be moved (such as for load-balancing to new nodes). regards Kannan (Co-founder @ YugaByte)
- mpercy 8y ago
  FWIW, we implemented dynamic consensus membership change in Kudu way back in 2015 (https://github.com/apache/kudu/commit/535dae https://github.com/apache/kudu/commit/535dae) but presumably that was after the fork. We still haven't implemented leader leases or distributed transactions in Kudu though due to prioritizing other features. It's very cool that you have implemented those consistency features.