7 ms·

Jepsen: YugaByte DB 1.3.1

106 points by aphyr 7y ago

darshanime 7y ago
Great. I am not sure about using yugabytedb now.
- rkarthik007 7y ago
  Would much appreciate some constructive criticism! - CTO/co-founder
- YawningAngel 7y ago
  All distributed databases (and, frankly, all software) have bugs. At leat with Yugabyte you have some idea of what the bugs are because they tested them and published the results.
- aphyr 7y ago
  Keep in mind that this analysis focuses on features which are still in beta; while YugaByte DB advertises SQL features throughout their marketing, the documentation does state that SQL isn't production-ready yet. We should expect bugs at this stage! :-)
pdeva1 7y ago
seems like yugabyte, like postgres, also spawns a process per connection; thus requiring use of external connection pool
- rkarthik007 7y ago
  Very astute observation! Two things: * We implemented a feature to share memory between these connection handling processes so that makes it a bit more efficient * Longer term we are thinking of switching to a thread based model. This is how our other API YCQL works. (cto / co-founder)
  - ralusek 7y ago
    I think this is a very important thing to prioritize for a DB of this nature. Postgres/RDBMS connections in general are an extreme weak spot for containerized/serverless applications. DynamoDB/ElasticSearch's approach of not needing to establish/maintain a connection seems like the ideal solution. I think for applications that would utilize your DB, a minor hit to the speed of an individual request is well worth it as compared to being able to scale up the amount of simultaneous requests.
    - rkarthik007 7y ago
      Yes absolutely spot on! Peeling the onion one more layer, there are two underlying features that are required: 1) Move to a threaded model to be able to scale instantaneously. This is already the case with YCQL, we're planning on doing this for YSQL. 2) There needs to be a change to the client drivers in order to become smarter to deal with how the app connects to the DB. In fact, we're working on enabling this for the Spring (Java) ecosystem by enhancing the JDSB driver (still in the early stages): https://github.com/YugaByte/ybjdbc https://github.com/YugaByte/ybjdbc (cto/co-founder)
      - pdeva1 7y ago
        what does the JDBC driver has to do with thread per connection in the database? jdbc doesnt care if you use threads or processes for the connection
        rkarthik007 7y ago
        True for a single node. However, there will be multiple IP address/connections anyway, since YugaByte DB is a distributed DB and it runs across multiple nodes. JDBC drivers connect to only one node - so to provide true connection scaling, the client side would need to become aware of a multi-node cluster and do connection pooling across these. Of course there are other issues also that need to get solved that are slightly related: * The node given to the JDBC driver could be down, or nodes could get added/removed over time. This makes "discovery of nodes" a problem. * We could use a random round robin strategy to connect to nodes - where client connects to a random cluster node which internally connects to the appropriate node. This would result in an increased latency, and also an increase in the net number of connections needed. These may not matter if the load balancer is smart like in the case of Kubernetes (and where the extra hop becomes mandatory as well). But for non-k8s deployments, these help.
dougwbrunton 7y ago
I cannot recall seeing a public Jensen report that didn’t result in better software. Kudos to the Jepsen team and their clients.
pdeva1 7y ago
the db doesnt guarantee linearizability (assuming clock skew). even the post says stale counters values can be read. Doesn't that mean that following use cases are not possible: 1. keeping count of page views 2. vote count per ticket/post 3. like/dislike counts per entity a reason to use highly scalabale db like Yuga would be to keep such counts.
- aphyr 7y ago
  All those should be possible; you just might not observe the state of the counter as it was a few milliseconds(?) ago, rather than what it might be right now. If you're dealing with extremely frequent updates where the chances of observing that sort of anomaly are more common, chances are the precise value of the counter doesn't matter so much, and an occasional off-by-one or off-by-two isn't the end of the world. YMMV, of course. :)
  - pdeva1 7y ago
    >you just might not observe the state of the counter as it was a few milliseconds did you mean 'might observe' instead of 'might not'? stale read would result in the older value being read. i believe the anomaly would then occur depending on how close the 2 events are in time, regardless of frequency. in that case, precise counts can be needed even with low frequency. say you are a restaurant with 1 burger left. 2 folks order the same time, but you decrement the count by 1 only. also anything involving monetary values, you definitely don't want to read stale values ever.
    - grogers 7y ago
      If you earmarked the last burger to multiple people, it would violate serializability, not linearizability. So you should be good on that. Violating linearizability would look like this: with no burgers left, the kitchen cooks a burger, marks it as available in the DB and then shouts that it's ready. A customer hears them shout and immediately tries to order it, but fails because the customer's transaction happened at an earlier logical time than the kitchen's. Kyle is saying there are still relatively small time bounds on such violations (usually?) so it's probably not the biggest problem your restaurant has to worry about (just apologize and have the customer try again). Dealing with money is similarly not a problem. ACH takes much longer to clear than the uncertainty window of linearizability violations. The client probably won't be able to tell the difference.

106 points by aphyr 7y ago

darshanime 7y ago

Great. I am not sure about using yugabytedb now.

rkarthik007 7y ago
Would much appreciate some constructive criticism! - CTO/co-founder
YawningAngel 7y ago
All distributed databases (and, frankly, all software) have bugs. At leat with Yugabyte you have some idea of what the bugs are because they tested them and published the results.
aphyr 7y ago
Keep in mind that this analysis focuses on features which are still in beta; while YugaByte DB advertises SQL features throughout their marketing, the documentation does state that SQL isn't production-ready yet. We should expect bugs at this stage! :-)

pdeva1 7y ago

seems like yugabyte, like postgres, also spawns a process per connection; thus requiring use of external connection pool

rkarthik007 7y ago
Very astute observation! Two things: * We implemented a feature to share memory between these connection handling processes so that makes it a bit more efficient * Longer term we are thinking of switching to a thread based model. This is how our other API YCQL works. (cto / co-founder)
- ralusek 7y ago
  I think this is a very important thing to prioritize for a DB of this nature. Postgres/RDBMS connections in general are an extreme weak spot for containerized/serverless applications. DynamoDB/ElasticSearch's approach of not needing to establish/maintain a connection seems like the ideal solution. I think for applications that would utilize your DB, a minor hit to the speed of an individual request is well worth it as compared to being able to scale up the amount of simultaneous requests.
  - rkarthik007 7y ago
    Yes absolutely spot on! Peeling the onion one more layer, there are two underlying features that are required: 1) Move to a threaded model to be able to scale instantaneously. This is already the case with YCQL, we're planning on doing this for YSQL. 2) There needs to be a change to the client drivers in order to become smarter to deal with how the app connects to the DB. In fact, we're working on enabling this for the Spring (Java) ecosystem by enhancing the JDSB driver (still in the early stages): https://github.com/YugaByte/ybjdbc https://github.com/YugaByte/ybjdbc (cto/co-founder)
    - pdeva1 7y ago
      what does the JDBC driver has to do with thread per connection in the database? jdbc doesnt care if you use threads or processes for the connection
      - rkarthik007 7y ago
        True for a single node. However, there will be multiple IP address/connections anyway, since YugaByte DB is a distributed DB and it runs across multiple nodes. JDBC drivers connect to only one node - so to provide true connection scaling, the client side would need to become aware of a multi-node cluster and do connection pooling across these. Of course there are other issues also that need to get solved that are slightly related: * The node given to the JDBC driver could be down, or nodes could get added/removed over time. This makes "discovery of nodes" a problem. * We could use a random round robin strategy to connect to nodes - where client connects to a random cluster node which internally connects to the appropriate node. This would result in an increased latency, and also an increase in the net number of connections needed. These may not matter if the load balancer is smart like in the case of Kubernetes (and where the extra hop becomes mandatory as well). But for non-k8s deployments, these help.

dougwbrunton 7y ago

I cannot recall seeing a public Jensen report that didn’t result in better software. Kudos to the Jepsen team and their clients.

pdeva1 7y ago

the db doesnt guarantee linearizability (assuming clock skew). even the post says stale counters values can be read. Doesn't that mean that following use cases are not possible: 1. keeping count of page views 2. vote count per ticket/post 3. like/dislike counts per entity a reason to use highly scalabale db like Yuga would be to keep such counts.

aphyr 7y ago
All those should be possible; you just might not observe the state of the counter as it was a few milliseconds(?) ago, rather than what it might be right now. If you're dealing with extremely frequent updates where the chances of observing that sort of anomaly are more common, chances are the precise value of the counter doesn't matter so much, and an occasional off-by-one or off-by-two isn't the end of the world. YMMV, of course. :)
- pdeva1 7y ago
  >you just might not observe the state of the counter as it was a few milliseconds did you mean 'might observe' instead of 'might not'? stale read would result in the older value being read. i believe the anomaly would then occur depending on how close the 2 events are in time, regardless of frequency. in that case, precise counts can be needed even with low frequency. say you are a restaurant with 1 burger left. 2 folks order the same time, but you decrement the count by 1 only. also anything involving monetary values, you definitely don't want to read stale values ever.
  - grogers 7y ago
    If you earmarked the last burger to multiple people, it would violate serializability, not linearizability. So you should be good on that. Violating linearizability would look like this: with no burgers left, the kitchen cooks a burger, marks it as available in the DB and then shouts that it's ready. A customer hears them shout and immediately tries to order it, but fails because the customer's transaction happened at an earlier logical time than the kitchen's. Kyle is saying there are still relatively small time bounds on such violations (usually?) so it's probably not the biggest problem your restaurant has to worry about (just apologize and have the customer try again). Dealing with money is similarly not a problem. ACH takes much longer to clear than the uncertainty window of linearizability violations. The client probably won't be able to tell the difference.