8 ms·

Call me maybe: Cassandra

268 points by lucian1900 13y ago

jbellis 13y ago
The bugs he ran into with lightweight transactions were fixed within days, two weeks ago, and included in the 2.0.1 release shortly afterwards.
- itp 13y ago
  The fact that these bugs were so trivially reproduced and made it into an initial release at all should be cause for concern. Conflict of interest disclosure -- I work for FoundationDB, where we put a shockingly high level of effort into testing our software in simulation and the real world. [1][2] [1] https://foundationdb.com/white-papers/testing https://foundationdb.com/white-papers/testing [2] https://foundationdb.com/blog/quicksand-continuous-real-world-fault-tolerance-testing https://foundationdb.com/blog/quicksand-continuous-real-worl...
  - j_baker 13y ago
    Please. I have yet to work in any environment that hasn't had a brown bag release or two, regardless of how well tested it is. Nice marketing attempt though.
  - argvzero 13y ago
    advice from someone who's been at this nosql thing for about 6 years now. schadenfreude will always come back to bite you. tread lightly.
    - calibraxis 13y ago
      Sorry for the offtopic post: just noticed now that three months ago, you mentioned I misrepresented your words. I deeply apologize; that's not at all what I meant. I had found your Ricon talk refreshingly honest and quoted it admiringly to people I know. (I had zero issue with anything you said; but rather with certain institutions.) I'm sorry for any errors/misrepresentations I made about your words, and will be more careful in the future. (Unfortunately, I can no longer edit, delete nor respond to that post.)
      - argvzero 13y ago
        haha no worries my friend :) glad you enjoyed the talk!
  - cbsmith 13y ago
    > The fact that these bugs were so trivially reproduced and made it into an initial release at all should be cause for concern. Umm... no. The .0 releases of the project are actually where you'd expect the most bugs. That's where you have the first commit with an implementation of a new design. You'll note that DataStax's commercial version of DSE is still based on 1.2.x...
  - deleted 13y ago
    [deleted]
  - rdtsc 13y ago
    If foundation db released and is it open sourced? I remember others (employees of company) hailing its wonderful qualities for quite a while now (years), then I go to the website and all I could find was a bunch of white papers and a registration form. And here it seems a bit of a "my vaporware's features are better than your shipped product's features". Now matter how many white papers there are I would still put my data in Cassandra rather than this new thing (last I checked I couldn't even download it, I had to fill out a form of some sort).
    - itp 13y ago
      FoundationDB isn't open source software, but it is a 1.0 product that's freely available to download right now[1]. (You are right in recalling that during our alpha and beta programs there was a simple registration form, and there's still an account signup for our community site and of course for Enterprise licensing and support). [1] https://foundationdb.com/get https://foundationdb.com/get
- tel 13y ago
  DataStax still claimed these properties while the bugs were live. It's important they fixed them quickly, but the point still stands—the point of the entire series, I think—that database companies very frequently make big claims that few people go through the process of checking... even at the companies themselves.
  - jbellis 13y ago
    It's important to distinguish architecture and design from implementation. The former may be correct, even if the latter is found to have a bug.
    - seiji 13y ago
      If you distinguish between those two, and implementation doesn't match design, then you must concede you have a level of incompetence in your development chain. When dealing with data, you don't get any points for "well, we meant to do it the right way, but it didn't happen. sorry."
      - jbellis 13y ago
        I guess I'll have to settle for joining the crowd of incompetents behind PostgreSQL, Linux, the JVM, et al., who have also been known to release bug fixes on occasion. :)
        tel 13y ago
        I guess I'm just quite surprised Aphyr turned that one up. I didn't look at the fix, but it's a pretty severe bug if it completely, silently disables a feature.
    - scott_s 13y ago
      It is, and I think the author did.
  - cbsmith 13y ago
    Have you ever seen a product feature description edited on account of a bug report, particularly one that was resolved within days?
- dilap 13y ago
  Of course, if aphyr hadn't reported the bugs before publishing, they'd be live right now, right?
- jamesaguilar 13y ago
  And who knows how many more are lurking? I think that's the issue exposed by this post -- if these really basic things are broken, why would anyone believe that the actual tricky stuff has been tested?
  - jbellis 13y ago
    It's awfully hard to prove a negative, but our test suites are open [1] [2], and so is our bug tracker [3]. [1] https://github.com/apache/cassandra/tree/trunk/test/unit/org/apache/cassandra https://github.com/apache/cassandra/tree/trunk/test/unit/org... [2] https://github.com/riptano/cassandra-dtest https://github.com/riptano/cassandra-dtest [3] https://issues.apache.org/jira/browse/CASSANDRA https://issues.apache.org/jira/browse/CASSANDRA
    - jamesaguilar 13y ago
      Interesting. Care to comment on why you think the issues brought up in this blog post weren't covered by your test cases? To me they seem to be approximately the first thing you would test with any transaction feature, but I don't work on a mutable datastore, so maybe I have a misapprehension of the situation. I guess what I'm trying to say is, if I see something I think should obviously be tested not tested, my go-to assumption is that the failure was in bad testing practices. To dispel that belief, I need something to replace it with that explains the observation even better. If you're willing to share your view of the situation, maybe that would do it. (There's no real business case for you to do it, since I'm not in your market. But as a hacker, I'm interested in what I can learn from this.)
      - jbellis 13y ago
        No excuses. It's as simple as me dropping the ball on this one.
  - cbsmith 13y ago
    > I think that's the issue exposed by this post -- if these really basic things are broken, why would anyone believe that the actual tricky stuff has been tested? So, I think it is relevant that LWT is still a very new feature in Cassandra and not something basic to it at all (arguably counter to a lot of its original design goals). Personally, I was much more concerned by the server side timestamps only using millisecond granularity (and even that is somewhat understandable given the JVM's limitations).
tomjohnson3 13y ago
i've been following kyle's jepsen project for a few months (check out his other posts) - and it seems (thankfully) to be nudging the focus of discussion away from only "performance metrics" (transactions/sec, etc.) of distributed systems and toward data consistency and behavior in the face of partitions and other faults. ...what's the use of being able to make 20k updates per second if half your data is lost during a common failure? kudos!
fleitz 13y ago
Great article on the fundamental problems associated with mutable state. The fundamental problem is that the idea of an object with a set of state that is the same to all observers violates pretty much the whole of information theory. It's not a problem that will ever be fixed with out changing the fundamental laws of the universe. Ditch the mutable data and you can stop asking questions like what do we do if 10 becomes 10.5 before it becomes 11 and start storing values which never change.
- aphyr 13y ago
  The question of consensus and mutability are, to some extent, separable; immutable data does not provide liveness without consensus guarantees. See, for instance, Datomic's design, where a strong coordinator is needed to serialize updates to otherwise immutable/replayable state.
  - mtrimpe 13y ago
    Indeed. Although the use of confluent data structures where possible combined with immutable data stores combined and 'live' functional reactive programming across device boundaries -would- drastically simplify cases like these.
- fusiongyro 13y ago
  The argument here is basically to replace mutable state with event sourcing. It's an interesting idea and sometimes the right one, but if each user action that triggered a one cell update becomes an event I have to keep forever, I see my database size exploding. This is also going to yield performance problems I'll be tempted to avert with caching, leading to fantastically bad performance or cascading failures whenever the caches fail or are restarted. I'm sure it's the right answer for some people, some of the time. In fact, I'm sure it's not applied as frequently as it should be. But it's definitely a specialized tool with special applications, where mutable state is, for better or worse, the hammer we can and ought to continue relying on.
  - deleted 13y ago
    [deleted]
dschiptsov 13y ago
What is the reason for trying to squeeze into JVM an engine which must be implemented on the one level up - an OS level? ,) Is it still not obvious that "universal object storage" is a naive idea?)
penguindev 13y ago
I'm concerned that the paxos test couldn't handle 50/sec without timing out practically all of them. Also, the 'official' datastax blog post I read describing the feature was not clear at all describing the granularity / sharding of the paxos state machines. It said a 'partition'. Does that mean a 'vnode' in cassandra speak? Otherwise, kudos for adding a potentially useful feature to cassandra.
- jbellis 13y ago
  As I mentioned above, we fixed the bugs he ran into; this is not representative behavior of 2.0.1. The granularity is a CQL partition: http://www.datastax.com/documentation/cql/3.0/webhelp/index.html#cql/ddl/ddl_anatomy_table_c.html http://www.datastax.com/documentation/cql/3.0/webhelp/index....
penguindev 13y ago
Interesting findings re. the paxos bugs. Dare I ask how reliable cass's sharded paxos is (in theory, at least) while you're adding/removing nodes from the cluster? How do all clients see consistent membership?
- jbellis 13y ago
  The principle is the same as with normal reads; expand the quorum to include both old and new owners until the transition is done: https://issues.apache.org/jira/browse/CASSANDRA-833?focusedCommentId=13028232&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13028232 https://issues.apache.org/jira/browse/CASSANDRA-833?focusedC...

268 points by lucian1900 13y ago

The bugs he ran into with lightweight transactions were fixed within days, two weeks ago, and included in the 2.0.1 release shortly afterwards.

itp 13y ago
The fact that these bugs were so trivially reproduced and made it into an initial release at all should be cause for concern. Conflict of interest disclosure -- I work for FoundationDB, where we put a shockingly high level of effort into testing our software in simulation and the real world. [1][2] [1] https://foundationdb.com/white-papers/testing https://foundationdb.com/white-papers/testing [2] https://foundationdb.com/blog/quicksand-continuous-real-world-fault-tolerance-testing https://foundationdb.com/blog/quicksand-continuous-real-worl...
- j_baker 13y ago
  Please. I have yet to work in any environment that hasn't had a brown bag release or two, regardless of how well tested it is. Nice marketing attempt though.
- argvzero 13y ago
  advice from someone who's been at this nosql thing for about 6 years now. schadenfreude will always come back to bite you. tread lightly.
  - calibraxis 13y ago
    Sorry for the offtopic post: just noticed now that three months ago, you mentioned I misrepresented your words. I deeply apologize; that's not at all what I meant. I had found your Ricon talk refreshingly honest and quoted it admiringly to people I know. (I had zero issue with anything you said; but rather with certain institutions.) I'm sorry for any errors/misrepresentations I made about your words, and will be more careful in the future. (Unfortunately, I can no longer edit, delete nor respond to that post.)
    - argvzero 13y ago
      haha no worries my friend :) glad you enjoyed the talk!
- cbsmith 13y ago
  > The fact that these bugs were so trivially reproduced and made it into an initial release at all should be cause for concern. Umm... no. The .0 releases of the project are actually where you'd expect the most bugs. That's where you have the first commit with an implementation of a new design. You'll note that DataStax's commercial version of DSE is still based on 1.2.x...
- deleted 13y ago
  [deleted]
- rdtsc 13y ago
  If foundation db released and is it open sourced? I remember others (employees of company) hailing its wonderful qualities for quite a while now (years), then I go to the website and all I could find was a bunch of white papers and a registration form. And here it seems a bit of a "my vaporware's features are better than your shipped product's features". Now matter how many white papers there are I would still put my data in Cassandra rather than this new thing (last I checked I couldn't even download it, I had to fill out a form of some sort).
  - itp 13y ago
    FoundationDB isn't open source software, but it is a 1.0 product that's freely available to download right now[1]. (You are right in recalling that during our alpha and beta programs there was a simple registration form, and there's still an account signup for our community site and of course for Enterprise licensing and support). [1] https://foundationdb.com/get https://foundationdb.com/get
tel 13y ago
DataStax still claimed these properties while the bugs were live. It's important they fixed them quickly, but the point still stands—the point of the entire series, I think—that database companies very frequently make big claims that few people go through the process of checking... even at the companies themselves.
- jbellis 13y ago
  It's important to distinguish architecture and design from implementation. The former may be correct, even if the latter is found to have a bug.
  - seiji 13y ago
    If you distinguish between those two, and implementation doesn't match design, then you must concede you have a level of incompetence in your development chain. When dealing with data, you don't get any points for "well, we meant to do it the right way, but it didn't happen. sorry."
    - jbellis 13y ago
      I guess I'll have to settle for joining the crowd of incompetents behind PostgreSQL, Linux, the JVM, et al., who have also been known to release bug fixes on occasion. :)
      - tel 13y ago
        I guess I'm just quite surprised Aphyr turned that one up. I didn't look at the fix, but it's a pretty severe bug if it completely, silently disables a feature.
  - scott_s 13y ago
    It is, and I think the author did.
- cbsmith 13y ago
  Have you ever seen a product feature description edited on account of a bug report, particularly one that was resolved within days?
dilap 13y ago
Of course, if aphyr hadn't reported the bugs before publishing, they'd be live right now, right?
jamesaguilar 13y ago
And who knows how many more are lurking? I think that's the issue exposed by this post -- if these really basic things are broken, why would anyone believe that the actual tricky stuff has been tested?
- jbellis 13y ago
  It's awfully hard to prove a negative, but our test suites are open [1] [2], and so is our bug tracker [3]. [1] https://github.com/apache/cassandra/tree/trunk/test/unit/org/apache/cassandra https://github.com/apache/cassandra/tree/trunk/test/unit/org... [2] https://github.com/riptano/cassandra-dtest https://github.com/riptano/cassandra-dtest [3] https://issues.apache.org/jira/browse/CASSANDRA https://issues.apache.org/jira/browse/CASSANDRA
  - jamesaguilar 13y ago
    Interesting. Care to comment on why you think the issues brought up in this blog post weren't covered by your test cases? To me they seem to be approximately the first thing you would test with any transaction feature, but I don't work on a mutable datastore, so maybe I have a misapprehension of the situation. I guess what I'm trying to say is, if I see something I think should obviously be tested not tested, my go-to assumption is that the failure was in bad testing practices. To dispel that belief, I need something to replace it with that explains the observation even better. If you're willing to share your view of the situation, maybe that would do it. (There's no real business case for you to do it, since I'm not in your market. But as a hacker, I'm interested in what I can learn from this.)
    - jbellis 13y ago
      No excuses. It's as simple as me dropping the ball on this one.
- cbsmith 13y ago
  > I think that's the issue exposed by this post -- if these really basic things are broken, why would anyone believe that the actual tricky stuff has been tested? So, I think it is relevant that LWT is still a very new feature in Cassandra and not something basic to it at all (arguably counter to a lot of its original design goals). Personally, I was much more concerned by the server side timestamps only using millisecond granularity (and even that is somewhat understandable given the JVM's limitations).

tomjohnson3 13y ago

i've been following kyle's jepsen project for a few months (check out his other posts) - and it seems (thankfully) to be nudging the focus of discussion away from only "performance metrics" (transactions/sec, etc.) of distributed systems and toward data consistency and behavior in the face of partitions and other faults. ...what's the use of being able to make 20k updates per second if half your data is lost during a common failure? kudos!

fleitz 13y ago

Great article on the fundamental problems associated with mutable state. The fundamental problem is that the idea of an object with a set of state that is the same to all observers violates pretty much the whole of information theory. It's not a problem that will ever be fixed with out changing the fundamental laws of the universe. Ditch the mutable data and you can stop asking questions like what do we do if 10 becomes 10.5 before it becomes 11 and start storing values which never change.

aphyr 13y ago
The question of consensus and mutability are, to some extent, separable; immutable data does not provide liveness without consensus guarantees. See, for instance, Datomic's design, where a strong coordinator is needed to serialize updates to otherwise immutable/replayable state.
- mtrimpe 13y ago
  Indeed. Although the use of confluent data structures where possible combined with immutable data stores combined and 'live' functional reactive programming across device boundaries -would- drastically simplify cases like these.
fusiongyro 13y ago
The argument here is basically to replace mutable state with event sourcing. It's an interesting idea and sometimes the right one, but if each user action that triggered a one cell update becomes an event I have to keep forever, I see my database size exploding. This is also going to yield performance problems I'll be tempted to avert with caching, leading to fantastically bad performance or cascading failures whenever the caches fail or are restarted. I'm sure it's the right answer for some people, some of the time. In fact, I'm sure it's not applied as frequently as it should be. But it's definitely a specialized tool with special applications, where mutable state is, for better or worse, the hammer we can and ought to continue relying on.
- deleted 13y ago
  [deleted]

dschiptsov 13y ago

What is the reason for trying to squeeze into JVM an engine which must be implemented on the one level up - an OS level? ,) Is it still not obvious that "universal object storage" is a naive idea?)

penguindev 13y ago

I'm concerned that the paxos test couldn't handle 50/sec without timing out practically all of them. Also, the 'official' datastax blog post I read describing the feature was not clear at all describing the granularity / sharding of the paxos state machines. It said a 'partition'. Does that mean a 'vnode' in cassandra speak? Otherwise, kudos for adding a potentially useful feature to cassandra.

jbellis 13y ago
As I mentioned above, we fixed the bugs he ran into; this is not representative behavior of 2.0.1. The granularity is a CQL partition: http://www.datastax.com/documentation/cql/3.0/webhelp/index.html#cql/ddl/ddl_anatomy_table_c.html http://www.datastax.com/documentation/cql/3.0/webhelp/index....

penguindev 13y ago

Interesting findings re. the paxos bugs. Dare I ask how reliable cass's sharded paxos is (in theory, at least) while you're adding/removing nodes from the cluster? How do all clients see consistent membership?

jbellis 13y ago
The principle is the same as with normal reads; expand the quorum to include both old and new owners until the transition is done: https://issues.apache.org/jira/browse/CASSANDRA-833?focusedCommentId=13028232&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13028232 https://issues.apache.org/jira/browse/CASSANDRA-833?focusedC...