4 ms·

WTF is a SuperColumn? An Intro to the Cassandra Data Model

53 points by ropiku 17y ago

StrawberryFrog 17y ago
So it's not in first normal form. http://en.wikipedia.org/wiki/First_normal_form http://en.wikipedia.org/wiki/First_normal_form What's old is new again, and I wonder if they even know it. What's not addressed is that this may not be an advance over Relational databases. In fact, when relational databases were first designed, databases like this were all-to-common, and RDBMS rejected this kind of data model explicitly, for good design reasons. For instance, how do you select from or join to parts of the complex values inside a column?
- seiji 17y ago
  I believe the answer is http://browsertoolkit.com/fault-tolerance.png http://browsertoolkit.com/fault-tolerance.png People are just trying to store data in the most efficient system possible. Using RDBMS software to store data only queried by primary key that you know will never need to be aggregated or dynamically queried can be overkill.
- bhousel 17y ago
  You have some catching up to do... :) They are not at all concerned about normal form or joins - it's just not that kind of a database. Here are some good blog posts that elaborate on the current non-RDBMS fad: http://www.readwriteweb.com/enterprise/2009/02/is-the-relational-database-doomed.php http://www.readwriteweb.com/enterprise/2009/02/is-the-relati... http://www.metabrew.com/article/anti-rdbms-a-list-of-distributed-key-value-stores/ http://www.metabrew.com/article/anti-rdbms-a-list-of-distrib... http://randomfoo.net/2009/04/20/some-notes-on-distributed-key-stores http://randomfoo.net/2009/04/20/some-notes-on-distributed-ke... http://bcbio.wordpress.com/2009/05/10/evaluating-key-value-and-document-stores-for-short-read-data/ http://bcbio.wordpress.com/2009/05/10/evaluating-key-value-a... http://blog.boxedice.com/2009/07/25/choosing-a-non-relational-database-why-we-migrated-from-mysql-to-mongodb/ http://blog.boxedice.com/2009/07/25/choosing-a-non-relationa...
- th0ma5 17y ago
  The guy that invented the normal forms also went on to invent analytical forms, which are the exact opposite. For example, the Entity Attribute Value (EAV) model, used by many including the W3C Semantic Web.
  - blasdel 17y ago
    It's too bad that RDF has been tainted by the W3C's pipe -dreams and been bludgeoned with a terrible XML serialization. EAV is by far my favorite data model.
- jbellis 17y ago
  > What's not addressed is that this may not be an advance over Relational databases It's not. The relational model is much better, for many applications. BUT. The missing context is, the relational model cannot be sanely scaled out across multiple machines. Replication mostly works with some pain, but scaling writes is just a nightmare -- and what you have to do to partition your data across multiple rdbms nodes means giving up all that relational goodness. So what you end up with is neither of {easily scaled, relational}. So if you can't have a scalable rdbms, the next best thing is a scalable key/columnfamily dbms. Something like Cassandra. Which, as a bonus, gives you significantly better per-node performance on modern hardware.
  - gruseom 17y ago
    Which, as a bonus, gives you significantly better per-node performance on modern hardware Could you expand on that last bit? i.e. what is it about modern hardware (and about Cassandra) that makes this true?
    - jbellis 17y ago
      Summarized here: http://news.ycombinator.com/item?id=813894 http://news.ycombinator.com/item?id=813894
  - Maro 17y ago
    Can you explain what the major advantage (and disadvantage) of the data model of Cassandra over a flat key-value store is?
    - jbellis 17y ago
      You can update at the column granularity, so having a row of 100k columns is fine since you don't have to rewrite the whole thing to update a small part. (Cassandra rows have a sparse list of columns, so you can treat a row as a sorted set, not just a fixed-size vector. The model Digg describes in another article uses this: http://blog.digg.com/?p=966 http://blog.digg.com/?p=966) Columns are also indexed so you can retrieve from large rows efficiently too. Finally Cassandra also adds the concept of a SuperColumn, which is a column that contains other columns.
cosmohh 17y ago
looks pretty interesting. anyone already used it and maybe compared it with CouchDB, too ?
- jbellis 17y ago
  Cassandra is a scalable key/columnfamily database designed for supporting low-latency applications with vast amounts of data partitioned across many machines. (Facebook has 40TB on a 150-machine Cassandra cluster.) CouchDB is a document database that supports two-way replication so you can re-sync after taking part of the data offline, but it's still designed around the concept of a single master that holds all the data. Completely different animals, in other words.
  - ddiljoy 17y ago
    Does a Cassandra cluster stay write-available in the event of a network partition? If so, how does it reconcile writes when the partition heals? Last I looked, Cassandra doesn't use vector/logical clocks - doesn't that potentially cause data loss when the partition heals if you're using a simple last-write-wins based on physical timestamps for a reconciliation policy? Does Cassandra use merkle trees for anti-entropy? From what I can tell, although Cassandra claims to be write-fault-tolerant, the dependence on physical timestamps and the lack of the self-healing properties that merkle trees provide make me nervous about data loss and inconsistency when deploying it at scale.
    - jbellis 17y ago
      > Does a Cassandra cluster stay write-available in the event of a network partition? The client can specify whether it wants consistency (refuse writes if not enough write targets are there) or availability. If it chooses availability, then Cassandra sends extra copies to nodes it _can_ reach, with a tag that specifies who the "real" destination is. When that node is reachable again it will be forwarded. ("Hinted handoff.") > how does it reconcile writes when the partition heals? As you said, last-write-wins. The experience with Dynamo showed that most apps don't want to deal with explicit conflict resolution, and don't need it. (But, I suspect we will end up adding it as an option for those apps that do. In the meantime, if Cassandra isn't a good fit, we're not trying to hard-sell anyone. :) > Does Cassandra use merkle trees for anti-entropy? Not yet, but my co-worker Stu Hood is working on this. Should be part of the 0.5 release. > the dependence on physical timestamps and the lack of the self-healing properties Whether the first is an issue is app-specific. As to the latter, I'm excited to get the merkle tree code in, too. In the meantime, Cassandra _does_ do read repair and hinted handoff, so in practice it's what I would call "barely adequate." :)

53 points by ropiku 17y ago

So it's not in first normal form. http://en.wikipedia.org/wiki/First_normal_form http://en.wikipedia.org/wiki/First_normal_form What's old is new again, and I wonder if they even know it. What's not addressed is that this may not be an advance over Relational databases. In fact, when relational databases were first designed, databases like this were all-to-common, and RDBMS rejected this kind of data model explicitly, for good design reasons. For instance, how do you select from or join to parts of the complex values inside a column?

seiji 17y ago
I believe the answer is http://browsertoolkit.com/fault-tolerance.png http://browsertoolkit.com/fault-tolerance.png People are just trying to store data in the most efficient system possible. Using RDBMS software to store data only queried by primary key that you know will never need to be aggregated or dynamically queried can be overkill.
bhousel 17y ago
You have some catching up to do... :) They are not at all concerned about normal form or joins - it's just not that kind of a database. Here are some good blog posts that elaborate on the current non-RDBMS fad: http://www.readwriteweb.com/enterprise/2009/02/is-the-relational-database-doomed.php http://www.readwriteweb.com/enterprise/2009/02/is-the-relati... http://www.metabrew.com/article/anti-rdbms-a-list-of-distributed-key-value-stores/ http://www.metabrew.com/article/anti-rdbms-a-list-of-distrib... http://randomfoo.net/2009/04/20/some-notes-on-distributed-key-stores http://randomfoo.net/2009/04/20/some-notes-on-distributed-ke... http://bcbio.wordpress.com/2009/05/10/evaluating-key-value-and-document-stores-for-short-read-data/ http://bcbio.wordpress.com/2009/05/10/evaluating-key-value-a... http://blog.boxedice.com/2009/07/25/choosing-a-non-relational-database-why-we-migrated-from-mysql-to-mongodb/ http://blog.boxedice.com/2009/07/25/choosing-a-non-relationa...
th0ma5 17y ago
The guy that invented the normal forms also went on to invent analytical forms, which are the exact opposite. For example, the Entity Attribute Value (EAV) model, used by many including the W3C Semantic Web.
- blasdel 17y ago
  It's too bad that RDF has been tainted by the W3C's pipe -dreams and been bludgeoned with a terrible XML serialization. EAV is by far my favorite data model.
jbellis 17y ago
> What's not addressed is that this may not be an advance over Relational databases It's not. The relational model is much better, for many applications. BUT. The missing context is, the relational model cannot be sanely scaled out across multiple machines. Replication mostly works with some pain, but scaling writes is just a nightmare -- and what you have to do to partition your data across multiple rdbms nodes means giving up all that relational goodness. So what you end up with is neither of {easily scaled, relational}. So if you can't have a scalable rdbms, the next best thing is a scalable key/columnfamily dbms. Something like Cassandra. Which, as a bonus, gives you significantly better per-node performance on modern hardware.
- gruseom 17y ago
  Which, as a bonus, gives you significantly better per-node performance on modern hardware Could you expand on that last bit? i.e. what is it about modern hardware (and about Cassandra) that makes this true?
  - jbellis 17y ago
    Summarized here: http://news.ycombinator.com/item?id=813894 http://news.ycombinator.com/item?id=813894
- Maro 17y ago
  Can you explain what the major advantage (and disadvantage) of the data model of Cassandra over a flat key-value store is?
  - jbellis 17y ago
    You can update at the column granularity, so having a row of 100k columns is fine since you don't have to rewrite the whole thing to update a small part. (Cassandra rows have a sparse list of columns, so you can treat a row as a sorted set, not just a fixed-size vector. The model Digg describes in another article uses this: http://blog.digg.com/?p=966 http://blog.digg.com/?p=966) Columns are also indexed so you can retrieve from large rows efficiently too. Finally Cassandra also adds the concept of a SuperColumn, which is a column that contains other columns.

cosmohh 17y ago

looks pretty interesting. anyone already used it and maybe compared it with CouchDB, too ?

jbellis 17y ago
Cassandra is a scalable key/columnfamily database designed for supporting low-latency applications with vast amounts of data partitioned across many machines. (Facebook has 40TB on a 150-machine Cassandra cluster.) CouchDB is a document database that supports two-way replication so you can re-sync after taking part of the data offline, but it's still designed around the concept of a single master that holds all the data. Completely different animals, in other words.
- ddiljoy 17y ago
  Does a Cassandra cluster stay write-available in the event of a network partition? If so, how does it reconcile writes when the partition heals? Last I looked, Cassandra doesn't use vector/logical clocks - doesn't that potentially cause data loss when the partition heals if you're using a simple last-write-wins based on physical timestamps for a reconciliation policy? Does Cassandra use merkle trees for anti-entropy? From what I can tell, although Cassandra claims to be write-fault-tolerant, the dependence on physical timestamps and the lack of the self-healing properties that merkle trees provide make me nervous about data loss and inconsistency when deploying it at scale.
  - jbellis 17y ago
    > Does a Cassandra cluster stay write-available in the event of a network partition? The client can specify whether it wants consistency (refuse writes if not enough write targets are there) or availability. If it chooses availability, then Cassandra sends extra copies to nodes it _can_ reach, with a tag that specifies who the "real" destination is. When that node is reachable again it will be forwarded. ("Hinted handoff.") > how does it reconcile writes when the partition heals? As you said, last-write-wins. The experience with Dynamo showed that most apps don't want to deal with explicit conflict resolution, and don't need it. (But, I suspect we will end up adding it as an option for those apps that do. In the meantime, if Cassandra isn't a good fit, we're not trying to hard-sell anyone. :) > Does Cassandra use merkle trees for anti-entropy? Not yet, but my co-worker Stu Hood is working on this. Should be part of the 0.5 release. > the dependence on physical timestamps and the lack of the self-healing properties Whether the first is an issue is app-specific. As to the latter, I'm excited to get the merkle tree code in, too. In the meantime, Cassandra _does_ do read repair and hinted handoff, so in practice it's what I would call "barely adequate." :)