5 ms·

Digg Saying Yes to NoSQL; Going Steady with Cassandra

69 points by jbellis 17y ago

rgrieselhuber 17y ago
I'm looking pretty seriously at MongoDB and I've heard that Cassandra is worth considering. I do a lot of data warehousing / statistical analytics which generally means some sort of star schema-based reporting with lots of crosstabs, dimensions, etc. If anyone can relate their experience with either of these two platforms, would either be a good choice for live querying for these types of applications? I know you can use MapReduce to eventually get the data you need, but I need to support queries that respond in (well) less than a second, even for very large data sets.
- dangrossman 17y ago
  Like any other data store, MongoDB will do what you need in less than a second... * if the database fits in memory *. Once it's on disk, it doesn't matter as much how little overhead the storage engine has, it's going to be slow.
- rbranson 17y ago
  HBase is probably closer to what you're looking for.
  - rgrieselhuber 17y ago
    Why?
    - deleted 17y ago
      [deleted]
- jbooth 17y ago
  It basically depends on 2 things: 1) What kind of API do you need? It'll have to be simpler than SQL, likely key-value with secondary indices managed at either the API layer or application layer. 2) Which 2 of Consistency, Availability and Partition Tolerance do you need? Is "gets there eventually" enough or do you need "can read it right after I write it" guarantees?
- jbellis 17y ago
  MongoDB (and SQL dbs) does ad-hoc queries quite well as long as your data fits on a single machine. Cassandra does ad-hoc queries quite well if you're willing to use mapreduce. If you have a smaller number of frequently done queries and are willing to use MR for the others maybe that is a happy place for you.
zaidf 17y ago
For someone who knows nothing about NoSQL and decent with MySQL, can someone give a brief Idiot's overview of how NoSQL works? If I add a record in say a table named "news", where is the data stored? If I need to do a search by news id or description, what's the front-end api like and what happens at the backend when the api is called?
- nreece 17y ago
  Have a read at http://en.wikipedia.org/wiki/NoSQL http://en.wikipedia.org/wiki/NoSQL
- hachiya 17y ago
  You can try MongoDB and Redis in your browser, with these excellent interactive tutorials: http://try.mongodb.org/ http://try.mongodb.org/ http://try.redis-db.com/ http://try.redis-db.com/ The above site has been discussed today on HN: http://news.ycombinator.com/item?id=1181714 http://news.ycombinator.com/item?id=1181714
  - lurkerperpetual 17y ago
    the mongo shell above needs some Stop Eval/Reset button as it can get into a non-responsive state after a bad parse > db.scores.find({'sdf':) >> }) also no copy paste in the browser version AFAICT
- ronnier 17y ago
  This gives a little background and might help answer some of your questions (I have similar questions to yours): http://www.facebook.com/note.php?note_id=24413138919 http://www.facebook.com/note.php?note_id=24413138919
- code_duck 17y ago
  There isn't an answer that fits all of the different systems called 'NoSQL'. While you can count on any common RDBMS to be accesible through SQL, there is not a 'NoSQL' language or specification. In general, access is somewhat like using an ORM. The data is stored in files of a special format, just like an SQL database.
thinker 17y ago
For those who may have missed this a couple weeks ago, Twitter also considering Cassandra - http://nosql.mypopescu.com/post/407159447/cassandra-twitter-an-interview-with-ryan-king http://nosql.mypopescu.com/post/407159447/cassandra-twitter-...
- jbellis 17y ago
  Past the considering stage and to the "deploying" stage.
JulianMorrison 17y ago
Nice to see Digg pouring effort into making Cassandra itself better for everyone.
joshd 17y ago
I'd love to see an example of how people are redefining their schema with NoSQL databases (especially with document based databases like CouchDB). A common example you hear is "Your blog document can contain an array of comment nodes". Which is all great in theory but obviously won't scale. If Digg are using Cassandra as a big key-value store then how do they look up comments for posts? If they're storing one entry per comment with an index on post ID then it's not really key based any more. We hear a lot of case studies about performance increases, but I never seem to see any practical details on how the databases should be used correctly.
- ieure 17y ago
  Comments are denormalized and stored two ways: One in a plain ColumnFamily (for random access) and one in a SuperCF (for sequential access). The plain CF ("Comments") uses the comment ID (which is a timestamp+salt) as the row key. The fields of the comment are columns in the row (username, text, date_created etc). The SuperCF ("StoryComments") uses the story ID as the row key. Each row contains one SuperColumn per comment. The SC name is the comment_id, and the columns are the fields of the comment. So, say you want to get the first 50 comments for a given story, you'd do a get_slice on StoryComments, passing the story ID in as the row key, "" as the start column, and a count of 50. You get back 50 SuperColumns, each of which contains one comment. Cassandra is extremely good for sequential reads, not just random lookups. Just about any list of things can be efficiently stored and retrieved with its batch insertion and slicing operations.
- jbellis 17y ago
  Digg described the model for their "green badges" feature in http://about.digg.com/blog/looking-future-cassandra http://about.digg.com/blog/looking-future-cassandra, fwiw.

69 points by jbellis 17y ago

I'm looking pretty seriously at MongoDB and I've heard that Cassandra is worth considering. I do a lot of data warehousing / statistical analytics which generally means some sort of star schema-based reporting with lots of crosstabs, dimensions, etc. If anyone can relate their experience with either of these two platforms, would either be a good choice for live querying for these types of applications? I know you can use MapReduce to eventually get the data you need, but I need to support queries that respond in (well) less than a second, even for very large data sets.

dangrossman 17y ago
Like any other data store, MongoDB will do what you need in less than a second... * if the database fits in memory *. Once it's on disk, it doesn't matter as much how little overhead the storage engine has, it's going to be slow.
rbranson 17y ago
HBase is probably closer to what you're looking for.
- rgrieselhuber 17y ago
  Why?
  - deleted 17y ago
    [deleted]
jbooth 17y ago
It basically depends on 2 things: 1) What kind of API do you need? It'll have to be simpler than SQL, likely key-value with secondary indices managed at either the API layer or application layer. 2) Which 2 of Consistency, Availability and Partition Tolerance do you need? Is "gets there eventually" enough or do you need "can read it right after I write it" guarantees?
jbellis 17y ago
MongoDB (and SQL dbs) does ad-hoc queries quite well as long as your data fits on a single machine. Cassandra does ad-hoc queries quite well if you're willing to use mapreduce. If you have a smaller number of frequently done queries and are willing to use MR for the others maybe that is a happy place for you.

zaidf 17y ago

For someone who knows nothing about NoSQL and decent with MySQL, can someone give a brief Idiot's overview of how NoSQL works? If I add a record in say a table named "news", where is the data stored? If I need to do a search by news id or description, what's the front-end api like and what happens at the backend when the api is called?

nreece 17y ago
Have a read at http://en.wikipedia.org/wiki/NoSQL http://en.wikipedia.org/wiki/NoSQL
hachiya 17y ago
You can try MongoDB and Redis in your browser, with these excellent interactive tutorials: http://try.mongodb.org/ http://try.mongodb.org/ http://try.redis-db.com/ http://try.redis-db.com/ The above site has been discussed today on HN: http://news.ycombinator.com/item?id=1181714 http://news.ycombinator.com/item?id=1181714
- lurkerperpetual 17y ago
  the mongo shell above needs some Stop Eval/Reset button as it can get into a non-responsive state after a bad parse > db.scores.find({'sdf':) >> }) also no copy paste in the browser version AFAICT
ronnier 17y ago
This gives a little background and might help answer some of your questions (I have similar questions to yours): http://www.facebook.com/note.php?note_id=24413138919 http://www.facebook.com/note.php?note_id=24413138919
code_duck 17y ago
There isn't an answer that fits all of the different systems called 'NoSQL'. While you can count on any common RDBMS to be accesible through SQL, there is not a 'NoSQL' language or specification. In general, access is somewhat like using an ORM. The data is stored in files of a special format, just like an SQL database.

thinker 17y ago

For those who may have missed this a couple weeks ago, Twitter also considering Cassandra - http://nosql.mypopescu.com/post/407159447/cassandra-twitter-an-interview-with-ryan-king http://nosql.mypopescu.com/post/407159447/cassandra-twitter-...

jbellis 17y ago
Past the considering stage and to the "deploying" stage.

JulianMorrison 17y ago

Nice to see Digg pouring effort into making Cassandra itself better for everyone.

joshd 17y ago

I'd love to see an example of how people are redefining their schema with NoSQL databases (especially with document based databases like CouchDB). A common example you hear is "Your blog document can contain an array of comment nodes". Which is all great in theory but obviously won't scale. If Digg are using Cassandra as a big key-value store then how do they look up comments for posts? If they're storing one entry per comment with an index on post ID then it's not really key based any more. We hear a lot of case studies about performance increases, but I never seem to see any practical details on how the databases should be used correctly.

ieure 17y ago
Comments are denormalized and stored two ways: One in a plain ColumnFamily (for random access) and one in a SuperCF (for sequential access). The plain CF ("Comments") uses the comment ID (which is a timestamp+salt) as the row key. The fields of the comment are columns in the row (username, text, date_created etc). The SuperCF ("StoryComments") uses the story ID as the row key. Each row contains one SuperColumn per comment. The SC name is the comment_id, and the columns are the fields of the comment. So, say you want to get the first 50 comments for a given story, you'd do a get_slice on StoryComments, passing the story ID in as the row key, "" as the start column, and a count of 50. You get back 50 SuperColumns, each of which contains one comment. Cassandra is extremely good for sequential reads, not just random lookups. Just about any list of things can be efficiently stored and retrieved with its batch insertion and slicing operations.
jbellis 17y ago
Digg described the model for their "green badges" feature in http://about.digg.com/blog/looking-future-cassandra http://about.digg.com/blog/looking-future-cassandra, fwiw.