19 ms·

NoSQL Databases: A Survey and Decision Guidance

297 points by DivineTraube 10y ago

DivineTraube 10y ago
Felix, author of the article and founder of Baqend [1] here. I'm happy to answer any questions. If you have suggestions for the next version of the article, we're eager to hear them. [1] http://www.baqend.com http://www.baqend.com
lostcolony 10y ago
One slightly concerning thing about this is the table at the end dictating what popular NoSQL DBs do is populated by, and here I quote - "The methodology used to identify the specific system properties consists of an in-depth analysis of publicly available documentation and literature on the systems. Furthermore, some properties had to be evaluated by researching the open-source code bases, personal communication with the developers as well as a meta-analysis of reports and benchmarks by practitioners." Which looking at the table, doesn't seem to include Aphyr's work (Jepsen), so much of what is being said may in fact be wrong. I'd prefer a chart populated from that (though admittedly such a chart would largely be "undefined", "inconsistent", etc).
- DivineTraube 10y ago
  We tried our best to incorporate these results in the cases where they reflect fundamental shortcomings not bugs (of which Aphyr found quite a few). But you are right, we could include a list of popular examples, where description and experimental findings diverge. Which cases did you have in mind? MongoDB being CP?
  - lostcolony 10y ago
    And Redis, yeah. In general, seeing his work has made me very cagey at trusting any DB's marketing claims; I'd love to see citations as to what resources went into each claim, as that would inform me "this has been tested by a third party" vs "this has been claimed in marketing docs and YMMV; you should probably test it yourself"
SmellTheGlove 10y ago
This is pretty useful for someone like me who's been in large companies using Teradata forever, just to understand the NoSQL products out there and potential use cases. On any given day, I feel like I've fallen way behind on database technology simply because we're fairly locked in, admittedly very happy with TD. It's helpful to keep up here though, because we do have some projects we'd like to do that would probably work better outside of an RDBMS. I'd love to see something similar on the various ETL/data pipeline tools. In that regard, we still write a ton of SAS code because we have a variety of sources, which SAS does well, pretty solid if you need to do some data cleansing into the target, and the code itself is very batch-friendly and maintainable. It's been a while since I've surveyed alternatives.
dvirsky 10y ago
> Redis, as the only non-partitioned system in this comparison ... This is a bit misguided, redis cluster has been in GA for over a year.
- DivineTraube 10y ago
  I agree, Redis Cluster has been available for a while. From a practical perspective we did not deem it production-ready yet, due to various shortcomings, e.g.: -Redis Cluter being neither AP nor CP with rather unsatisfying justifications [1] -Strong scalability issues, for instance with PubSub [2] But you are right, I think Redis Cluster could be added as a separate system. [1] https://aphyr.com/posts/283-jepsen-redis https://aphyr.com/posts/283-jepsen-redis [2] https://github.com/antirez/redis/issues/2672 https://github.com/antirez/redis/issues/2672
  - dvirsky 10y ago
    That's a good idea. At the very least add a note about it in the mean time maybe. BTW as a caching-only system redis can scale exactly like memcache if your client supports it. I did this in one project and it worked great. EDIT: Also, it might be too niche, but you can also have a look at redis+dynomite, which makes redis behave like a distributed database. https://github.com/Netflix/dynomite https://github.com/Netflix/dynomite
  - antirez 10y ago
    A couple of points, just for clarity: 1) It is true that Redis is not AP or CP, but this is not covered by [1] that covers Redis Sentinel (a previous version with different behavior compared to Sentinel v2 btw). So, while Redis Cluster is just an eventually consistent system that does not guarantees strong consistency nor availability (so is not AP nor CP), the pointed post is not related. 2) Redis Cluster is very scalable actually, since it is a flatly partitioned no-proxed system. Pub/Sub is not very scalable in Redis Cluster but Pub/Sub is a minor sub-system of Redis, most users look at the ability to scale the key-value store that is... 90% of what Redis is, so I think your claim is not justified. 3) The fact of not being AP or CP does not mean that a system is not useful. Depends on the business requirements. In fact, most SQL databases + failover setups, that is what takes a seizable portion of the big services of the internet up, are not AP or CP as well. Database systems features are related to use cases, I believe that the Redis Cluster properties cover a huge set of real world use cases, in fact is the most actively mentioned/requested feature right now, even if we document in very clear terms the tradeoff and the actual behavior of the system. So people know what they want. IMHO excluding systems because of what you think being acceptable tradeoffs is not a good idea. Otherwise you should also mention that SQL+ Main_widely_used_failover_solutions systems have the same unacceptable shortcomings, for instance. Actually if the Redis Cluster implementation will be made the center of the future Redis development, it could easily become the most used sub-system of Redis, like Sentinel is already becoming. There are many devs that know what they are doing and know how to apply things depending on the requirements even if those things are not CP or AP.
mohanmca 10y ago
Great work!
jcadam 10y ago
On my personal projects, my default database choice is Postgres unless I have a glaring, obvious reason to choose something NoSQL (in which case I like CouchDB if the situation calls for a document store, don't know much about the others). At work, it doesn't matter if we have a perfect use case for a particular NoSQL solution (I'm working on something now that I think would be a great fit for a Graph database like Neo4J). We use Oracle, because it's expensive, we're already paying for it, so damn it, we're going to use it.
- DiabloD3 10y ago
  I agree. However, re: document store, I've been bouncing Postgres around in my head for that: all required columns (ie, columns that always exist, or will be queried on) are normal SQL columns, everything else is stashed in one or more hstore columns.
  - StavrosK 10y ago
    > everything else is stashed in one or more hstore columns. This has the same problem as most NoSQL solutions: You don't get the benefit of a schema, and schemas are damn useful things.
    - elmigranto 10y ago
      You can define arbitrary constraints of any level of complexity with Postgres for any column, be it plain int, hstore, jsonb or anything else. Most of that will also be efficiently indexable out of the box.
    - DiabloD3 10y ago
      Not everything needs a schema. The alternative I've also been bouncing around in my head is multiple tables: objects (rows in the "primary" table) have traits (rows in "secondary" tables with identical primary keys) that break out columns according to what they do. Having 9000 columns in a single table is a bit unwieldy.
      - 10y ago
deleted 10y ago
[deleted]
cwmma 10y ago
I'd point out that there are arguments[1] that 'CA' is not a real choose you can make when it comes to the CAP theorem. 1: https://codahale.com/you-cant-sacrifice-partition-tolerance/ https://codahale.com/you-cant-sacrifice-partition-tolerance/
morrbo 10y ago
serious question, what is to stop everyone (where a K/V store is not enough) from just chucking some form of SQL server in a VM in a ramdisk and using that? I've always wondered about that idea
- virmundi 10y ago
  Lost efficiency. If you want a semi-persistent relational DB, you can use BerkleyDB. There are also embedded DBs for Java, and probably other languages. If you do this, you're not wasting RAM for the VM and the guest OS. The DB gets its memory from the large application install. edited for typo.
virmundi 10y ago
I enjoyed this topic while researching New Data: a Field Guide [1]. In my research I've come to really enjoy ArangoDB. It has good support for documents, performance, relational data queries and basic graphs. Without doing the research I don't think I would have ever dug into the new tools. If you ever get a chance to do a small prototype, looking using these tools. 1- https://leanpub.com/NewDataAFieldGuide https://leanpub.com/NewDataAFieldGuide
Feneric 10y ago
Not even a mention of ZODB. While it's true it's Python-only, it's certainly one of the most mature NoSQL databases around.
- lgas 10y ago
  The context the author is describing is clustered databases, does ZODB offer a clustering option that is equally as mature?
  - Feneric 10y ago
    You mean like ZEO? Yes, it's probably been around for a couple of decades now.
    - lgas 10y ago
      I don't think so. I am referring to something that allows you to scale your database cluster horizontally by adding more servers. ZEO appears to only support a single server.
abannin 10y ago
This is a fantastic overview of the issues surrounding DB's. Would love to see some deeper discussions on graph databases in the next version!
antonios 10y ago
This article will soon become outdated: CouchDB 2.0 is in release candidate stage, and will soon be released. It features built-in and automatic sharding and clustering and a new, MongoDB-inspired document query language...and lots of small improvements as well. For more information, read the excellent blog posts from their official blog: https://blog.couchdb.org/ https://blog.couchdb.org/
- aeharding 10y ago
  Anyone looking for more information in CouchDB 2 redundancy should check out this article: http://docs.couchdb.org/en/1.6.1/cluster/theory.html http://docs.couchdb.org/en/1.6.1/cluster/theory.html (Ignore that it's under /en/1.6.1, built-in clustering is 2+ only.) I'm really excited about CouchDB 2.0.
- DivineTraube 10y ago
  I'm also very curious about CouchDB 2.0 and whether CouchDB will be able to make a comeback. When I talked to Adam Kocoloski about a year ago, I got the impression that it takes a very good technical approach (consistent hashing, causal hash histories, MongoDB queries, etc.). However, the implementation is now mainly driven by Cloudant and CouchDB kept alive by them. I hope they manage to rekindle the old fascination they once had. In any case I will be happy to update the article and include it.
fizixer 10y ago
Slightly off topic but: As someone out of touch on the topic of DB utilizing apps for a while, what's the update on the ORM impedance mismatch? (I mean the guidelines or best practices).
sandstrom 10y ago
Great to see RethinkDB covered (although it would have been better if it was included in the primary comparison tables too). It feels like a second-generation NoSQL database where most of the drawbacks with earlier NoSQL-databases has been taken care of!
deleted 10y ago
[deleted]
novaleaf 10y ago
boo. doesn't include a comparison vs Google's NoSql SaaS offerings (Cloud DataStore and BigTable). i personally use Cloud DataStore and find it a great fit, wish there was some comparisons against it.
- CydeWeys 10y ago
  Just wait for Cloud Spanner! http://siliconangle.com/blog/2016/06/27/google-tools-up-with-its-spanner-database-looks-for-a-fight-with-aws/ http://siliconangle.com/blog/2016/06/27/google-tools-up-with...
samwyse 10y ago
The TL;DR was TL;DR. Ho, seriously. About halfway through, the buzzwords and jargon got to be too much for me. Let me know if there's anything interesting Wining.
asdr 10y ago
If you wish an introduction to NoSQL there is a good webinar at http://www.prohuddle.com/webinars/nosql/NoSQL_Distilled.php http://www.prohuddle.com/webinars/nosql/NoSQL_Distilled.php, which includes discussions, examples and comparisons.

297 points by DivineTraube 10y ago

Felix, author of the article and founder of Baqend [1] here. I'm happy to answer any questions. If you have suggestions for the next version of the article, we're eager to hear them. [1] http://www.baqend.com http://www.baqend.com

lostcolony 10y ago

One slightly concerning thing about this is the table at the end dictating what popular NoSQL DBs do is populated by, and here I quote - "The methodology used to identify the specific system properties consists of an in-depth analysis of publicly available documentation and literature on the systems. Furthermore, some properties had to be evaluated by researching the open-source code bases, personal communication with the developers as well as a meta-analysis of reports and benchmarks by practitioners." Which looking at the table, doesn't seem to include Aphyr's work (Jepsen), so much of what is being said may in fact be wrong. I'd prefer a chart populated from that (though admittedly such a chart would largely be "undefined", "inconsistent", etc).

DivineTraube 10y ago
We tried our best to incorporate these results in the cases where they reflect fundamental shortcomings not bugs (of which Aphyr found quite a few). But you are right, we could include a list of popular examples, where description and experimental findings diverge. Which cases did you have in mind? MongoDB being CP?
- lostcolony 10y ago
  And Redis, yeah. In general, seeing his work has made me very cagey at trusting any DB's marketing claims; I'd love to see citations as to what resources went into each claim, as that would inform me "this has been tested by a third party" vs "this has been claimed in marketing docs and YMMV; you should probably test it yourself"

SmellTheGlove 10y ago

This is pretty useful for someone like me who's been in large companies using Teradata forever, just to understand the NoSQL products out there and potential use cases. On any given day, I feel like I've fallen way behind on database technology simply because we're fairly locked in, admittedly very happy with TD. It's helpful to keep up here though, because we do have some projects we'd like to do that would probably work better outside of an RDBMS. I'd love to see something similar on the various ETL/data pipeline tools. In that regard, we still write a ton of SAS code because we have a variety of sources, which SAS does well, pretty solid if you need to do some data cleansing into the target, and the code itself is very batch-friendly and maintainable. It's been a while since I've surveyed alternatives.

dvirsky 10y ago

> Redis, as the only non-partitioned system in this comparison ... This is a bit misguided, redis cluster has been in GA for over a year.

DivineTraube 10y ago
I agree, Redis Cluster has been available for a while. From a practical perspective we did not deem it production-ready yet, due to various shortcomings, e.g.: -Redis Cluter being neither AP nor CP with rather unsatisfying justifications [1] -Strong scalability issues, for instance with PubSub [2] But you are right, I think Redis Cluster could be added as a separate system. [1] https://aphyr.com/posts/283-jepsen-redis https://aphyr.com/posts/283-jepsen-redis [2] https://github.com/antirez/redis/issues/2672 https://github.com/antirez/redis/issues/2672
- dvirsky 10y ago
  That's a good idea. At the very least add a note about it in the mean time maybe. BTW as a caching-only system redis can scale exactly like memcache if your client supports it. I did this in one project and it worked great. EDIT: Also, it might be too niche, but you can also have a look at redis+dynomite, which makes redis behave like a distributed database. https://github.com/Netflix/dynomite https://github.com/Netflix/dynomite
- antirez 10y ago
  A couple of points, just for clarity: 1) It is true that Redis is not AP or CP, but this is not covered by [1] that covers Redis Sentinel (a previous version with different behavior compared to Sentinel v2 btw). So, while Redis Cluster is just an eventually consistent system that does not guarantees strong consistency nor availability (so is not AP nor CP), the pointed post is not related. 2) Redis Cluster is very scalable actually, since it is a flatly partitioned no-proxed system. Pub/Sub is not very scalable in Redis Cluster but Pub/Sub is a minor sub-system of Redis, most users look at the ability to scale the key-value store that is... 90% of what Redis is, so I think your claim is not justified. 3) The fact of not being AP or CP does not mean that a system is not useful. Depends on the business requirements. In fact, most SQL databases + failover setups, that is what takes a seizable portion of the big services of the internet up, are not AP or CP as well. Database systems features are related to use cases, I believe that the Redis Cluster properties cover a huge set of real world use cases, in fact is the most actively mentioned/requested feature right now, even if we document in very clear terms the tradeoff and the actual behavior of the system. So people know what they want. IMHO excluding systems because of what you think being acceptable tradeoffs is not a good idea. Otherwise you should also mention that SQL+ Main_widely_used_failover_solutions systems have the same unacceptable shortcomings, for instance. Actually if the Redis Cluster implementation will be made the center of the future Redis development, it could easily become the most used sub-system of Redis, like Sentinel is already becoming. There are many devs that know what they are doing and know how to apply things depending on the requirements even if those things are not CP or AP.

mohanmca 10y ago

Great work!

jcadam 10y ago

On my personal projects, my default database choice is Postgres unless I have a glaring, obvious reason to choose something NoSQL (in which case I like CouchDB if the situation calls for a document store, don't know much about the others). At work, it doesn't matter if we have a perfect use case for a particular NoSQL solution (I'm working on something now that I think would be a great fit for a Graph database like Neo4J). We use Oracle, because it's expensive, we're already paying for it, so damn it, we're going to use it.

DiabloD3 10y ago
I agree. However, re: document store, I've been bouncing Postgres around in my head for that: all required columns (ie, columns that always exist, or will be queried on) are normal SQL columns, everything else is stashed in one or more hstore columns.
- StavrosK 10y ago
  > everything else is stashed in one or more hstore columns. This has the same problem as most NoSQL solutions: You don't get the benefit of a schema, and schemas are damn useful things.
  - elmigranto 10y ago
    You can define arbitrary constraints of any level of complexity with Postgres for any column, be it plain int, hstore, jsonb or anything else. Most of that will also be efficiently indexable out of the box.
  - DiabloD3 10y ago
    Not everything needs a schema. The alternative I've also been bouncing around in my head is multiple tables: objects (rows in the "primary" table) have traits (rows in "secondary" tables with identical primary keys) that break out columns according to what they do. Having 9000 columns in a single table is a bit unwieldy.
    - 10y ago

deleted 10y ago

[deleted]

cwmma 10y ago

I'd point out that there are arguments[1] that 'CA' is not a real choose you can make when it comes to the CAP theorem. 1: https://codahale.com/you-cant-sacrifice-partition-tolerance/ https://codahale.com/you-cant-sacrifice-partition-tolerance/

morrbo 10y ago

serious question, what is to stop everyone (where a K/V store is not enough) from just chucking some form of SQL server in a VM in a ramdisk and using that? I've always wondered about that idea

virmundi 10y ago
Lost efficiency. If you want a semi-persistent relational DB, you can use BerkleyDB. There are also embedded DBs for Java, and probably other languages. If you do this, you're not wasting RAM for the VM and the guest OS. The DB gets its memory from the large application install. edited for typo.

virmundi 10y ago

I enjoyed this topic while researching New Data: a Field Guide [1]. In my research I've come to really enjoy ArangoDB. It has good support for documents, performance, relational data queries and basic graphs. Without doing the research I don't think I would have ever dug into the new tools. If you ever get a chance to do a small prototype, looking using these tools. 1- https://leanpub.com/NewDataAFieldGuide https://leanpub.com/NewDataAFieldGuide

Feneric 10y ago

Not even a mention of ZODB. While it's true it's Python-only, it's certainly one of the most mature NoSQL databases around.

lgas 10y ago
The context the author is describing is clustered databases, does ZODB offer a clustering option that is equally as mature?
- Feneric 10y ago
  You mean like ZEO? Yes, it's probably been around for a couple of decades now.
  - lgas 10y ago
    I don't think so. I am referring to something that allows you to scale your database cluster horizontally by adding more servers. ZEO appears to only support a single server.

abannin 10y ago

This is a fantastic overview of the issues surrounding DB's. Would love to see some deeper discussions on graph databases in the next version!

antonios 10y ago

This article will soon become outdated: CouchDB 2.0 is in release candidate stage, and will soon be released. It features built-in and automatic sharding and clustering and a new, MongoDB-inspired document query language...and lots of small improvements as well. For more information, read the excellent blog posts from their official blog: https://blog.couchdb.org/ https://blog.couchdb.org/

aeharding 10y ago
Anyone looking for more information in CouchDB 2 redundancy should check out this article: http://docs.couchdb.org/en/1.6.1/cluster/theory.html http://docs.couchdb.org/en/1.6.1/cluster/theory.html (Ignore that it's under /en/1.6.1, built-in clustering is 2+ only.) I'm really excited about CouchDB 2.0.
DivineTraube 10y ago
I'm also very curious about CouchDB 2.0 and whether CouchDB will be able to make a comeback. When I talked to Adam Kocoloski about a year ago, I got the impression that it takes a very good technical approach (consistent hashing, causal hash histories, MongoDB queries, etc.). However, the implementation is now mainly driven by Cloudant and CouchDB kept alive by them. I hope they manage to rekindle the old fascination they once had. In any case I will be happy to update the article and include it.

fizixer 10y ago

Slightly off topic but: As someone out of touch on the topic of DB utilizing apps for a while, what's the update on the ORM impedance mismatch? (I mean the guidelines or best practices).

sandstrom 10y ago

Great to see RethinkDB covered (although it would have been better if it was included in the primary comparison tables too). It feels like a second-generation NoSQL database where most of the drawbacks with earlier NoSQL-databases has been taken care of!

deleted 10y ago

[deleted]

novaleaf 10y ago

boo. doesn't include a comparison vs Google's NoSql SaaS offerings (Cloud DataStore and BigTable). i personally use Cloud DataStore and find it a great fit, wish there was some comparisons against it.

CydeWeys 10y ago
Just wait for Cloud Spanner! http://siliconangle.com/blog/2016/06/27/google-tools-up-with-its-spanner-database-looks-for-a-fight-with-aws/ http://siliconangle.com/blog/2016/06/27/google-tools-up-with...

samwyse 10y ago

The TL;DR was TL;DR. Ho, seriously. About halfway through, the buzzwords and jargon got to be too much for me. Let me know if there's anything interesting Wining.

asdr 10y ago

If you wish an introduction to NoSQL there is a good webinar at http://www.prohuddle.com/webinars/nosql/NoSQL_Distilled.php http://www.prohuddle.com/webinars/nosql/NoSQL_Distilled.php, which includes discussions, examples and comparisons.