7 ms·

MongoDB, Data Durability and Improvements coming in 1.8

100 points by mattyb 16y ago

noelwelsh 16y ago
I'm pleased Mongo is getting single server durability. I have never understood why it got so popular without this feature. I'd love to know why people choose Mongo over say, Riak, or CouchDB, as the majority of projects don't need more than one server.
- stilkov 16y ago
  Would that be that majority that doesn't really care about uptime?
  - deleted 16y ago
    [deleted]
- ericflo 16y ago
  Because they market infinite scalability, insane speeds, and they have a nice API. But people don't realize that, while the speed is fast, you're driving without a seatbelt, and the scalability story is more or less false. They do have a nice API though.
  - prosa 16y ago
    Can you point us toward data that backs up your point that "the scalability story is more or less false"? Seems like FUD. There are several well-documented, major production deployments of MongoDB already, that seem to contradict your assertion: * Etsy * CERN * BoxedIce (600MM+ documents) * BuzzFeed (400MM+ datapoints/month) That's just a few from http://www.mongodb.org/display/DOCS/Production+Deployments http://www.mongodb.org/display/DOCS/Production+Deployments
- weixiyen 16y ago
  For those projects I just ran backups of the file system. Linode does it for $5/month. If your data was worth anything you'd also be using replication in which case the single server durability point becomes moot.
- yummyfajitas 16y ago
  Because MongoDB is webscale. http://www.xtranormal.com/watch/6995033/ http://www.xtranormal.com/watch/6995033/
- harryh 16y ago
  Because single server durability is a myth when hardware can fail at any time.
  - lwat 16y ago
    We're talking about data durability. When my SQL server crashes due to hardware failure I know that when I eventually get it back up and running the data will be consistent with at worst the last couple of transactions being rolled back.
    - harryh 16y ago
      This is most certainly not true. Disks fail in ways where the whole volume becomes unreadable all the time.
      - lwat 16y ago
        What I'm saying is that I can go up to my SQL server and disconnect the power cord and my database will not be corrupt when I start it back up. Sure if your HDD gets taken out by a meteor then nothing will save you but that's why you have backups.
        bsg75 16y ago
        Not guaranteed. I have had more than one customer experience hardware failure, resulting in a corrupt or suspect SQL Server database, that was unrecoverable via normal means. In each of the cases where the customer had a true standby system, implemented via replication, log shipping, or mirroring, they were able to failover with little (log shipping) or no data loss. In the cases where they had a single, standalone server, the option was to restore the last known good backup, or sent the database files had to Microsoft for analysis and repair. ANY system (RDBMS, NoSQL, or otherwise), should have a standby replica to prevent data loss. If you data is stored on a single machine, you are doing it wrong.
        lwat 16y ago
- cdavid 16y ago
  I use mongodb to record fine grained log for performance tests, where it works like a charm. It is fragile (in that I have lost full databases when VM were full, for example) I don't know about riak, but we are forced to use couchdb at work, for the wrong reasons I think, and I was not impressed. First, its performances are pretty bad: document insertion is slow unless you batch them, but even though it is slower than in mysql, even though the guarantees about durability are certainly not the same (couchdb was not configured to fsync at each write). Building views is excruciatingly slow, and the view engine does not use all the CPU available (even when IO wait is low, so not a simple IO issue). There are bug reports about this issue (cannot find it at the moment). Replication is not very reliable either for large data (where large against means a few GB, so actually not that large). I am sure we are using couchdb the wrong way at work (out of my hands), and for the wrong application, but its design choices as well as my limited experience does seem to imply couchdb is not adapted for large amount of data, especially one which are written often (there was an interview last summer with D. Katz who said that there were not yet much optimization for large data).
  - cdavid 16y ago
    I cannot seem to edit my post, so here is the issue I am referring to w.r.t. couchdb and unused CPU: http://www.mail-archive.com/dev@couchdb.apache.org/msg06520.html http://www.mail-archive.com/dev@couchdb.apache.org/msg06520.... here is the interview of Damian Katz concerning couchdb: http://howsoftwareisbuilt.com/2010/06/18/interview-with-damien-katz-apache-couchdb http://howsoftwareisbuilt.com/2010/06/18/interview-with-dami..., which mentions that large data is not a focus. There are some companies which seem to use couchdb for large usage, for example bbc (http://enda.squarespace.com/tech/2010/3/4/couchdb-at-scale-4-billion-requests-so-far.html http://enda.squarespace.com/tech/2010/3/4/couchdb-at-scale-4...). I don't know their infrastructure, but they claim to server 4 billions requests as of 4th march 2010 since summer 2009 on a 32 nodes (16 master, 16 backups). Assuming that summer starts in september to get an upper bound of the traffic, this means 250 rq/sec on average, which is nothing impressive for an infrastructure with 32 machines without more information about what they do. Generally, I would not say much about this kind of usecases, but since it is often advertised by couchdb proponents, the burden of the proof is theirs.
- thenduks 16y ago
  You just have to tilt your head a bit and realize that you do need more than one server. For example - I don't know about you but I don't want to take down the site to deploy everytime I have to update some code (esp in the early days, this happens practically once a day). With MongoDB you simply add a cheap second box and use it as at least a replica of your database. When you get tired of "we're updating the site with shiny new code" interrupting your users, you simply make it into a web-node as well.
- va_coder 16y ago
  "I'd love to know why people choose Mongo over say, Riak, or CouchDB" Here are some reasons: * excellent documentation * runs right out of the box * excellent libraries like Mongoid * user can easily perform deep queries (e.g person.address.zip = '90901') * services are available like MongoHQ
JEggers2 16y ago
calling him an idiot? I'm sure this happens a lot, but as you said nobody wants to talk about it for fear of receiving the welcoming carriage that this guy received.
cagenut 16y ago
If I had a nickel for every time a coworker used "-9" for no other reason than being in a rush and not grok'ing what a bad idea it is.
- samstokes 16y ago
  I don't know if this is what happened to the reluctant hero of this story, but if the database really had hung, and wasn't responding to "sensible" methods of shutting down, what are you supposed to do besides kill -9? (By "sensible" I mean something the DB could trap in order to finish saving a consistent state to disk - shutdown command, -INT, -QUIT, etc.) (I'm not excusing trying -9 as a first resort, but if a process is deadlocked (or pegging the CPU in a tight loop) and won't respond to -QUIT, there's not much else you can do.)
prosa 16y ago
This is exciting. MongoDB (and the suite of libraries building up around it) has made prototyping web applications an order of magnitude easier than using SQL. Lack of single-server durability, however, is a showstopper in production, before you've grown enough to justify scaling the database beyond one machine. In my case, that meant going back to MySQL once our schema was finalized (sadly). Looks like that won't be necessary in the not-so-near future, which means MongoDB is usable in new projects without worrying about replication before otherwise necessary.
- utx00 16y ago
  what about couchdb?
  - jkkramer 16y ago
    In my experience, using CouchDB for prototyping is not fun. With SQL and (from what I gather) MongoDB, you can create ad-hoc queries as you go. Creating views in CouchDB requires thought and time -- usually lots of time -- to generate the views, unless you're working with a very small amount of data. Changing your mind is painful. You end up using weird hacks to get around rebuilding your views all the time. Maybe I was doing it wrong, but it seems like if your app has more than a little data and you plan on changing your mind about how best to present it, you will experience discomfort.
    - tmcneal 16y ago
      I don't have any experience with MongoDB, but I am building an application right now using CouchDB and I've found it to be pretty easy to use -- definitely easier to prototype with vs a relational database + ORM framework. CouchDB supports ad-hoc queries in the form of 'temporary views'. The downside (and I think a difference compared with MongoDB) is that temporary views can't be used in production since they are not indexed and thus are much slower than permanent views. I'm developing my application in Python, which has great support for CouchDB in the form of CouchDB-Python (http://packages.python.org/CouchDB/ http://packages.python.org/CouchDB/) and CouchDBKit (http://couchdbkit.org/ http://couchdbkit.org/). Using CouchDBKit, it was pretty easy to set up CouchDB artifacts (map functions, reduce functions, design documents) into a nice file/folder hierarchy and write the simple Python glue code to deploy updates to CouchDB via a single shell command.
megaman821 16y ago
I am never sure what niche MongoDB is supposed to fill. If I want a large cluster to handle "big data" Riak or Cassandra seem to fit the bill better. If I want speed Redis is great. If I want a schema-less SQL-like (but not SQL) database, MongoDB?
- pashields 16y ago
  In my admittedly limited experience with mongo, it felt like they were designing the ultimate database for the web. Not in a "web-scale" sort of way, more in a swiss army database for the web dev way. This came to me when I wanted to do location queries on a db. Most people using standard DBs do this with PostGIS for postgres, which is incredibly powerful and accurate. But mongoDB supports the "find shit near me with good enough accuracy" query that 90% of people want to run right now. I was able to make a simple test page that told you the IP of the previous visitor who was closest to you. It took me thirty minutes from "hey mongo has some sort of geo support" to that (and I'm not a web dev and was using ATT 3G to read doc). PostGIS took me longer to figure out how to set a lat,long pair in the DB. I'm not saying raw speed of development is the best measure of a database, but if your #1 priority is the ability to rapidly prototype, I'd imagine mongo is a good fit in your toolbelt.
  - lwat 16y ago
    I assure you that that anyone who has used any kind of SQL database can do the same stuff you did on Mongo in the same time or less. Development speed is much faster on the SQL database once you have some more data and you can start using joins and searches and all the other nice RDBMS features.
    - steveklabnik 16y ago
      I've built a lot of stuff on RDBMSen, and often, those 'nice features' are actually what I call 'a giant pain in the ass.' Sometimes, data doesn't fit a relational model well. In those cases, something like Mongo is a godsend. And yeah, you could de-normalize your data and get halfway to Mongo, but I'd rather use each tool for what it's good at.
agentultra 16y ago
I agree largely with this post. I'm tired of hearing the same arguments the author has. This technology we use is hardly infallible and as programmers and system administrators of these systems the defects should be obvious! Yet somehow people still think that the "clueless users" are to blame for these "defects." Sure, we all know the saying about assumptions. But why are we at a point where it's cool to use a data persistance layer that assumes it doesn't have to live up to any of those assumptions? It's why we have things like ACID, the LSB, etc. Contracts about how these systems should work for the end user. It's hard to get all that stuff right off the bat, sure. But losing the entire database when a few bytes land in the wrong place? Yikes. Hardly something I'd trust real data to. This stuff can always get better. Of course the solution as pointed out is tried and true. Glad to see that the developers aren't covering their ears...
richcollins 16y ago
You can also use fsync to guarantee that a transaction was written to disk at a severe hit to performance.
- mathias_10gen 16y ago
  Actually that doesn't add any additional durability guarantees in the general case. The only way to use MongoDB durably with a single server is to use the new --dur flag coming in 1.8. I assure you there is no other way.
  - oomkiller 16y ago
    And how far out is 1.8?
SoftwareMaven 16y ago
I read once that every developer ought to use 'kill -9' as their method of stopping services they are writing, then ensure that doing so never causes a problem. Even if you can keep people from executing 'kill -9' (hint: you can't), there are always cases that you can't control that can stop a box in its tracks. By developing explicitly for this, you save you and your customer a lot of pain. Honestly, I would never use a database that doesn't make safeguarding my data as its number one priority. Even if I'm just dumping logs in it, the time I really need to know what's going on is likely the time those safeguards are put to the test.
- steveklabnik 16y ago
  Are you thinking of Fail-fast software? http://en.wikipedia.org/wiki/Fail-fast http://en.wikipedia.org/wiki/Fail-fast
- wmf 16y ago
  http://www.usenix.org/events/hotos03/tech/full_papers/candea/candea_html/ http://www.usenix.org/events/hotos03/tech/full_papers/candea...
- po 16y ago
  It was briefly mentioned in the article, but this is exactly how couchdb shuts down: On-disk, CouchDB never overwrites committed data or associated structures, ensuring the database file is always in a consistent state. This is a “crash-only” design where the CouchDB server does not go through a shut down process, it’s simply terminated. http://couchdb.apache.org/docs/overview.html http://couchdb.apache.org/docs/overview.html (Please, no mongodb vs. couchdb wars here)
  - agazso 16y ago
    This is exactly how all databases work. Well, except for MongoDB ;)
  - SoftwareMaven 16y ago
    I think that is actually where I read it. It must have resonated internally and became the topic (in my mind).

100 points by mattyb 16y ago

I'm pleased Mongo is getting single server durability. I have never understood why it got so popular without this feature. I'd love to know why people choose Mongo over say, Riak, or CouchDB, as the majority of projects don't need more than one server.

stilkov 16y ago
Would that be that majority that doesn't really care about uptime?
- deleted 16y ago
  [deleted]
ericflo 16y ago
Because they market infinite scalability, insane speeds, and they have a nice API. But people don't realize that, while the speed is fast, you're driving without a seatbelt, and the scalability story is more or less false. They do have a nice API though.
- prosa 16y ago
  Can you point us toward data that backs up your point that "the scalability story is more or less false"? Seems like FUD. There are several well-documented, major production deployments of MongoDB already, that seem to contradict your assertion: * Etsy * CERN * BoxedIce (600MM+ documents) * BuzzFeed (400MM+ datapoints/month) That's just a few from http://www.mongodb.org/display/DOCS/Production+Deployments http://www.mongodb.org/display/DOCS/Production+Deployments
weixiyen 16y ago
For those projects I just ran backups of the file system. Linode does it for $5/month. If your data was worth anything you'd also be using replication in which case the single server durability point becomes moot.
yummyfajitas 16y ago
Because MongoDB is webscale. http://www.xtranormal.com/watch/6995033/ http://www.xtranormal.com/watch/6995033/
harryh 16y ago
Because single server durability is a myth when hardware can fail at any time.
- lwat 16y ago
  We're talking about data durability. When my SQL server crashes due to hardware failure I know that when I eventually get it back up and running the data will be consistent with at worst the last couple of transactions being rolled back.
  - harryh 16y ago
    This is most certainly not true. Disks fail in ways where the whole volume becomes unreadable all the time.
    - lwat 16y ago
      What I'm saying is that I can go up to my SQL server and disconnect the power cord and my database will not be corrupt when I start it back up. Sure if your HDD gets taken out by a meteor then nothing will save you but that's why you have backups.
      - bsg75 16y ago
        Not guaranteed. I have had more than one customer experience hardware failure, resulting in a corrupt or suspect SQL Server database, that was unrecoverable via normal means. In each of the cases where the customer had a true standby system, implemented via replication, log shipping, or mirroring, they were able to failover with little (log shipping) or no data loss. In the cases where they had a single, standalone server, the option was to restore the last known good backup, or sent the database files had to Microsoft for analysis and repair. ANY system (RDBMS, NoSQL, or otherwise), should have a standby replica to prevent data loss. If you data is stored on a single machine, you are doing it wrong.
        lwat 16y ago
cdavid 16y ago
I use mongodb to record fine grained log for performance tests, where it works like a charm. It is fragile (in that I have lost full databases when VM were full, for example) I don't know about riak, but we are forced to use couchdb at work, for the wrong reasons I think, and I was not impressed. First, its performances are pretty bad: document insertion is slow unless you batch them, but even though it is slower than in mysql, even though the guarantees about durability are certainly not the same (couchdb was not configured to fsync at each write). Building views is excruciatingly slow, and the view engine does not use all the CPU available (even when IO wait is low, so not a simple IO issue). There are bug reports about this issue (cannot find it at the moment). Replication is not very reliable either for large data (where large against means a few GB, so actually not that large). I am sure we are using couchdb the wrong way at work (out of my hands), and for the wrong application, but its design choices as well as my limited experience does seem to imply couchdb is not adapted for large amount of data, especially one which are written often (there was an interview last summer with D. Katz who said that there were not yet much optimization for large data).
- cdavid 16y ago
  I cannot seem to edit my post, so here is the issue I am referring to w.r.t. couchdb and unused CPU: http://www.mail-archive.com/dev@couchdb.apache.org/msg06520.html http://www.mail-archive.com/dev@couchdb.apache.org/msg06520.... here is the interview of Damian Katz concerning couchdb: http://howsoftwareisbuilt.com/2010/06/18/interview-with-damien-katz-apache-couchdb http://howsoftwareisbuilt.com/2010/06/18/interview-with-dami..., which mentions that large data is not a focus. There are some companies which seem to use couchdb for large usage, for example bbc (http://enda.squarespace.com/tech/2010/3/4/couchdb-at-scale-4-billion-requests-so-far.html http://enda.squarespace.com/tech/2010/3/4/couchdb-at-scale-4...). I don't know their infrastructure, but they claim to server 4 billions requests as of 4th march 2010 since summer 2009 on a 32 nodes (16 master, 16 backups). Assuming that summer starts in september to get an upper bound of the traffic, this means 250 rq/sec on average, which is nothing impressive for an infrastructure with 32 machines without more information about what they do. Generally, I would not say much about this kind of usecases, but since it is often advertised by couchdb proponents, the burden of the proof is theirs.
thenduks 16y ago
You just have to tilt your head a bit and realize that you do need more than one server. For example - I don't know about you but I don't want to take down the site to deploy everytime I have to update some code (esp in the early days, this happens practically once a day). With MongoDB you simply add a cheap second box and use it as at least a replica of your database. When you get tired of "we're updating the site with shiny new code" interrupting your users, you simply make it into a web-node as well.
va_coder 16y ago
"I'd love to know why people choose Mongo over say, Riak, or CouchDB" Here are some reasons: * excellent documentation * runs right out of the box * excellent libraries like Mongoid * user can easily perform deep queries (e.g person.address.zip = '90901') * services are available like MongoHQ

JEggers2 16y ago

calling him an idiot? I'm sure this happens a lot, but as you said nobody wants to talk about it for fear of receiving the welcoming carriage that this guy received.

cagenut 16y ago

If I had a nickel for every time a coworker used "-9" for no other reason than being in a rush and not grok'ing what a bad idea it is.

samstokes 16y ago
I don't know if this is what happened to the reluctant hero of this story, but if the database really had hung, and wasn't responding to "sensible" methods of shutting down, what are you supposed to do besides kill -9? (By "sensible" I mean something the DB could trap in order to finish saving a consistent state to disk - shutdown command, -INT, -QUIT, etc.) (I'm not excusing trying -9 as a first resort, but if a process is deadlocked (or pegging the CPU in a tight loop) and won't respond to -QUIT, there's not much else you can do.)

prosa 16y ago

This is exciting. MongoDB (and the suite of libraries building up around it) has made prototyping web applications an order of magnitude easier than using SQL. Lack of single-server durability, however, is a showstopper in production, before you've grown enough to justify scaling the database beyond one machine. In my case, that meant going back to MySQL once our schema was finalized (sadly). Looks like that won't be necessary in the not-so-near future, which means MongoDB is usable in new projects without worrying about replication before otherwise necessary.

utx00 16y ago
what about couchdb?
- jkkramer 16y ago
  In my experience, using CouchDB for prototyping is not fun. With SQL and (from what I gather) MongoDB, you can create ad-hoc queries as you go. Creating views in CouchDB requires thought and time -- usually lots of time -- to generate the views, unless you're working with a very small amount of data. Changing your mind is painful. You end up using weird hacks to get around rebuilding your views all the time. Maybe I was doing it wrong, but it seems like if your app has more than a little data and you plan on changing your mind about how best to present it, you will experience discomfort.
  - tmcneal 16y ago
    I don't have any experience with MongoDB, but I am building an application right now using CouchDB and I've found it to be pretty easy to use -- definitely easier to prototype with vs a relational database + ORM framework. CouchDB supports ad-hoc queries in the form of 'temporary views'. The downside (and I think a difference compared with MongoDB) is that temporary views can't be used in production since they are not indexed and thus are much slower than permanent views. I'm developing my application in Python, which has great support for CouchDB in the form of CouchDB-Python (http://packages.python.org/CouchDB/ http://packages.python.org/CouchDB/) and CouchDBKit (http://couchdbkit.org/ http://couchdbkit.org/). Using CouchDBKit, it was pretty easy to set up CouchDB artifacts (map functions, reduce functions, design documents) into a nice file/folder hierarchy and write the simple Python glue code to deploy updates to CouchDB via a single shell command.

megaman821 16y ago

I am never sure what niche MongoDB is supposed to fill. If I want a large cluster to handle "big data" Riak or Cassandra seem to fit the bill better. If I want speed Redis is great. If I want a schema-less SQL-like (but not SQL) database, MongoDB?

pashields 16y ago
In my admittedly limited experience with mongo, it felt like they were designing the ultimate database for the web. Not in a "web-scale" sort of way, more in a swiss army database for the web dev way. This came to me when I wanted to do location queries on a db. Most people using standard DBs do this with PostGIS for postgres, which is incredibly powerful and accurate. But mongoDB supports the "find shit near me with good enough accuracy" query that 90% of people want to run right now. I was able to make a simple test page that told you the IP of the previous visitor who was closest to you. It took me thirty minutes from "hey mongo has some sort of geo support" to that (and I'm not a web dev and was using ATT 3G to read doc). PostGIS took me longer to figure out how to set a lat,long pair in the DB. I'm not saying raw speed of development is the best measure of a database, but if your #1 priority is the ability to rapidly prototype, I'd imagine mongo is a good fit in your toolbelt.
- lwat 16y ago
  I assure you that that anyone who has used any kind of SQL database can do the same stuff you did on Mongo in the same time or less. Development speed is much faster on the SQL database once you have some more data and you can start using joins and searches and all the other nice RDBMS features.
  - steveklabnik 16y ago
    I've built a lot of stuff on RDBMSen, and often, those 'nice features' are actually what I call 'a giant pain in the ass.' Sometimes, data doesn't fit a relational model well. In those cases, something like Mongo is a godsend. And yeah, you could de-normalize your data and get halfway to Mongo, but I'd rather use each tool for what it's good at.

agentultra 16y ago

I agree largely with this post. I'm tired of hearing the same arguments the author has. This technology we use is hardly infallible and as programmers and system administrators of these systems the defects should be obvious! Yet somehow people still think that the "clueless users" are to blame for these "defects." Sure, we all know the saying about assumptions. But why are we at a point where it's cool to use a data persistance layer that assumes it doesn't have to live up to any of those assumptions? It's why we have things like ACID, the LSB, etc. Contracts about how these systems should work for the end user. It's hard to get all that stuff right off the bat, sure. But losing the entire database when a few bytes land in the wrong place? Yikes. Hardly something I'd trust real data to. This stuff can always get better. Of course the solution as pointed out is tried and true. Glad to see that the developers aren't covering their ears...

richcollins 16y ago

You can also use fsync to guarantee that a transaction was written to disk at a severe hit to performance.

mathias_10gen 16y ago
Actually that doesn't add any additional durability guarantees in the general case. The only way to use MongoDB durably with a single server is to use the new --dur flag coming in 1.8. I assure you there is no other way.
- oomkiller 16y ago
  And how far out is 1.8?

SoftwareMaven 16y ago

I read once that every developer ought to use 'kill -9' as their method of stopping services they are writing, then ensure that doing so never causes a problem. Even if you can keep people from executing 'kill -9' (hint: you can't), there are always cases that you can't control that can stop a box in its tracks. By developing explicitly for this, you save you and your customer a lot of pain. Honestly, I would never use a database that doesn't make safeguarding my data as its number one priority. Even if I'm just dumping logs in it, the time I really need to know what's going on is likely the time those safeguards are put to the test.

steveklabnik 16y ago
Are you thinking of Fail-fast software? http://en.wikipedia.org/wiki/Fail-fast http://en.wikipedia.org/wiki/Fail-fast
wmf 16y ago
http://www.usenix.org/events/hotos03/tech/full_papers/candea/candea_html/ http://www.usenix.org/events/hotos03/tech/full_papers/candea...
po 16y ago
It was briefly mentioned in the article, but this is exactly how couchdb shuts down: On-disk, CouchDB never overwrites committed data or associated structures, ensuring the database file is always in a consistent state. This is a “crash-only” design where the CouchDB server does not go through a shut down process, it’s simply terminated. http://couchdb.apache.org/docs/overview.html http://couchdb.apache.org/docs/overview.html (Please, no mongodb vs. couchdb wars here)
- agazso 16y ago
  This is exactly how all databases work. Well, except for MongoDB ;)
- SoftwareMaven 16y ago
  I think that is actually where I read it. It must have resonated internally and became the topic (in my mind).