12 ms·

MongoDB Performance & Durability

81 points by ericflo 16y ago

forsaken 16y ago
Scary.
- angelbob 16y ago
  Eh. Yes and no. The Mongo guys are very, very upfront about this. Check their web site. There are good workarounds, they just require spending a bit more on hardware.
  - tszming 16y ago
    Replication does not always work if you deploy MongoDB as VM since master and slave instances may be running in a single physical machine.
    - angelbob 16y ago
      Sure. And again, they make that clear. It's quite easy to find out that MongoDB doesn't "have single-server durability". If, like me (and at least one commenter here) you say, "what does that mean?", you can quickly find an explanation on their web site, or in many other places.
  - mikealrogers 16y ago
    that's kind of like saying "we told you our parachutes won't open sometimes. well, not in our conference talks or in our marketing but it's on our wiki and we blogged about it". if you want to see a message that is clearly delivered about (lack of) durability look at memcached. nobody misunderstands memcached's durability/consistency guarantees. -Mikeal (after a few drinks)
    - angelbob 16y ago
      I went to the MongoSF conference. I found they explained it extensively in their conference talks. I don't know about their marketing, but it's easy to find on the web site.
    - kristina 16y ago
      Actually, it's more like saying, "we HIGHLY recommend having a backup parachute" (as, I'd imagine, most skydiving instructors would). Also, I think I've always mention it in my talks. I'm not sure how VoltDB escaped your wrath, it's an in-memory db that claims to be durable.
alexpopescu 16y ago
Posted about this months ago: http://nosql.mypopescu.com/post/392868405/mongodb-durability-a-tradeoff-to-be-aware-of http://nosql.mypopescu.com/post/392868405/mongodb-durability... I do agree durability is important, but as long as you are aware of this behavior and you consider it in your design, you may still find scenarios where the gained speed is a good trade off. Another aspect that tends to be forgot when speaking about MongoDB is its fire-and-forget API. Combined with the behavior of automatic collection creation, innexistent data validation this may lead to "interesting" results.
- jamwt 16y ago
  I'm a big fan of MongoDB, and I think its replicated durability characteristics are good enough for many classes of applications. If you could lose a minute of data and have it just be "bad" instead of "customer enraging and business threatening", then it's a very nice database system. However--I do think the decision to make writes "fire and forget" is just a mistake. If you use abstractions like a connection pool under heavy concurrency, you can get unpredictable behavior in terms of when the data is "actually there." For example, all in one thread: with connection pool: do_write operation A do other things... with connection pool: read something, assuming A has been applied Specifically, the fact that you get an arbitrary connection out of the pool means you cannot be sure that the database has completely processed operation A before executing your new query. MongoDB has a "safe=" flag in their Python bindings that implements the project's official (http://www.mongodb.org/display/DOCS/Last+Error+Commands#LastErrorCommands-UseCases http://www.mongodb.org/display/DOCS/Last+Error+Commands#Last...) recommendation for "it is written" consistency. It's a bit of a hack, but it calls "getLastError()" on the connection that does the write before returning from the update()/save()/insert()/delete() call. I think it's astonishing this behavior isn't the default. In fact, I recently updated diesel's bindings to be safe=True, so getLastError() is always executed on write operations to make sure they succeeded before the call returns: http://github.com/jamwt/diesel/commit/95cd71d82ffc5c308060b651b6d1056dd1908b45 http://github.com/jamwt/diesel/commit/95cd71d82ffc5c308060b6...
  - ericflo 16y ago
    It's not really about losing a minute of data though--the whole thing could become corrupted, and repairing it will be extremely difficult.
dm_mongodb 16y ago
Single server durability is coming, v1.8 in the fall. We chose to prioritize some other things before that. http://jira.mongodb.org/browse/SERVER-980 http://jira.mongodb.org/browse/SERVER-980 We've always clearly said don't build a bond trading system with it. Our philosophy is one-size-fits-all is over; use the right tool for the right problem. Based on how I like to define the term, there is nothing in the NoSQL space that does full "ACID", including complex transactional semantics involving many objects, on many server clusters. That is ok : the perf + scale problem isn't really solvable if you don't give on something.
- ericflo 16y ago
  If MongoDB doesn't have single server durability, what kind of durability does it have?
  - angelbob 16y ago
    Clustered durability. You can request, per-write, "don't return until this data has been persisted to N other servers," with N chosen per-write. So if you're running master/slave and choose N to be 1, you're durable (at the two server level). Run 5-replica sets and choose N to be 3 or 4, and you're basically guaranteed durability. One of your nodes goes down hard and bad things happen? You have four additional clones of it sitting and waiting to be copied. This is actually more durable in cases of extreme hardware failure like, say, your RAID controller going out. However, it requires that you spend more on hardware. So do several other things Mongo does, like using a lot of RAM and disk to get faster writes, so that's in keeping with a lot of their other design decisions. It's not perfect for every project, but it's a great choice for most of the same projects where you'd use Mongo in the first place.
cgbystrom 16y ago
Here we go again. Getting a bit tiresome hearing about this "durability issue". I really don't understand the problem. No, MongoDB isn't durable the same way MyISAM/InnoDB is with MySQL. But I think that is clear as day, certainly no news to me when choosing MongoDB. Been running it in production for 3 months, working fine. If it for some reason would be data loss or corruption, I wouldn't come crying to the devs. I know what risks I took when making the decision. If the backups aren't there I can only blame myself when such event happen.
mjs 16y ago
There's a blog post where the MongoDB people explain why they haven't prioritised single server durability: http://blog.mongodb.org/post/381927266/what-about-durability http://blog.mongodb.org/post/381927266/what-about-durability Basically: (a) for real single server durability you need to turn off hardware buffering or have a battery-backed RAID controller to ensure your write really hit disk; (b) this won't help you if your disks fail, and this failure mode is as likely as any other; and (c) for some applications, the delay required to replay a transaction log in unacceptable--you need 100% uptime. I am thinking though, that the fact that MongoDB writes to files at all is somewhat misleading, and that they may as well say that all your data is loaded into virtual memory. (Since they make no guarantees that the database files will be consistent except in the case of a controlled shutdown. Neither does MySQL, they point out, but I think in practice MySQL database files will be easier to recover from, since their structure is presumably more regular.)
- ww520 16y ago
  It's more interesting to see how they recover in case of failure, rather than how write is completely on disk. Machine failure will happen. Writes will not be fully flushed to disk. It's how one recovers from failure that defines how robust the system is.
  - jchrisa 16y ago
    Gotta say, as a database developer, this is what interests me as well. Let's assume you've had a datacenter-wide power failure, and you're running the repair process on your master and slaves -- what does that look like? How long does it take? What are the factors that influence whether or not it will complete successfully?
- ekidd 16y ago
  We run MongoDB on development machines, and they're frequently shut down unexpectedly. But some our development databases take an hour or two to regenerate from scratch, so we prefer to run recovery. In our experience, MongoDB single-node recovery is very robust. It makes no guarantees of transactional integrity _between_ objects, but in our experience the individual objects have always been recovered intact. According to the MongoDB documentation, recovery will occasionally fail to rebuild objects that span disk pages which were flushed in an inconvenient order, but this will not prevent it from recovering other objects. So even though MongoDB doesn't have single node durability, and you _really_ ought to run it in replicated mode, it actually manages to have a robust recovery tool.
kristina 16y ago
I've tried to post this as a comment on the blog, but it's not showing up (moderated?): ----------- Full disclosure: I work for 10gen. You strategically posted this when my air conditioning was broken, so here are a few thoughts before I go find somewhere cooler. Since CouchDB is "not a competitor" to MongoDB, it's nice of you to put all this time into a public service. > MongoDB, by default, doesn’t actually have a response for writes. Whoopsy, got your emphasis wrong there. We did this to make MongoDB look good in stupid benchmarks (http://www.snailinaturtleneck.com/blog/2009/06/29/couchdb-vs-mongodb-benchmark/ http://www.snailinaturtleneck.com/blog/2009/06/29/couchdb-vs...). Seriously, though, this "unchecked" type of write is just supposed to be for stuff like analytics or sensor data, when you're getting a zillion a second and don't really care some get lost if the server crashes. You can do an insert that not only waits for a database response, but waits for N slaves (user configurable) to have replicated that insert. Note that this is very similar to Cassandra's "write to multiple nodes" promise. You can also fsync after every write. > MongoDB writes to a mem-mapped file and lets the kernel fsync it whenever > the kernel feels like it. fsyncs are configurable. You can fsync once a second, never, or after every single insert, remove, and update if you wish. > When you look at MongoDB more critically I don’t see how you could actually > justify using it for anything resembling the traditional role of a database. This is because you assume you'll run it on single server. MongoDB's documentation clearly, repeatedly, and earnestly tells people to run MongoDB on multiple servers. Also, as another commenter mentioned, full single-server durability is scheduled for the fall. > Stories like this (http://www.korokithakis.net/node/119 http://www.korokithakis.net/node/119) are dubious not > because they expose a few bugs in MongoDB but because they show inherent > architectural problems you cannot overcome long term without something > append-only. Stories "like this" show that MongoDB doesn't work for everyone, particularly people who give no specifics about their architecture, setup, what happened, or anything else. Isn't it irritating how people will write, "MongoDB lost my data" or "CouchDB is really slow" and provide no specifics? That's not to say that things never go wrong, MongoDB is definitely not perfect and has lots of room for improvement. I hope that users with questions and problems will contact us on the list, our wiki, the bug tracker, or IRC (or, heck, write a snarky blog post). Anything to contact the community and let us try to help. I wish every person who tried MongoDB had a great experience with it. Lots of users, hopefully most, love MongoDB and are using it happily and successfully in production.
gfodor 16y ago
Ok, so the standard response seems to be "single server durability" isn't supported, but replication makes up for that. How can this be? If no single server guarantees durability of the writes, how can a cluster of those types of machines suddenly cause those writes to be durable? Maybe it's a pedantic argument, but it seems to me that semantically speaking you are simply relying upon luck that your replicated nodes don't become corrupt or die for some systemtic reason. The fact that there is not a replayable, append-only transaction log says to me that no matter what you build on top of it, it will never by definition be durable because the whole cannot be greater than the sum of its parts in this case.
- wmf 16y ago
  They should say that replication makes up for it under the assumption of uncorrelated failures. As you point out, some people may agree with that assumption and some may not.
  - po 16y ago
    You're right. I've had a whole datacenter lose power. I am definitely not comfortable with assuming that multiple servers somehow magically makes it safe. I think the word "durable" implies that the data is written onto a disk. I think of what they are talking about as "redundant". Redundant not-necessarily-durable data.
    - kristina 16y ago
      Writing to disk and transaction logs are nice, but they aren't magic bullets. What if a data center catches fire? More mundanely, I've heard ~6% of hard drive fail/year. Only replication can help you there. I'd argue that durability is a sliding scale. You have to figure out how much risk you're willing to take and you cannot have a perfectly durable system.
      - mikealrogers 16y ago
        traditionally durability isn't considered a sliding scale, it's a goal/priority which requires you to implement multiple features and fallbacks to handle everything from invalid writes, crash during write, to the data center catching on fire. thinking about durability this way may work great for MongoDB but it isn't how durability is framed in the rest of the database world.
cloudkj 16y ago
Actually, I feel that these are some of the reasons that I find MongoDB to be an attractive solution for certain purposes. You can give up some facets of durability and make up for it in ease and performance. MongoDB actually fits quite well into various web apps that have a need for a secondary, non-authoritative datastore that needs to be highly available but not as reliable. Any kind of real-time stream feature would probably benefit from this. Just keep an authoritative copy of your data in your relational datastore of your choice, and asynchronously replicate data as needed to MongoDB. Since it's a non-authoritative datastore, you can denormalize it as much as you need. Works pretty well, IMHO.
deleted 16y ago
[deleted]
mathias_10gen 16y ago
Since Mikeal decided to take down the comments I've archived what I had in an open tab and put it up on s3: http://bit.ly/diuWH8 http://bit.ly/diuWH8. If anyone has a newer snapshot please let me know and I'll update.

81 points by ericflo 16y ago

forsaken 16y ago

Scary.

angelbob 16y ago
Eh. Yes and no. The Mongo guys are very, very upfront about this. Check their web site. There are good workarounds, they just require spending a bit more on hardware.
- tszming 16y ago
  Replication does not always work if you deploy MongoDB as VM since master and slave instances may be running in a single physical machine.
  - angelbob 16y ago
    Sure. And again, they make that clear. It's quite easy to find out that MongoDB doesn't "have single-server durability". If, like me (and at least one commenter here) you say, "what does that mean?", you can quickly find an explanation on their web site, or in many other places.
- mikealrogers 16y ago
  that's kind of like saying "we told you our parachutes won't open sometimes. well, not in our conference talks or in our marketing but it's on our wiki and we blogged about it". if you want to see a message that is clearly delivered about (lack of) durability look at memcached. nobody misunderstands memcached's durability/consistency guarantees. -Mikeal (after a few drinks)
  - angelbob 16y ago
    I went to the MongoSF conference. I found they explained it extensively in their conference talks. I don't know about their marketing, but it's easy to find on the web site.
  - kristina 16y ago
    Actually, it's more like saying, "we HIGHLY recommend having a backup parachute" (as, I'd imagine, most skydiving instructors would). Also, I think I've always mention it in my talks. I'm not sure how VoltDB escaped your wrath, it's an in-memory db that claims to be durable.

alexpopescu 16y ago

Posted about this months ago: http://nosql.mypopescu.com/post/392868405/mongodb-durability-a-tradeoff-to-be-aware-of http://nosql.mypopescu.com/post/392868405/mongodb-durability... I do agree durability is important, but as long as you are aware of this behavior and you consider it in your design, you may still find scenarios where the gained speed is a good trade off. Another aspect that tends to be forgot when speaking about MongoDB is its fire-and-forget API. Combined with the behavior of automatic collection creation, innexistent data validation this may lead to "interesting" results.

jamwt 16y ago
I'm a big fan of MongoDB, and I think its replicated durability characteristics are good enough for many classes of applications. If you could lose a minute of data and have it just be "bad" instead of "customer enraging and business threatening", then it's a very nice database system. However--I do think the decision to make writes "fire and forget" is just a mistake. If you use abstractions like a connection pool under heavy concurrency, you can get unpredictable behavior in terms of when the data is "actually there." For example, all in one thread: with connection pool: do_write operation A do other things... with connection pool: read something, assuming A has been applied Specifically, the fact that you get an arbitrary connection out of the pool means you cannot be sure that the database has completely processed operation A before executing your new query. MongoDB has a "safe=" flag in their Python bindings that implements the project's official (http://www.mongodb.org/display/DOCS/Last+Error+Commands#LastErrorCommands-UseCases http://www.mongodb.org/display/DOCS/Last+Error+Commands#Last...) recommendation for "it is written" consistency. It's a bit of a hack, but it calls "getLastError()" on the connection that does the write before returning from the update()/save()/insert()/delete() call. I think it's astonishing this behavior isn't the default. In fact, I recently updated diesel's bindings to be safe=True, so getLastError() is always executed on write operations to make sure they succeeded before the call returns: http://github.com/jamwt/diesel/commit/95cd71d82ffc5c308060b651b6d1056dd1908b45 http://github.com/jamwt/diesel/commit/95cd71d82ffc5c308060b6...
- ericflo 16y ago
  It's not really about losing a minute of data though--the whole thing could become corrupted, and repairing it will be extremely difficult.

dm_mongodb 16y ago

Single server durability is coming, v1.8 in the fall. We chose to prioritize some other things before that. http://jira.mongodb.org/browse/SERVER-980 http://jira.mongodb.org/browse/SERVER-980 We've always clearly said don't build a bond trading system with it. Our philosophy is one-size-fits-all is over; use the right tool for the right problem. Based on how I like to define the term, there is nothing in the NoSQL space that does full "ACID", including complex transactional semantics involving many objects, on many server clusters. That is ok : the perf + scale problem isn't really solvable if you don't give on something.

ericflo 16y ago
If MongoDB doesn't have single server durability, what kind of durability does it have?
- angelbob 16y ago
  Clustered durability. You can request, per-write, "don't return until this data has been persisted to N other servers," with N chosen per-write. So if you're running master/slave and choose N to be 1, you're durable (at the two server level). Run 5-replica sets and choose N to be 3 or 4, and you're basically guaranteed durability. One of your nodes goes down hard and bad things happen? You have four additional clones of it sitting and waiting to be copied. This is actually more durable in cases of extreme hardware failure like, say, your RAID controller going out. However, it requires that you spend more on hardware. So do several other things Mongo does, like using a lot of RAM and disk to get faster writes, so that's in keeping with a lot of their other design decisions. It's not perfect for every project, but it's a great choice for most of the same projects where you'd use Mongo in the first place.

cgbystrom 16y ago

Here we go again. Getting a bit tiresome hearing about this "durability issue". I really don't understand the problem. No, MongoDB isn't durable the same way MyISAM/InnoDB is with MySQL. But I think that is clear as day, certainly no news to me when choosing MongoDB. Been running it in production for 3 months, working fine. If it for some reason would be data loss or corruption, I wouldn't come crying to the devs. I know what risks I took when making the decision. If the backups aren't there I can only blame myself when such event happen.

mjs 16y ago

There's a blog post where the MongoDB people explain why they haven't prioritised single server durability: http://blog.mongodb.org/post/381927266/what-about-durability http://blog.mongodb.org/post/381927266/what-about-durability Basically: (a) for real single server durability you need to turn off hardware buffering or have a battery-backed RAID controller to ensure your write really hit disk; (b) this won't help you if your disks fail, and this failure mode is as likely as any other; and (c) for some applications, the delay required to replay a transaction log in unacceptable--you need 100% uptime. I am thinking though, that the fact that MongoDB writes to files at all is somewhat misleading, and that they may as well say that all your data is loaded into virtual memory. (Since they make no guarantees that the database files will be consistent except in the case of a controlled shutdown. Neither does MySQL, they point out, but I think in practice MySQL database files will be easier to recover from, since their structure is presumably more regular.)

ww520 16y ago
It's more interesting to see how they recover in case of failure, rather than how write is completely on disk. Machine failure will happen. Writes will not be fully flushed to disk. It's how one recovers from failure that defines how robust the system is.
- jchrisa 16y ago
  Gotta say, as a database developer, this is what interests me as well. Let's assume you've had a datacenter-wide power failure, and you're running the repair process on your master and slaves -- what does that look like? How long does it take? What are the factors that influence whether or not it will complete successfully?
ekidd 16y ago
We run MongoDB on development machines, and they're frequently shut down unexpectedly. But some our development databases take an hour or two to regenerate from scratch, so we prefer to run recovery. In our experience, MongoDB single-node recovery is very robust. It makes no guarantees of transactional integrity _between_ objects, but in our experience the individual objects have always been recovered intact. According to the MongoDB documentation, recovery will occasionally fail to rebuild objects that span disk pages which were flushed in an inconvenient order, but this will not prevent it from recovering other objects. So even though MongoDB doesn't have single node durability, and you _really_ ought to run it in replicated mode, it actually manages to have a robust recovery tool.

kristina 16y ago

I've tried to post this as a comment on the blog, but it's not showing up (moderated?): ----------- Full disclosure: I work for 10gen. You strategically posted this when my air conditioning was broken, so here are a few thoughts before I go find somewhere cooler. Since CouchDB is "not a competitor" to MongoDB, it's nice of you to put all this time into a public service. > MongoDB, by default, doesn’t actually have a response for writes. Whoopsy, got your emphasis wrong there. We did this to make MongoDB look good in stupid benchmarks (http://www.snailinaturtleneck.com/blog/2009/06/29/couchdb-vs-mongodb-benchmark/ http://www.snailinaturtleneck.com/blog/2009/06/29/couchdb-vs...). Seriously, though, this "unchecked" type of write is just supposed to be for stuff like analytics or sensor data, when you're getting a zillion a second and don't really care some get lost if the server crashes. You can do an insert that not only waits for a database response, but waits for N slaves (user configurable) to have replicated that insert. Note that this is very similar to Cassandra's "write to multiple nodes" promise. You can also fsync after every write. > MongoDB writes to a mem-mapped file and lets the kernel fsync it whenever > the kernel feels like it. fsyncs are configurable. You can fsync once a second, never, or after every single insert, remove, and update if you wish. > When you look at MongoDB more critically I don’t see how you could actually > justify using it for anything resembling the traditional role of a database. This is because you assume you'll run it on single server. MongoDB's documentation clearly, repeatedly, and earnestly tells people to run MongoDB on multiple servers. Also, as another commenter mentioned, full single-server durability is scheduled for the fall. > Stories like this (http://www.korokithakis.net/node/119 http://www.korokithakis.net/node/119) are dubious not > because they expose a few bugs in MongoDB but because they show inherent > architectural problems you cannot overcome long term without something > append-only. Stories "like this" show that MongoDB doesn't work for everyone, particularly people who give no specifics about their architecture, setup, what happened, or anything else. Isn't it irritating how people will write, "MongoDB lost my data" or "CouchDB is really slow" and provide no specifics? That's not to say that things never go wrong, MongoDB is definitely not perfect and has lots of room for improvement. I hope that users with questions and problems will contact us on the list, our wiki, the bug tracker, or IRC (or, heck, write a snarky blog post). Anything to contact the community and let us try to help. I wish every person who tried MongoDB had a great experience with it. Lots of users, hopefully most, love MongoDB and are using it happily and successfully in production.

gfodor 16y ago

Ok, so the standard response seems to be "single server durability" isn't supported, but replication makes up for that. How can this be? If no single server guarantees durability of the writes, how can a cluster of those types of machines suddenly cause those writes to be durable? Maybe it's a pedantic argument, but it seems to me that semantically speaking you are simply relying upon luck that your replicated nodes don't become corrupt or die for some systemtic reason. The fact that there is not a replayable, append-only transaction log says to me that no matter what you build on top of it, it will never by definition be durable because the whole cannot be greater than the sum of its parts in this case.

wmf 16y ago
They should say that replication makes up for it under the assumption of uncorrelated failures. As you point out, some people may agree with that assumption and some may not.
- po 16y ago
  You're right. I've had a whole datacenter lose power. I am definitely not comfortable with assuming that multiple servers somehow magically makes it safe. I think the word "durable" implies that the data is written onto a disk. I think of what they are talking about as "redundant". Redundant not-necessarily-durable data.
  - kristina 16y ago
    Writing to disk and transaction logs are nice, but they aren't magic bullets. What if a data center catches fire? More mundanely, I've heard ~6% of hard drive fail/year. Only replication can help you there. I'd argue that durability is a sliding scale. You have to figure out how much risk you're willing to take and you cannot have a perfectly durable system.
    - mikealrogers 16y ago
      traditionally durability isn't considered a sliding scale, it's a goal/priority which requires you to implement multiple features and fallbacks to handle everything from invalid writes, crash during write, to the data center catching on fire. thinking about durability this way may work great for MongoDB but it isn't how durability is framed in the rest of the database world.

cloudkj 16y ago

Actually, I feel that these are some of the reasons that I find MongoDB to be an attractive solution for certain purposes. You can give up some facets of durability and make up for it in ease and performance. MongoDB actually fits quite well into various web apps that have a need for a secondary, non-authoritative datastore that needs to be highly available but not as reliable. Any kind of real-time stream feature would probably benefit from this. Just keep an authoritative copy of your data in your relational datastore of your choice, and asynchronously replicate data as needed to MongoDB. Since it's a non-authoritative datastore, you can denormalize it as much as you need. Works pretty well, IMHO.

deleted 16y ago

[deleted]

mathias_10gen 16y ago

Since Mikeal decided to take down the comments I've archived what I had in an open tab and put it up on s3: http://bit.ly/diuWH8 http://bit.ly/diuWH8. If anyone has a newer snapshot please let me know and I'll update.