5 ms·

From the beginning I've understood mongodb to be built with it's approach for scaling, performance, redundancy and backup to be horizontal scaling. They recentl

From the beginning I've understood mongodb to be built with it's approach for scaling, performance, redundancy and backup to be horizontal scaling. They recently added journaling for single server durability, but before that replication was how you made sure you data was safe. It seems to me when I see complaints about mongodb it's because people don't want to horizontally scale it and instead believe vertical scaling should be more available. Just seems to me people don't like how mongodb is built, but if used as intended I think mongodb performs as advertised. In most cases I don't think it's the tool, rather the one using it.

LeafStorm 15y ago
While I agree that you should definitely use your tools in the best way you're capable of, I think for most people there's a baseline expectation that if you save data in a database, that data will be safe (at the very least, recoverable) unless something happens like the server catching on fire. Nearly every other major database has this as the default -- MySQL, PostgreSQL, CouchDB, and Berkeley DB to name a few. (Redis doesn't, but it's also very upfront about it, and does provide this kind of durability as an option from early on.) So when MongoDB breaks this expectation, and when asked to support it as an option, just says, "That's what more hardware is for," it's a pretty big turnoff.
- antirez 15y ago
  You are wrong about Redis. Cheers
  - justinsb 15y ago
    Care to explain? I believe for Redis, "appendfsync everysec" is the default. The poster's point was that MySQL and Postgres both ship with something like "appendfsync always", and you have to opt-in to the the less safe mode if you want to get more performance. Redis ships with the less safe mode pre-selected, and so has higher performance out-of-the-box.
    - corford 15y ago
      You're right. The Postgres equivalent to "appendfsync always" is "synchronous_commit = on". Which AFAIK is the default. However, one of the nice things about redis is that even if you run "appendfsync everysec" you never run the risk of corruption. You're only risk is losing a maximum of 2 seconds worth of data. If you missed it, there's a wonderful blog post by antirez covering all of this (and a lot more) here: http://antirez.com/post/redis-persistence-demystified.html http://antirez.com/post/redis-persistence-demystified.html
      - justinsb 15y ago
        All modern relational databases use this approach (it's called ARIES), so all offer the same guarantees as Redis.
        moe 15y ago
        And to praise redis some more; it also comes with the sanest (async) replication support of any database. By far. It's literally a one-liner in the config file. No bootstrap needed, maintenance free, absolutely no strings attached. 10 seconds and you're done. Which means you'll actually use it from day 1 and never worry about it. Compare that to any other database.
    - deleted 15y ago
      [deleted]
    - antirez 15y ago
      What I mean is simply that the default configuration file does not matter, but the point is, what are the guarantees and tradeoffs that a database system can provide? and Redis is no different than other non-in-memory DBs about this. To put Redis apart because in the default configuration file we just enable RDB persistence seems odd, it's like if with a one line commit tomorrow I "change Redis" because I alter the default configuration to AOF + fsync always. Obviously the system remains exactly the same, so systems should be compared by capabilities and not by default config files.
- mnutt 15y ago
  There's actually two separate concerns: Mongo client's "safe" operator causes the client to see if the database threw an error and throw an error itself. Like someone mentioned, it mostly falls on the mongo clients to implement this. We mostly use fire-and-forget for our application, since it is just logging stats and speed is more important than losing an increment here or there. There should probably be better documentation telling people to always use the safe operations for important data. There is also the durability issue. Early versions shipped with durability turned off by default and required replication to maintain durability. Mongo has had the journal feature since 1.8 and has it enabled by default since 1.9.2. (current version is 2.0) So while mongo has definitely been unsafe in the past, both kinds of safety are now supported, and one is default. The other is either not that big a deal or egregious, depending on the way you view mongo.
paulsutter 15y ago
You're right that MongoDB is designed from the start for horizontal scaling, and the author could have better articulated his reluctance to add machines. However, his suggestions would improve mongodb performance in all configurations, not limited to single node. The mongodb guys did a great job simplfying the internal architecture and focusing on a great API to get the product out quickly. I'm confident they can make these sorts of internal improvements to broaden the use of mongodb.
- heretohelp 15y ago
  >You're right that MongoDB is designed from the start for horizontal scaling, Nope.
mitchellh 15y ago
[Note: I wrote the blog post] I'm not at all against horizontally scaling. However, I don't believe that horizontally scaling should be necessary doing a mere 200 updates to per second to a data store that isn't even fsyncing writes to disk. Think of it in terms of cost per ops. Let's just say 200 update ops per second is the point at which you need to shard (not scientific, but let's just use that as a benchmark since that is what we saw at Kiip). MongoDB likes memory, so let's use high-memory AWS instances as a cost benchmark. I think this is fair since MongoDB advertises itself as a DB built for the cloud. The cheapest high-memory instance is around $330/month. That gives you a cost per op of 6.37e-5 cents per update operation. Let's compare this to PostgreSQL, which we've had in production for a couple months at Kiip now. Our PostgreSQL master server has peaked at around 1000 updates per second without issue, and also with the bonus that it doesn't block reads for no reason. The cost per op for PostgreSQL is 1.27e-5 cents. Therefore, if you're swimming in money, then MongoDB seems like a great way to scale. However, we try to be more efficient with our infrastructure expenditures. EDIT: Updated numbers, math is hard.
- zzzeek 15y ago
  typo in the numbers ? I can get your mongoDB number but not pg: >>> seconds_per_month = 60 * 60 * 24 * 30 >>> ops_per_month_pg = 1000 * seconds_per_month >>> ops_per_month_mg = 200 * seconds_per_month >>> 330.0 / ops_per_month_pg * 100 1.2731481481481482e-05 >>> 330.0 / ops_per_month_mg * 100 6.36574074074074e-05
  - mitchellh 15y ago
    You're absolutely right. Decimal points are hard. Fixed my comment (and noted it).
  - nivertech 15y ago
    You could have skipped all the calculations and just say, that 1000/200=5, I.e PostgreSQL 5 times more cost effective than MongoDB.
- jabwork 15y ago
  Can you give any sort of indication of the value of a schemaless database and the flexibility it provided as the team fleshed out the data model? Was this a mere convenience over traditional schema migration or something more?
cheald 15y ago
I think the article is a fair criticism, and I think your response is likewise fair. Mongo was built with horizontal scaling in mind, and to that end, it tends to suffer noticeably when you overload a single node. Things like the global write lock and single-threaded map/reduce are definitely problems, and shouldn't be pooh-pooh'd away as "oh, just scale horizontally". Uncompacted key names are a real problem, and a consequence of it is that Mongo tends to take more disk/RAM for the same data (+indexes) than a comparable RDBMS does. Maintenance is clunky - you end up having to repair or compact each slave individually, then step down your master and repair or compact it, then step it back up (which does work decently, but is time consuming!) None of these are showstoppers, but they are pain points, especially if you're riding the "Mongo is magical scaling sauce" hype train. It takes a lot of work to really get it humming, and once you do, it's pretty damn good at the things it's good at, but there are gotchas and they will bite you if you aren't looking out for them.
- ericflo 15y ago
  Mongo was not designed with horizontal scaling in mind. Riak, Cassandra, HBase, Project Voldemort...these are the projects that were designed with horizontal scaling in mind (as evidenced by their architectures.) But not Mongo.
  - jasonmccay 15y ago
    I have to respectfully disagree. Your comment is a bit sweeping and I really don't think that MongoDB's sharding solution is a bad one ... simply the strategies are different. There is a large set of nice features that makes Mongo, for most people, nice to use long before you even need to address sharding. The percentage of people that will need to shard is much lower than the percentage of people that can get considerable benefits of how you can query data in MongoDB, for example, vs. Riak. You are such a Riak-lover. :P
    - ericflo 15y ago
      I am definitely a Riak-lover :) And I also agree that there is a large set of nice features that make Mongo nice to use before sharding.
saurik 15y ago
The behavior described in the article, though, is just a fundamental misuse of the hardware resources. If I'm hitting write bottlenecks, I need to shard: that's just a fact of reality. However, write throughput and read throughput should be unrelated (as durable writes are bottlenecks by disk and reads of hot data are bottlenecked by CPU/RAM): if I have saturated five machines with writes I should have five machines worth of read throughput, and that is a /lot/ of throughput... with the behavior in this article, you have no read capacity as you are blocked by writes, even if the writes are to unrelated collections of data (which is downright non-sensical behavior). Even with more machines (I guess, twice as many), the latency of your reads is now being effected horribly by having to wait for the write locks... it is just a bad solution.

LeafStorm 15y ago

While I agree that you should definitely use your tools in the best way you're capable of, I think for most people there's a baseline expectation that if you save data in a database, that data will be safe (at the very least, recoverable) unless something happens like the server catching on fire. Nearly every other major database has this as the default -- MySQL, PostgreSQL, CouchDB, and Berkeley DB to name a few. (Redis doesn't, but it's also very upfront about it, and does provide this kind of durability as an option from early on.) So when MongoDB breaks this expectation, and when asked to support it as an option, just says, "That's what more hardware is for," it's a pretty big turnoff.

antirez 15y ago
You are wrong about Redis. Cheers
- justinsb 15y ago
  Care to explain? I believe for Redis, "appendfsync everysec" is the default. The poster's point was that MySQL and Postgres both ship with something like "appendfsync always", and you have to opt-in to the the less safe mode if you want to get more performance. Redis ships with the less safe mode pre-selected, and so has higher performance out-of-the-box.
  - corford 15y ago
    You're right. The Postgres equivalent to "appendfsync always" is "synchronous_commit = on". Which AFAIK is the default. However, one of the nice things about redis is that even if you run "appendfsync everysec" you never run the risk of corruption. You're only risk is losing a maximum of 2 seconds worth of data. If you missed it, there's a wonderful blog post by antirez covering all of this (and a lot more) here: http://antirez.com/post/redis-persistence-demystified.html http://antirez.com/post/redis-persistence-demystified.html
    - justinsb 15y ago
      All modern relational databases use this approach (it's called ARIES), so all offer the same guarantees as Redis.
      - moe 15y ago
        And to praise redis some more; it also comes with the sanest (async) replication support of any database. By far. It's literally a one-liner in the config file. No bootstrap needed, maintenance free, absolutely no strings attached. 10 seconds and you're done. Which means you'll actually use it from day 1 and never worry about it. Compare that to any other database.
  - deleted 15y ago
    [deleted]
  - antirez 15y ago
    What I mean is simply that the default configuration file does not matter, but the point is, what are the guarantees and tradeoffs that a database system can provide? and Redis is no different than other non-in-memory DBs about this. To put Redis apart because in the default configuration file we just enable RDB persistence seems odd, it's like if with a one line commit tomorrow I "change Redis" because I alter the default configuration to AOF + fsync always. Obviously the system remains exactly the same, so systems should be compared by capabilities and not by default config files.
mnutt 15y ago
There's actually two separate concerns: Mongo client's "safe" operator causes the client to see if the database threw an error and throw an error itself. Like someone mentioned, it mostly falls on the mongo clients to implement this. We mostly use fire-and-forget for our application, since it is just logging stats and speed is more important than losing an increment here or there. There should probably be better documentation telling people to always use the safe operations for important data. There is also the durability issue. Early versions shipped with durability turned off by default and required replication to maintain durability. Mongo has had the journal feature since 1.8 and has it enabled by default since 1.9.2. (current version is 2.0) So while mongo has definitely been unsafe in the past, both kinds of safety are now supported, and one is default. The other is either not that big a deal or egregious, depending on the way you view mongo.

paulsutter 15y ago

You're right that MongoDB is designed from the start for horizontal scaling, and the author could have better articulated his reluctance to add machines. However, his suggestions would improve mongodb performance in all configurations, not limited to single node. The mongodb guys did a great job simplfying the internal architecture and focusing on a great API to get the product out quickly. I'm confident they can make these sorts of internal improvements to broaden the use of mongodb.

heretohelp 15y ago
>You're right that MongoDB is designed from the start for horizontal scaling, Nope.

mitchellh 15y ago

[Note: I wrote the blog post] I'm not at all against horizontally scaling. However, I don't believe that horizontally scaling should be necessary doing a mere 200 updates to per second to a data store that isn't even fsyncing writes to disk. Think of it in terms of cost per ops. Let's just say 200 update ops per second is the point at which you need to shard (not scientific, but let's just use that as a benchmark since that is what we saw at Kiip). MongoDB likes memory, so let's use high-memory AWS instances as a cost benchmark. I think this is fair since MongoDB advertises itself as a DB built for the cloud. The cheapest high-memory instance is around $330/month. That gives you a cost per op of 6.37e-5 cents per update operation. Let's compare this to PostgreSQL, which we've had in production for a couple months at Kiip now. Our PostgreSQL master server has peaked at around 1000 updates per second without issue, and also with the bonus that it doesn't block reads for no reason. The cost per op for PostgreSQL is 1.27e-5 cents. Therefore, if you're swimming in money, then MongoDB seems like a great way to scale. However, we try to be more efficient with our infrastructure expenditures. EDIT: Updated numbers, math is hard.

zzzeek 15y ago
typo in the numbers ? I can get your mongoDB number but not pg: >>> seconds_per_month = 60 * 60 * 24 * 30 >>> ops_per_month_pg = 1000 * seconds_per_month >>> ops_per_month_mg = 200 * seconds_per_month >>> 330.0 / ops_per_month_pg * 100 1.2731481481481482e-05 >>> 330.0 / ops_per_month_mg * 100 6.36574074074074e-05
- mitchellh 15y ago
  You're absolutely right. Decimal points are hard. Fixed my comment (and noted it).
- nivertech 15y ago
  You could have skipped all the calculations and just say, that 1000/200=5, I.e PostgreSQL 5 times more cost effective than MongoDB.
jabwork 15y ago
Can you give any sort of indication of the value of a schemaless database and the flexibility it provided as the team fleshed out the data model? Was this a mere convenience over traditional schema migration or something more?

cheald 15y ago

I think the article is a fair criticism, and I think your response is likewise fair. Mongo was built with horizontal scaling in mind, and to that end, it tends to suffer noticeably when you overload a single node. Things like the global write lock and single-threaded map/reduce are definitely problems, and shouldn't be pooh-pooh'd away as "oh, just scale horizontally". Uncompacted key names are a real problem, and a consequence of it is that Mongo tends to take more disk/RAM for the same data (+indexes) than a comparable RDBMS does. Maintenance is clunky - you end up having to repair or compact each slave individually, then step down your master and repair or compact it, then step it back up (which does work decently, but is time consuming!) None of these are showstoppers, but they are pain points, especially if you're riding the "Mongo is magical scaling sauce" hype train. It takes a lot of work to really get it humming, and once you do, it's pretty damn good at the things it's good at, but there are gotchas and they will bite you if you aren't looking out for them.

ericflo 15y ago
Mongo was not designed with horizontal scaling in mind. Riak, Cassandra, HBase, Project Voldemort...these are the projects that were designed with horizontal scaling in mind (as evidenced by their architectures.) But not Mongo.
- jasonmccay 15y ago
  I have to respectfully disagree. Your comment is a bit sweeping and I really don't think that MongoDB's sharding solution is a bad one ... simply the strategies are different. There is a large set of nice features that makes Mongo, for most people, nice to use long before you even need to address sharding. The percentage of people that will need to shard is much lower than the percentage of people that can get considerable benefits of how you can query data in MongoDB, for example, vs. Riak. You are such a Riak-lover. :P
  - ericflo 15y ago
    I am definitely a Riak-lover :) And I also agree that there is a large set of nice features that make Mongo nice to use before sharding.

saurik 15y ago

The behavior described in the article, though, is just a fundamental misuse of the hardware resources. If I'm hitting write bottlenecks, I need to shard: that's just a fact of reality. However, write throughput and read throughput should be unrelated (as durable writes are bottlenecks by disk and reads of hot data are bottlenecked by CPU/RAM): if I have saturated five machines with writes I should have five machines worth of read throughput, and that is a /lot/ of throughput... with the behavior in this article, you have no read capacity as you are blocked by writes, even if the writes are to unrelated collections of data (which is downright non-sensical behavior). Even with more machines (I guess, twice as many), the latency of your reads is now being effected horribly by having to wait for the write locks... it is just a bad solution.