21 ms·

Scaling with MongoDB (or how Urban Airship abandoned it for PostgreSQL)

180 points by rbranson 15y ago

someone13 15y ago
Thanks for this. I've heard lots of good things about MongoDB (and auto-sharding makes me drool), but I've heard rumours of problems with it too. Always cool to read something from the other side of the fence. Too bad that there's no other product out there that makes automatically sharding large amounts of data quite so easy...
- megaman821 15y ago
  Doesn't Riak and Cassandra make it easy? There is just no fast ad-hoc querying in those two.
  - rbranson 15y ago
    Arguably, with a data set of any serious magnitude, there's no such thing as ad-hoc. As soon as an RDBMS is sharded, it loses most of it's ad-hoc powers. That being said, Riak Search allows Riak to perform ad-hoc queries and Cassandra now supports secondary indices along with a SQL-like query language called CQL. Of course, you'll have to pre-plan your indexes with either of those options.
    - schmichael 15y ago
      "As soon as an RDBMS is sharded, it loses most of it's ad-hoc powers." That's an overly broad statement I would avoid. We only need to query across our various PostgreSQL partitions for offline batch operations. Our application code has yet to feel restricted by the manual sharding. Secondary indexes - or rather: not restricting a query to a single partition (using your primary key) - is going to be expensive in any distributed system.
      - rbranson 15y ago
        You lose aggregates and joins, arguably the most powerful parts of an ad-hoc query system (business intelligence kind of stuff). Without those features, you must move a great deal of these concerns into the application layer (or whatever manage your shards), even if it's just rolling up aggregates across shards. This works, but again, at the loss of the real advantages of the database here as you've got to do it yourself. I agree with your second assertion. Secondary indexes should be avoided, but are a useful tool if absolutely needed. I've found that Cassandra's richer data model really allows them to be avoided in most scenarios.
    - jbellis 15y ago
      Cassandra's CREATE INDEX will happily run in the background against existing data. Without blocking other reads or writes, of course.
  - someone13 15y ago
    Yeah, I've considered other options. Riak seems really nice, actually - I'm going to be trying that out next. I do like MongoDB's support for Windows, though. I'm also hesitant to use Cassandra, considering all the trouble that Reddit's had with it [1]. [1] http://blog.reddit.com/2010/05/reddits-may-2010-state-of-servers.html http://blog.reddit.com/2010/05/reddits-may-2010-state-of-ser...
    - jbellis 15y ago
      Reddit's initial problems were from being woefully underprovisioned, which they acknowledged. Here's what David King [ketralnis] said six months later: "Running any large website is a constant race between scaling your user base and scaling your infrastructure to support it," said David King, Lead Developer at Reddit. "Our traffic more than tripled this year, and the transparent scalability afforded to us by Apache Cassandra is in large part what allowed us to do it on our limited resources. Cassandra v0.7 represents the real-life operations lessons learned from installations like ours and provides further features like column expiration that allow us to scale even more of our infrastructure." [https://blogs.apache.org/foundation/entry/the_apache_software_foundation_announces9 https://blogs.apache.org/foundation/entry/the_apache_softwar...]
      - rbranson 15y ago
        Reddit also just uses it as a dumb K/V store and don't take advantage of the column-store model. In theory, all of the comments for a story could be stored as columns in a column family. While this would reduce the write throughput to something like ~1000/second -- for a single story -- because partition granularity is at the row level, within a row, all the columns in a CF are stored sequentially on disk, and it also means only a single node would need to be interrogated to get the data, rather than having to broadcast a massive multi-get across the entire cluster. EDIT: the column keys are also returned in sorted order, so this could be used to pre-sort the comment list by it's optimal access pattern.
gurgeous 15y ago
Money quote based on trying to restore their MongoDB during the EBS downtime: "Waiting for indexes to build takes approximately as long as it takes AWS engineers to fix EBS issues."
- dolinsky 15y ago
  As far as I'm aware this would be no different when reimporting a dump from MySQL/Postgres. If you're not restoring from a snapshot then the indexes have to rebuild themselves.
  - cheald 15y ago
    My experience has been that MySQL backup/restore ops (mysqldump; mysql < dump.sql) are orders of magnitude slower than MongoDB backup/restore ops. We use LVM snapshotting for MySQL backups because the "normal" method is just too slow. I just did a test dump of a 1.7GB MongoDB database: real 0m5.781s user 0m0.570s sys 0m4.817s I'm not in a position to do a restore just now, but my experience has been that restores are similarly quick. 120GB is obviously a far larger amount of data, but unless you've gone insanely nuts with the indexing, I can't imagine that it would take hours upon hours to do.
    - dolinsky 15y ago
      If you are in a position to do so, I can't recommend enough using the Percona XtraBackup[1] tool to enable online snapshotting of MySQL. (Also, if you are able to upgrade to XtraDB I would highly recommend looking into it). A mysqldump / restore can be dependent on many variables when discussing the rate of import. Grouped indexes, delayed FK constraints, buffer sizes, and hardware specs to name a few. My point was more about how one is going to suffer this reindexing problem regardless of data storage facility :) [1]http://www.percona.com/docs/wiki/percona-xtrabackup:start http://www.percona.com/docs/wiki/percona-xtrabackup:start
    - cdavid 15y ago
      5 seconds for a 1.7 GB working set sounds really low: it would mean you could write > 200 Mb / sec assuming you could get data from mongodb infinitely fast. Also, what happens to your mongo database while you export data (can you still write to it ?). In any case, mysqldump is rarely the bottleneck - restoring is. For the latter, you really want to use mysql from percona, because their version of mysqldump has an option to void the data without index first, and create the index later (--innodb-optimize-keys=True). The effect will depend on the database schema of course, but I often see one order of magnitude difference for simple, large tables with a few simple indexes.
ConceitedCode 15y ago
Great read. Gives a great example of when NoSQL is not a good solution instead of a relational database.
- travisglines 15y ago
  I think you may be over generalizing here. Yes there are vast similarities between MongoDB and other NoSQL databases but many of the issues he pointed out here are specific to MongoDB (and certain versions of it at that).
- Nizumzen 15y ago
  I think the real lesson here is to choose technologies based on actual experience rather than hype. So that means extensive testing before nailing your colours to the mast.
  - mtkd 15y ago
    Or choose technology based on requirements. If the port to SQL was so easy and used half the storage and didn't appear to impact performance - it was possibly the better solution in the first place. Mongodb is awesome - one of best technologies I've played with in a long while - but don't be too quick to write SQL off - especially if you have a lot of relationships. Off topic - prototyping in Rails with Mongoid and the schemaless Mongodb underneath is frictionless - it's a step change in speed from prototyping with SQL.
    - tedkalaw 15y ago
      Would you happen to have any anecdotes of a project or dataset that MongoDB worked particularly well for you? I'm working on a project where documents seem to work particularly well, though the core of the project is still in MySQL. I decided to give Mongo a shot for what I'm working on, but posts like this are a reminder to constantly be evaluating the technology you choose to use.
      - jjm 15y ago
geophile 15y ago
This is a really fascinating presentation, and an easy read. The major conclusion is to think really carefully about abandoning a SQL database for noSQL. A little more info on the postgres solution and their experience with replication would have been good.
- trustfundbaby 15y ago
  exactly ... I was hoping to hear how they finally made it work on Postgres, since they initially migrated from postgres to mongo
- schmichael 15y ago
  Horizontal scaling is key. If we had more faith in auto-sharding in MongoDB 1.6, that may have worked (however horizontal scaling wasn't our only issue with MongoDB). With PostgreSQL we manually partition the data (luckily we have a really nice key to partition on). PostgreSQL 9's streaming replication has been adequate so far. We've only been using it in production for a few months and recently moved off of AWS* which drastically reduced our error rates overall. * http://urbanairship.com/blog/2011/06/01/hybrid-cloud-were-up/ http://urbanairship.com/blog/2011/06/01/hybrid-cloud-were-up...
  - jacquesgt 15y ago
    Totally off-topic here, but have you noticed that the body text size of your company's blog stays really small even when trying to zoom in? I've tried zooming in using Safari and Chrome, and with both of them the text stays so small that it's difficult to read.
invisible 15y ago
I think a huge issue that he somewhat glazed over is that he is refusing to use auto-sharding. That is a pretty large benefit that MongoDB provides. Oh, and he refuses to upgrade his 1 year old build (1.5, now at 1.8) - no explanation of why but he was willing to switch the entire stack to PostgreSQL.
- robotadam 15y ago
  We made the switch before 1.8 was released, and made the decision during the 1.6 cycle, give or take. Auto-sharding never worked reliably in our tests. I think schmichael does a good job explaining the new features that would have helped -- spares indexes, in particular -- but the core issues still remain (lock contention, no non-blocking online compaction).
  - schmichael 15y ago
    (speaker here) Adam is exactly right. I wish I could say I mentioned those details in the talk, but I'm afraid I ran out of time. Just to re-iterate: * Most slides refer to 1.4 * We evaluated auto-sharding around 1.5.x & 1.6.0 and it did not work reliably. * We now use 1.8 for a few remaining small MongoDB datasets.
  - kristina 15y ago
    For 2.0, there is some major work on locking being done (concurrency will be an ongoing theme in 2011) and online compaction has been implemented (although you have to kick it off, it's not yet automatic).
    - robotadam 15y ago
      According to the docs, this is a blocking operation -- so it's inplace, but I wouldn't consider it to be online, as I would PostgreSQL's vacuum or Cassandra's major compactions: http://www.mongodb.org/display/DOCS/compact+Command http://www.mongodb.org/display/DOCS/compact+Command
      - 15y ago
i34159 15y ago
Had some of the same issues with HBase. Ended up going back to Postgres too, but with a twist: http://www.slideshare.net/cloudflare/sortasql http://www.slideshare.net/cloudflare/sortasql
- someone13 15y ago
  What filesystem did you end up using? Ever looked at GlusterFS (http://www.gluster.org/ http://www.gluster.org/)? From an (admittedly unscientific) comparison of the various free/open-source distributed filesystems, it seems the best to me. Ceph is still in beta, and the fact that Lustre only supports a single Metadata Server is really scary to me.
AlexC04 15y ago
Although I'm very interested in the subject matter (I have a production site that uses MongoDB and AWS), I often find presentations like this incredibly annoying to read. The problem is that presentations are short-form jot notes that are meant to be supported by a delivery as well. The vast majority of the value comes from what's not written down. What's worse is that the "signal to noise" ratio detracts from my ability to understand what you're really going on about. Some examples: You say "MongoStat is Amazing" why? how? what does it do? (I've personally played with and used it a little, but I'd like to know why you're amazed by it. How can the things you've learned improve my practice? You mention starting with 1.2 - does that mean you're not using 1.8? I've used 1.2 and was suffering 80% - 99.9% CPU load, as soon as I updated to 1.81 the load dropped to closer to 0.03% for most of the same operations. Does that mean your talk is invalid? Obviously there's a lot more to your story that I can get from your slides. All things considered though I'm no trying to come off as a jackass. Normally when I see a HackerNews->slidedeck I just get frustrated and annoyed by things like the "inside joke" and "cute kitten slides" that rely on the speaker to deliver some insightful punchline. In this case I did read one or two interesting points. Though I find I'm still suffering from the signal-to-noise problem that accompanies all these decks, this was better than most. I'd love a bit more narrative around the points that matter and a bit less around things that don't (how to query in mongo isn't really necessary in a story of why you abandoned it, is it?) Anyways - was that a rant, review or stream of consciousness? TD;DR Thanks for the slide deck; I hate slide decks; Yours was less terrible than most.
- schmichael 15y ago
  (I'm the speaker) Audio should be coming, but I'm not sure it will help. I definitely tried to pack way too much into a 45 minute talk. Not sure how to explain mongostat further... have you run it on a loaded server? Just a great little near-real-time stats tool. Like vmstat for a db. It could probably have a talk all its own just to explain how to diagnose performance issues with it. Most of my comments apply to MongoDB 1.4 although our little bit of data that remains in MongoDB has been upgraded to 1.8 (via 1.6 first). I've never had MongoDB use a significant amount of CPU load (from 1.2 - 1.8).
imack 15y ago
This reminds me of a video I saw about scaling with ruby on a niche social network (http://zenbach.com/scaling-fetlife-com-with-ruby http://zenbach.com/scaling-fetlife-com-with-ruby). It was an interesting talk, but they also tried MongoDb and moved away from it, though, they were also using an older version.
cloudhead 15y ago
Well... no shit, people. For whatever reason, many developers think MongoDB is an SQL replacement. To be fair, MongoDB did a very good job at hiding information which made it look bad, while posting misleading benchmarks. I think the majority of the user-base doesn't know MongoDB isn't durable out of the box.. they think it's just a "faster", better MySQL. Doing solid research goes a long way to tone down the hype.
- schmichael 15y ago
  (speaker here) For the record something I did not mention in my slides or talk is that our data was not highly relational. For very similar data we've used non-SQL databases with success. Definitely should be something I make clear in my talk, so people don't think we foolishly tried MongoDB for highly relational data.
  - cloudhead 15y ago
    Thanks for the clarification, my comment wasn't targeted at you actually (it seems like MongoDB could have been a good fit here), but at developers who don't even consider the kind of usage, but instead think of MongoDB as a "better MySQL", when they are in fact fundamentally different. So thank you for posting this, it may make some people reconsider before they jump ship, or at least understand the trade-offs.
  - fiaz 15y ago
    What other non-SQL databases have you used with success?
    - schmichael 15y ago
      Cassandra, but mainly because we have a couple devs who are excellent at debugging Java/JVM issues. Cassandra is young enough you really need to expect to get your hands dirty in its code to keep things humming. (I may be more forgiving of Cassandra because, unlike Mongo, Cassandra's major version number is 0). Another team seems awfully happy with Hadoop, although they've been bitten by an HDFS related bug. I can't speak too much to their experience. Memcached of course is wonderful, and the little bit I've used Redis it has lived up to its promises. Riak is the only non-relational database I can think of that I've only heard positive things about, but I've never personally used it.
dolinsky 15y ago
One particular part of that slide deck stuck out, and b/c there's no accompanying audio to go with it I thought I'd mention it. Towards the beginning of the deck in one slide he says 'MongoDB is fast.... until your data / indexes no longer fit in memory'. Later on there's the '120GB of data in MongoDB became 70GB of data in Postgres'. The next slide talks about using short key names + a key-mapping layer to preserve space. While using longer key names in your objects will lead to larger objects / larger storage requirements, indexes do not contain the names of the keys of the documents they are created on, so if I have 10M documents in a collection and an index on a field 'ts', that index has the same size as if the field were named 'timestamp'. It's only the storage of the object that grows in size, and by the time you are handling 120GB of data it's very reasonable to imagine you would be using Memcached in front of MongoDB to obtain the actual objects.
- schmichael 15y ago
  We used verbose key names and paid for it. We even failed to override _id with a useful primary key. So... not great. That being said, our data is not completely normalized in PostgreSQL, so it's not a totally unfair comparison. It may be possible to simulate an apples to apples comparison (small keys in mongo + perfectly normalized pgsql tables), but I'm not sure that would be a useful indicator for real world use cases. Suffice it to say: repeating your schema in every row/document is going to take more space than not repeating it.
  - dolinsky 15y ago
    Oh no doubt, and I think naming conventions is something that takes practice over time in this situation - too verbose leads to 'wasted' space, too little leads to making it near impossible for someone to view the database. Since indexes don't store the key names that does alleviate part of the problem as well. I tend to leave the default _id in place as it also acts as a 'created_on' timestamp, saving the need to add an additional field. Thanks for the deck. It's always good to hear about situations that didn't work out well and what can be learned going forward (though I do think you'd have a better experience using the 1.8 branch than you did with 1.4).
deleted 15y ago
[deleted]
FooBarWidget 15y ago
The presentation claims that MongoDB performance drops off a cliff once the data or the indices no longer fit in RAM. This is not entirely complete/correct. First off, the same problem occurs no matter which database. Oracle, PostgreSQL, MySQL, whatever, they all grind to a halt when stuff no longer fits in RAM. Second, it's not really the whole data set or the whole index that needs to fit in RAM, it's the working set. This is a pretty big difference. I'm currently running a production MongoDB database which is about 20 times larger than RAM and at first I made the mistake of using random string primary keys (_ids). This ensured good distribution between shards but also ensured good distribution within the primary key index. This essentially made the working set of the primary key index equal to the entire index. Once the primary key index no longer fits in RAM, the database grinded to a halt. I now use a primary key consisting of a timestamp followed by some random text. The timestamp ensures that inserts only hit the end of the index, thereby keeping the working set small. The database is now almost as fast as when it was fresh.
- schmichael 15y ago
  (speaker here) You're absolutely right about the working set being the important bit to keep in memory (and all the indexes you use). I clarified that in person thanks to a great question from the audience. My entire point wrt it slowing-down-outside-of-RAM is that it happens with any database. My point was that any database can be fast if it's in RAM and non-durable; yet Mongo trumpets performance under these unsafe scenarios while other databases use safe defaults. To make matters worse, MongoDB data can suffer from poor data locality over time. Since caching is completely left up to the kernel, 1 hot document, even if it's only 200 bytes, will cause an entire page to remain in memory. The other bytes in the page may be cold documents or null data. This is why flip/flopping is useful in current versions and in-place compaction so important for future versions. For prior research see defrag.exe. I wish you luck in your attempts to tune performance by tweaking _id. I could only take doing hacks like that for so long before I looked for greener pastures.
  - FooBarWidget 15y ago
    Why do you consider tweaking _id a hack, and what is the greener pasture for you? If you use random string primary keys in any other database you'd run into the same problem. My app is much more write-intensive than read-intensive. My stuff wouldn't work on PostgreSQL at all, I have so much data that I must use sharding.
thadeus_venture 15y ago
It seems that the main criticisms are: a) Lack of online compaction (causing data set to be much bigger than necessary) b) Broken auto sharding (complicating horizontal scaling) c) "Scary" durability and scaling stories with auto sharding Looks like auto-sharding has had a lot of work done, while online compaction still isn't there. But in particular I'm curious about (c). What exactly is so scary durability wise with auto-sharding? Is it just the fact that it didn't work in 1.5/1.6?
- schmichael 15y ago
  "But in particular I'm curious about (c). What exactly is so scary durability wise with auto-sharding? Is it just the fact that it didn't work in 1.5/1.6?" Yes. During the 1.5 dev cycle we ran out of time. Our main database server couldn't vertically scale any further, so we had to do something. Despite 1.6.0 being released before our migration finishing, auto-sharding still didn't seem to be anywhere near the stability we needed. I've heard some horror stories regarding auto-sharding & replica sets (in 1.8) since then that sadly I'm not at liberty to share. I think you missed a number of finer points as well: * We were not guided to make the best schema decisions initially, and migrating schemas is tricky and manual. * Some schema decisions exposed us to short-comings in MongoDB (the double updating issue). * The global write lock is severely limiting. * It's unsafe by default (drivers default to not checking for errors when writing, among other things) * The ops load is greater for auto-sharding+replica-sets than for manually sharded master/slave setups (granted the former has features the latter does not). Perhaps the most frustrating part was the constant unfortunate surprises. There are stories that didn't even make it into the slides. Many you could blame on us. Some we could blame on EC2. However, strong benefits to using MongoDB over PgSQL never materialized, so we opted to switch to a system with far fewer unknowns for our fairly modest needs. If you think fully utilizing MongoDB's architecture will give your business a competitive advantage: go nuts. That was far from our experience.
  - thadeus_venture 15y ago
    Thank you, that's very interesting. "Going nuts" with MongoDB is exactly what I'm contemplating right now at my start up, and this is something I haven't quite heard before. Is there more info on the double update issue anywhere? It sounds like a really bad race condition.
rosser 15y ago
If you want to use MongoDB in conjunction with PostgreSQL, the latest release of Bucardo (5.0) includes support for replicating between them.
- schmichael 15y ago
  Goodness.
- bsg75 15y ago
  In the PostgreSQL -> MongoDB direction only. While I understand the reasons for implementing this first (nested document replication to an RDBMS is a bigger task), I am hoping that it will see the reverse direction someday. Using MongoDB as data collectors on the edge, replication to a central RDBMS for use with established BI tools would be a win.
code_duck 15y ago
Etsy.com unveiled a new gallery list system, the Treasury last year and published a couple of articles about their use of Mongo. Apparently they had some problems and rewrote the backend recently... haven't heard what yet. Hopefully they'll publish their experiences in their dev blog after the dust settles.
schmichael 15y ago
A few clarifications: * Slides w/o notes available here: http://opensourcebridge.org/2011/wiki/Scaling_with_MongoDB http://opensourcebridge.org/2011/wiki/Scaling_with_MongoDB * Most slides referred to MongoDB 1.4 * This was our experience, YMMV. I just wanted to share our story in hopes it would help others avoid some pitfalls. I'll try to give honest responses to questions asked. I'm not trying to be unnecessarily cynical, overly negative, or anti-10gen in my slides, talk, or comments here. I'm sure many people are using MongoDB happily, and I've heard 10gen's support is outstanding.
- Joakal 15y ago
  A good part of your presentation should have been what your requirements are. For example: MongoDB's schemaless feature has a downside that means more data as a key must be defined individually. Great if you constantly need to fix/change the schema. A structured schema solution obviously has a space saving.
  - schmichael 15y ago
    Schemaless is not great when you need to actually fix or change your schema. It's very difficult to migrate every document's schema without downtime. Schemaless is great if you have a dynamic schema. We did not for the main dataset in question, and I probably should have mentioned that. We do have 1 smaller dataset still in Mongo that currently has a dynamic schema and works perfectly (very little data & traffic though).
    - jamwt 15y ago
      Kind of a side topic, but lazy, JIT migrations work well for this. You keep a "migration chain" in your fetch layer that goes v1 -> v2, v2 -> v3, and every persisted doc is tagged with the schema version at the time of persistence. Then, your binding code just runs it through whatever subset of the chain brings the document to v-latest. If the document is re-written, it will (naturally) be re-written in updated form, and you'll only pay the upgrade penalty once (or, not at all for cold data). It sounds kinda creepy to old-school schema theorists, but in practice it actually works pretty well (provided you always use this library to retrieve your objects.) Using this method, I've never done a stop-the-world schema migration on a schema-free system, and I don't think I ever would... otherwise, you're tossing out one of the major benefits of schema-free: no migrations! If you're going to be rigorous about full-collection schema consistency, you maybe should have just used a proper RDBMS to begin with...
bdarfler 15y ago
It seems a bit disingenuous to post slides now reviewing a year old version of MongoDB that is almost 3 major revs behind. These days auto-sharding works and works well. There is a learning curve to optimize it (i.e. picking a good shard key, analogous to picking a good _id field) but that is to be expected. Replica sets work exactly as advertised in my experience. Automatic fail over and no need to wake up at 3am is a huge win. Finally single server durability directly addresses durability concerns that many have. The first two points are the reason why my company and so many others are switching even if that means needing to understand how to pick good ids/shard keys and having to shorten our field names by hand. I can see why, if they were not available at the time, MongoDB might not be a good choice but the fact of the matter is that they are available today.
andrew311 15y ago
Michael, interesting presentation. I'm the guy that created the "Ops per second" slide included in the presentation. It was originally from my presentation at MongoNYC: http://www.10gen.com/presentation/mongonyc-2011/optimizing-mongodb-lessons-learned-at-localytics http://www.10gen.com/presentation/mongonyc-2011/optimizing-m... I almost regret including that chart because a lot of people are citing it out of context without explanation. That chart illustrates a very extreme case, and it makes an exaggerated point so that people know to be conscious of working set size vs RAM. FooBarWidget made some very valid points about working set elsewhere in these comments. It's important to note that any database will suffer when data spills out of RAM. The degree to which it suffers varies in different workloads, DBs, and hardware. Databases like PostgreSQL do better in some cases because they have more granular/robust locking and yielding, so disk access is handled better and generally leads to less degradation in performance. For example, in MongoDB, when you do a write, the whole DB locks. Additionally, it does not yield the lock while reading a page from disk in the case of an update (this is changing). So if you spill out of RAM, and you update something, now you need to traverse a b-tree, pull in data, and write to a data extent, possibly all against disk, while blocking other operations. In other databases, they might yield while the data is being fetched from disk for update (or have more granular locks), so a read on an entirely different piece of data can go through. That other piece of data might reside on another disk (imagine RAID setups), so it is able to finish while the write is also going. Beyond that, MVCC DBs can do a read with no read lock at all. Yes, MongoDB lacks this robustness, and it's why it suffers worse under disk bound workloads than other databases. However, 10gen is very aware of this, and they are making a strong effort to introduce better yielding and locking over the next year. Regarding indexing and working sets, FooBarWidget points out that there are very naive ways to approach indexing that lead to a much larger working set than is necessary. With very simple tweaks you can have a working set that only includes a sub-tree of the index and smaller portions of data extents. These tweaks apply to many DBs out there. The classic example is using random hashes or GUIDs for primary ids. If you do this, any new record will end up somewhere entirely arbitrary in the index. This means the whole index is your working set, because placing the new record could traverse any arbitrary path. Imagine instead that you prefix your GUID with time (or use a GUID that has time as the first component). Now your tree will arrange itself according to time, and you will only need the sub-trees for the data that has time ranges you frequently look at. Most workloads only reference recent data, so this helps a ton. You’ll still find yourself doing stuff like this elsewhere in other databases to improve locality. What’s really interesting is when you look at MVCC, WAL log systems and how they compare. They get a huge win by not needing locks to read things in a committed state. They also convert a lot of ops that involve random IO into sequential IO, but you pay a price somewhere else, usually needing constant compaction on live, query handling nodes. Rbranson elaborated really nicely on this stuff: http://news.ycombinator.com/item?id=2688848 http://news.ycombinator.com/item?id=2688848 Other DBs also win in some ways by managing their own row cache. Letting the OS do it against disk extents means holes in RAM data. If you handle it yourself, you focus on caching actual rows and not deleted data. Cassandra does this. Once MongoDB addresses compaction, locks, and yielding, it will start to be very competitive. Keep in mind that I think there's something to be said for having native replica sets and sharding. I can say that it definitely does work in 1.8, we use it all over the place. With the recent journaling improvements, durability is also better, but even without it our experience has been more than adequate. Secondary nodes keep up with our primaries just fine. In the rare case where a primary falls over, we lose maybe only a handful of operations, which for our workload is acceptable. Overall, we're happy with our choice of MongoDB, it's already doing what we need. With the improvements coming down the road, I think it'll be a major force in a year from now, so I'm also excited for what the future holds.

180 points by rbranson 15y ago

Thanks for this. I've heard lots of good things about MongoDB (and auto-sharding makes me drool), but I've heard rumours of problems with it too. Always cool to read something from the other side of the fence. Too bad that there's no other product out there that makes automatically sharding large amounts of data quite so easy...

megaman821 15y ago
Doesn't Riak and Cassandra make it easy? There is just no fast ad-hoc querying in those two.
- rbranson 15y ago
  Arguably, with a data set of any serious magnitude, there's no such thing as ad-hoc. As soon as an RDBMS is sharded, it loses most of it's ad-hoc powers. That being said, Riak Search allows Riak to perform ad-hoc queries and Cassandra now supports secondary indices along with a SQL-like query language called CQL. Of course, you'll have to pre-plan your indexes with either of those options.
  - schmichael 15y ago
    "As soon as an RDBMS is sharded, it loses most of it's ad-hoc powers." That's an overly broad statement I would avoid. We only need to query across our various PostgreSQL partitions for offline batch operations. Our application code has yet to feel restricted by the manual sharding. Secondary indexes - or rather: not restricting a query to a single partition (using your primary key) - is going to be expensive in any distributed system.
    - rbranson 15y ago
      You lose aggregates and joins, arguably the most powerful parts of an ad-hoc query system (business intelligence kind of stuff). Without those features, you must move a great deal of these concerns into the application layer (or whatever manage your shards), even if it's just rolling up aggregates across shards. This works, but again, at the loss of the real advantages of the database here as you've got to do it yourself. I agree with your second assertion. Secondary indexes should be avoided, but are a useful tool if absolutely needed. I've found that Cassandra's richer data model really allows them to be avoided in most scenarios.
  - jbellis 15y ago
    Cassandra's CREATE INDEX will happily run in the background against existing data. Without blocking other reads or writes, of course.
- someone13 15y ago
  Yeah, I've considered other options. Riak seems really nice, actually - I'm going to be trying that out next. I do like MongoDB's support for Windows, though. I'm also hesitant to use Cassandra, considering all the trouble that Reddit's had with it [1]. [1] http://blog.reddit.com/2010/05/reddits-may-2010-state-of-servers.html http://blog.reddit.com/2010/05/reddits-may-2010-state-of-ser...
  - jbellis 15y ago
    Reddit's initial problems were from being woefully underprovisioned, which they acknowledged. Here's what David King [ketralnis] said six months later: "Running any large website is a constant race between scaling your user base and scaling your infrastructure to support it," said David King, Lead Developer at Reddit. "Our traffic more than tripled this year, and the transparent scalability afforded to us by Apache Cassandra is in large part what allowed us to do it on our limited resources. Cassandra v0.7 represents the real-life operations lessons learned from installations like ours and provides further features like column expiration that allow us to scale even more of our infrastructure." [https://blogs.apache.org/foundation/entry/the_apache_software_foundation_announces9 https://blogs.apache.org/foundation/entry/the_apache_softwar...]
    - rbranson 15y ago
      Reddit also just uses it as a dumb K/V store and don't take advantage of the column-store model. In theory, all of the comments for a story could be stored as columns in a column family. While this would reduce the write throughput to something like ~1000/second -- for a single story -- because partition granularity is at the row level, within a row, all the columns in a CF are stored sequentially on disk, and it also means only a single node would need to be interrogated to get the data, rather than having to broadcast a massive multi-get across the entire cluster. EDIT: the column keys are also returned in sorted order, so this could be used to pre-sort the comment list by it's optimal access pattern.

gurgeous 15y ago

Money quote based on trying to restore their MongoDB during the EBS downtime: "Waiting for indexes to build takes approximately as long as it takes AWS engineers to fix EBS issues."

dolinsky 15y ago
As far as I'm aware this would be no different when reimporting a dump from MySQL/Postgres. If you're not restoring from a snapshot then the indexes have to rebuild themselves.
- cheald 15y ago
  My experience has been that MySQL backup/restore ops (mysqldump; mysql < dump.sql) are orders of magnitude slower than MongoDB backup/restore ops. We use LVM snapshotting for MySQL backups because the "normal" method is just too slow. I just did a test dump of a 1.7GB MongoDB database: real 0m5.781s user 0m0.570s sys 0m4.817s I'm not in a position to do a restore just now, but my experience has been that restores are similarly quick. 120GB is obviously a far larger amount of data, but unless you've gone insanely nuts with the indexing, I can't imagine that it would take hours upon hours to do.
  - dolinsky 15y ago
    If you are in a position to do so, I can't recommend enough using the Percona XtraBackup[1] tool to enable online snapshotting of MySQL. (Also, if you are able to upgrade to XtraDB I would highly recommend looking into it). A mysqldump / restore can be dependent on many variables when discussing the rate of import. Grouped indexes, delayed FK constraints, buffer sizes, and hardware specs to name a few. My point was more about how one is going to suffer this reindexing problem regardless of data storage facility :) [1]http://www.percona.com/docs/wiki/percona-xtrabackup:start http://www.percona.com/docs/wiki/percona-xtrabackup:start
  - cdavid 15y ago
    5 seconds for a 1.7 GB working set sounds really low: it would mean you could write > 200 Mb / sec assuming you could get data from mongodb infinitely fast. Also, what happens to your mongo database while you export data (can you still write to it ?). In any case, mysqldump is rarely the bottleneck - restoring is. For the latter, you really want to use mysql from percona, because their version of mysqldump has an option to void the data without index first, and create the index later (--innodb-optimize-keys=True). The effect will depend on the database schema of course, but I often see one order of magnitude difference for simple, large tables with a few simple indexes.

ConceitedCode 15y ago

Great read. Gives a great example of when NoSQL is not a good solution instead of a relational database.

travisglines 15y ago
I think you may be over generalizing here. Yes there are vast similarities between MongoDB and other NoSQL databases but many of the issues he pointed out here are specific to MongoDB (and certain versions of it at that).
Nizumzen 15y ago
I think the real lesson here is to choose technologies based on actual experience rather than hype. So that means extensive testing before nailing your colours to the mast.
- mtkd 15y ago
  Or choose technology based on requirements. If the port to SQL was so easy and used half the storage and didn't appear to impact performance - it was possibly the better solution in the first place. Mongodb is awesome - one of best technologies I've played with in a long while - but don't be too quick to write SQL off - especially if you have a lot of relationships. Off topic - prototyping in Rails with Mongoid and the schemaless Mongodb underneath is frictionless - it's a step change in speed from prototyping with SQL.
  - tedkalaw 15y ago
    Would you happen to have any anecdotes of a project or dataset that MongoDB worked particularly well for you? I'm working on a project where documents seem to work particularly well, though the core of the project is still in MySQL. I decided to give Mongo a shot for what I'm working on, but posts like this are a reminder to constantly be evaluating the technology you choose to use.
    - jjm 15y ago

geophile 15y ago

This is a really fascinating presentation, and an easy read. The major conclusion is to think really carefully about abandoning a SQL database for noSQL. A little more info on the postgres solution and their experience with replication would have been good.

trustfundbaby 15y ago
exactly ... I was hoping to hear how they finally made it work on Postgres, since they initially migrated from postgres to mongo
schmichael 15y ago
Horizontal scaling is key. If we had more faith in auto-sharding in MongoDB 1.6, that may have worked (however horizontal scaling wasn't our only issue with MongoDB). With PostgreSQL we manually partition the data (luckily we have a really nice key to partition on). PostgreSQL 9's streaming replication has been adequate so far. We've only been using it in production for a few months and recently moved off of AWS* which drastically reduced our error rates overall. * http://urbanairship.com/blog/2011/06/01/hybrid-cloud-were-up/ http://urbanairship.com/blog/2011/06/01/hybrid-cloud-were-up...
- jacquesgt 15y ago
  Totally off-topic here, but have you noticed that the body text size of your company's blog stays really small even when trying to zoom in? I've tried zooming in using Safari and Chrome, and with both of them the text stays so small that it's difficult to read.

invisible 15y ago

I think a huge issue that he somewhat glazed over is that he is refusing to use auto-sharding. That is a pretty large benefit that MongoDB provides. Oh, and he refuses to upgrade his 1 year old build (1.5, now at 1.8) - no explanation of why but he was willing to switch the entire stack to PostgreSQL.

robotadam 15y ago
We made the switch before 1.8 was released, and made the decision during the 1.6 cycle, give or take. Auto-sharding never worked reliably in our tests. I think schmichael does a good job explaining the new features that would have helped -- spares indexes, in particular -- but the core issues still remain (lock contention, no non-blocking online compaction).
- schmichael 15y ago
  (speaker here) Adam is exactly right. I wish I could say I mentioned those details in the talk, but I'm afraid I ran out of time. Just to re-iterate: * Most slides refer to 1.4 * We evaluated auto-sharding around 1.5.x & 1.6.0 and it did not work reliably. * We now use 1.8 for a few remaining small MongoDB datasets.
- kristina 15y ago
  For 2.0, there is some major work on locking being done (concurrency will be an ongoing theme in 2011) and online compaction has been implemented (although you have to kick it off, it's not yet automatic).
  - robotadam 15y ago
    According to the docs, this is a blocking operation -- so it's inplace, but I wouldn't consider it to be online, as I would PostgreSQL's vacuum or Cassandra's major compactions: http://www.mongodb.org/display/DOCS/compact+Command http://www.mongodb.org/display/DOCS/compact+Command
    - 15y ago

i34159 15y ago

Had some of the same issues with HBase. Ended up going back to Postgres too, but with a twist: http://www.slideshare.net/cloudflare/sortasql http://www.slideshare.net/cloudflare/sortasql

someone13 15y ago
What filesystem did you end up using? Ever looked at GlusterFS (http://www.gluster.org/ http://www.gluster.org/)? From an (admittedly unscientific) comparison of the various free/open-source distributed filesystems, it seems the best to me. Ceph is still in beta, and the fact that Lustre only supports a single Metadata Server is really scary to me.

AlexC04 15y ago

Although I'm very interested in the subject matter (I have a production site that uses MongoDB and AWS), I often find presentations like this incredibly annoying to read. The problem is that presentations are short-form jot notes that are meant to be supported by a delivery as well. The vast majority of the value comes from what's not written down. What's worse is that the "signal to noise" ratio detracts from my ability to understand what you're really going on about. Some examples: You say "MongoStat is Amazing" why? how? what does it do? (I've personally played with and used it a little, but I'd like to know why you're amazed by it. How can the things you've learned improve my practice? You mention starting with 1.2 - does that mean you're not using 1.8? I've used 1.2 and was suffering 80% - 99.9% CPU load, as soon as I updated to 1.81 the load dropped to closer to 0.03% for most of the same operations. Does that mean your talk is invalid? Obviously there's a lot more to your story that I can get from your slides. All things considered though I'm no trying to come off as a jackass. Normally when I see a HackerNews->slidedeck I just get frustrated and annoyed by things like the "inside joke" and "cute kitten slides" that rely on the speaker to deliver some insightful punchline. In this case I did read one or two interesting points. Though I find I'm still suffering from the signal-to-noise problem that accompanies all these decks, this was better than most. I'd love a bit more narrative around the points that matter and a bit less around things that don't (how to query in mongo isn't really necessary in a story of why you abandoned it, is it?) Anyways - was that a rant, review or stream of consciousness? TD;DR Thanks for the slide deck; I hate slide decks; Yours was less terrible than most.

schmichael 15y ago
(I'm the speaker) Audio should be coming, but I'm not sure it will help. I definitely tried to pack way too much into a 45 minute talk. Not sure how to explain mongostat further... have you run it on a loaded server? Just a great little near-real-time stats tool. Like vmstat for a db. It could probably have a talk all its own just to explain how to diagnose performance issues with it. Most of my comments apply to MongoDB 1.4 although our little bit of data that remains in MongoDB has been upgraded to 1.8 (via 1.6 first). I've never had MongoDB use a significant amount of CPU load (from 1.2 - 1.8).

imack 15y ago

This reminds me of a video I saw about scaling with ruby on a niche social network (http://zenbach.com/scaling-fetlife-com-with-ruby http://zenbach.com/scaling-fetlife-com-with-ruby). It was an interesting talk, but they also tried MongoDb and moved away from it, though, they were also using an older version.

cloudhead 15y ago

Well... no shit, people. For whatever reason, many developers think MongoDB is an SQL replacement. To be fair, MongoDB did a very good job at hiding information which made it look bad, while posting misleading benchmarks. I think the majority of the user-base doesn't know MongoDB isn't durable out of the box.. they think it's just a "faster", better MySQL. Doing solid research goes a long way to tone down the hype.

schmichael 15y ago
(speaker here) For the record something I did not mention in my slides or talk is that our data was not highly relational. For very similar data we've used non-SQL databases with success. Definitely should be something I make clear in my talk, so people don't think we foolishly tried MongoDB for highly relational data.
- cloudhead 15y ago
  Thanks for the clarification, my comment wasn't targeted at you actually (it seems like MongoDB could have been a good fit here), but at developers who don't even consider the kind of usage, but instead think of MongoDB as a "better MySQL", when they are in fact fundamentally different. So thank you for posting this, it may make some people reconsider before they jump ship, or at least understand the trade-offs.
- fiaz 15y ago
  What other non-SQL databases have you used with success?
  - schmichael 15y ago
    Cassandra, but mainly because we have a couple devs who are excellent at debugging Java/JVM issues. Cassandra is young enough you really need to expect to get your hands dirty in its code to keep things humming. (I may be more forgiving of Cassandra because, unlike Mongo, Cassandra's major version number is 0). Another team seems awfully happy with Hadoop, although they've been bitten by an HDFS related bug. I can't speak too much to their experience. Memcached of course is wonderful, and the little bit I've used Redis it has lived up to its promises. Riak is the only non-relational database I can think of that I've only heard positive things about, but I've never personally used it.

dolinsky 15y ago

One particular part of that slide deck stuck out, and b/c there's no accompanying audio to go with it I thought I'd mention it. Towards the beginning of the deck in one slide he says 'MongoDB is fast.... until your data / indexes no longer fit in memory'. Later on there's the '120GB of data in MongoDB became 70GB of data in Postgres'. The next slide talks about using short key names + a key-mapping layer to preserve space. While using longer key names in your objects will lead to larger objects / larger storage requirements, indexes do not contain the names of the keys of the documents they are created on, so if I have 10M documents in a collection and an index on a field 'ts', that index has the same size as if the field were named 'timestamp'. It's only the storage of the object that grows in size, and by the time you are handling 120GB of data it's very reasonable to imagine you would be using Memcached in front of MongoDB to obtain the actual objects.

schmichael 15y ago
We used verbose key names and paid for it. We even failed to override _id with a useful primary key. So... not great. That being said, our data is not completely normalized in PostgreSQL, so it's not a totally unfair comparison. It may be possible to simulate an apples to apples comparison (small keys in mongo + perfectly normalized pgsql tables), but I'm not sure that would be a useful indicator for real world use cases. Suffice it to say: repeating your schema in every row/document is going to take more space than not repeating it.
- dolinsky 15y ago
  Oh no doubt, and I think naming conventions is something that takes practice over time in this situation - too verbose leads to 'wasted' space, too little leads to making it near impossible for someone to view the database. Since indexes don't store the key names that does alleviate part of the problem as well. I tend to leave the default _id in place as it also acts as a 'created_on' timestamp, saving the need to add an additional field. Thanks for the deck. It's always good to hear about situations that didn't work out well and what can be learned going forward (though I do think you'd have a better experience using the 1.8 branch than you did with 1.4).

deleted 15y ago

[deleted]

FooBarWidget 15y ago

The presentation claims that MongoDB performance drops off a cliff once the data or the indices no longer fit in RAM. This is not entirely complete/correct. First off, the same problem occurs no matter which database. Oracle, PostgreSQL, MySQL, whatever, they all grind to a halt when stuff no longer fits in RAM. Second, it's not really the whole data set or the whole index that needs to fit in RAM, it's the working set. This is a pretty big difference. I'm currently running a production MongoDB database which is about 20 times larger than RAM and at first I made the mistake of using random string primary keys (_ids). This ensured good distribution between shards but also ensured good distribution within the primary key index. This essentially made the working set of the primary key index equal to the entire index. Once the primary key index no longer fits in RAM, the database grinded to a halt. I now use a primary key consisting of a timestamp followed by some random text. The timestamp ensures that inserts only hit the end of the index, thereby keeping the working set small. The database is now almost as fast as when it was fresh.

schmichael 15y ago
(speaker here) You're absolutely right about the working set being the important bit to keep in memory (and all the indexes you use). I clarified that in person thanks to a great question from the audience. My entire point wrt it slowing-down-outside-of-RAM is that it happens with any database. My point was that any database can be fast if it's in RAM and non-durable; yet Mongo trumpets performance under these unsafe scenarios while other databases use safe defaults. To make matters worse, MongoDB data can suffer from poor data locality over time. Since caching is completely left up to the kernel, 1 hot document, even if it's only 200 bytes, will cause an entire page to remain in memory. The other bytes in the page may be cold documents or null data. This is why flip/flopping is useful in current versions and in-place compaction so important for future versions. For prior research see defrag.exe. I wish you luck in your attempts to tune performance by tweaking _id. I could only take doing hacks like that for so long before I looked for greener pastures.
- FooBarWidget 15y ago
  Why do you consider tweaking _id a hack, and what is the greener pasture for you? If you use random string primary keys in any other database you'd run into the same problem. My app is much more write-intensive than read-intensive. My stuff wouldn't work on PostgreSQL at all, I have so much data that I must use sharding.

thadeus_venture 15y ago

It seems that the main criticisms are: a) Lack of online compaction (causing data set to be much bigger than necessary) b) Broken auto sharding (complicating horizontal scaling) c) "Scary" durability and scaling stories with auto sharding Looks like auto-sharding has had a lot of work done, while online compaction still isn't there. But in particular I'm curious about (c). What exactly is so scary durability wise with auto-sharding? Is it just the fact that it didn't work in 1.5/1.6?

schmichael 15y ago
"But in particular I'm curious about (c). What exactly is so scary durability wise with auto-sharding? Is it just the fact that it didn't work in 1.5/1.6?" Yes. During the 1.5 dev cycle we ran out of time. Our main database server couldn't vertically scale any further, so we had to do something. Despite 1.6.0 being released before our migration finishing, auto-sharding still didn't seem to be anywhere near the stability we needed. I've heard some horror stories regarding auto-sharding & replica sets (in 1.8) since then that sadly I'm not at liberty to share. I think you missed a number of finer points as well: * We were not guided to make the best schema decisions initially, and migrating schemas is tricky and manual. * Some schema decisions exposed us to short-comings in MongoDB (the double updating issue). * The global write lock is severely limiting. * It's unsafe by default (drivers default to not checking for errors when writing, among other things) * The ops load is greater for auto-sharding+replica-sets than for manually sharded master/slave setups (granted the former has features the latter does not). Perhaps the most frustrating part was the constant unfortunate surprises. There are stories that didn't even make it into the slides. Many you could blame on us. Some we could blame on EC2. However, strong benefits to using MongoDB over PgSQL never materialized, so we opted to switch to a system with far fewer unknowns for our fairly modest needs. If you think fully utilizing MongoDB's architecture will give your business a competitive advantage: go nuts. That was far from our experience.
- thadeus_venture 15y ago
  Thank you, that's very interesting. "Going nuts" with MongoDB is exactly what I'm contemplating right now at my start up, and this is something I haven't quite heard before. Is there more info on the double update issue anywhere? It sounds like a really bad race condition.

rosser 15y ago

If you want to use MongoDB in conjunction with PostgreSQL, the latest release of Bucardo (5.0) includes support for replicating between them.

schmichael 15y ago
Goodness.
bsg75 15y ago
In the PostgreSQL -> MongoDB direction only. While I understand the reasons for implementing this first (nested document replication to an RDBMS is a bigger task), I am hoping that it will see the reverse direction someday. Using MongoDB as data collectors on the edge, replication to a central RDBMS for use with established BI tools would be a win.

code_duck 15y ago

Etsy.com unveiled a new gallery list system, the Treasury last year and published a couple of articles about their use of Mongo. Apparently they had some problems and rewrote the backend recently... haven't heard what yet. Hopefully they'll publish their experiences in their dev blog after the dust settles.

schmichael 15y ago

A few clarifications: * Slides w/o notes available here: http://opensourcebridge.org/2011/wiki/Scaling_with_MongoDB http://opensourcebridge.org/2011/wiki/Scaling_with_MongoDB * Most slides referred to MongoDB 1.4 * This was our experience, YMMV. I just wanted to share our story in hopes it would help others avoid some pitfalls. I'll try to give honest responses to questions asked. I'm not trying to be unnecessarily cynical, overly negative, or anti-10gen in my slides, talk, or comments here. I'm sure many people are using MongoDB happily, and I've heard 10gen's support is outstanding.

Joakal 15y ago
A good part of your presentation should have been what your requirements are. For example: MongoDB's schemaless feature has a downside that means more data as a key must be defined individually. Great if you constantly need to fix/change the schema. A structured schema solution obviously has a space saving.
- schmichael 15y ago
  Schemaless is not great when you need to actually fix or change your schema. It's very difficult to migrate every document's schema without downtime. Schemaless is great if you have a dynamic schema. We did not for the main dataset in question, and I probably should have mentioned that. We do have 1 smaller dataset still in Mongo that currently has a dynamic schema and works perfectly (very little data & traffic though).
  - jamwt 15y ago
    Kind of a side topic, but lazy, JIT migrations work well for this. You keep a "migration chain" in your fetch layer that goes v1 -> v2, v2 -> v3, and every persisted doc is tagged with the schema version at the time of persistence. Then, your binding code just runs it through whatever subset of the chain brings the document to v-latest. If the document is re-written, it will (naturally) be re-written in updated form, and you'll only pay the upgrade penalty once (or, not at all for cold data). It sounds kinda creepy to old-school schema theorists, but in practice it actually works pretty well (provided you always use this library to retrieve your objects.) Using this method, I've never done a stop-the-world schema migration on a schema-free system, and I don't think I ever would... otherwise, you're tossing out one of the major benefits of schema-free: no migrations! If you're going to be rigorous about full-collection schema consistency, you maybe should have just used a proper RDBMS to begin with...

bdarfler 15y ago

It seems a bit disingenuous to post slides now reviewing a year old version of MongoDB that is almost 3 major revs behind. These days auto-sharding works and works well. There is a learning curve to optimize it (i.e. picking a good shard key, analogous to picking a good _id field) but that is to be expected. Replica sets work exactly as advertised in my experience. Automatic fail over and no need to wake up at 3am is a huge win. Finally single server durability directly addresses durability concerns that many have. The first two points are the reason why my company and so many others are switching even if that means needing to understand how to pick good ids/shard keys and having to shorten our field names by hand. I can see why, if they were not available at the time, MongoDB might not be a good choice but the fact of the matter is that they are available today.

andrew311 15y ago

Michael, interesting presentation. I'm the guy that created the "Ops per second" slide included in the presentation. It was originally from my presentation at MongoNYC: http://www.10gen.com/presentation/mongonyc-2011/optimizing-mongodb-lessons-learned-at-localytics http://www.10gen.com/presentation/mongonyc-2011/optimizing-m... I almost regret including that chart because a lot of people are citing it out of context without explanation. That chart illustrates a very extreme case, and it makes an exaggerated point so that people know to be conscious of working set size vs RAM. FooBarWidget made some very valid points about working set elsewhere in these comments. It's important to note that any database will suffer when data spills out of RAM. The degree to which it suffers varies in different workloads, DBs, and hardware. Databases like PostgreSQL do better in some cases because they have more granular/robust locking and yielding, so disk access is handled better and generally leads to less degradation in performance. For example, in MongoDB, when you do a write, the whole DB locks. Additionally, it does not yield the lock while reading a page from disk in the case of an update (this is changing). So if you spill out of RAM, and you update something, now you need to traverse a b-tree, pull in data, and write to a data extent, possibly all against disk, while blocking other operations. In other databases, they might yield while the data is being fetched from disk for update (or have more granular locks), so a read on an entirely different piece of data can go through. That other piece of data might reside on another disk (imagine RAID setups), so it is able to finish while the write is also going. Beyond that, MVCC DBs can do a read with no read lock at all. Yes, MongoDB lacks this robustness, and it's why it suffers worse under disk bound workloads than other databases. However, 10gen is very aware of this, and they are making a strong effort to introduce better yielding and locking over the next year. Regarding indexing and working sets, FooBarWidget points out that there are very naive ways to approach indexing that lead to a much larger working set than is necessary. With very simple tweaks you can have a working set that only includes a sub-tree of the index and smaller portions of data extents. These tweaks apply to many DBs out there. The classic example is using random hashes or GUIDs for primary ids. If you do this, any new record will end up somewhere entirely arbitrary in the index. This means the whole index is your working set, because placing the new record could traverse any arbitrary path. Imagine instead that you prefix your GUID with time (or use a GUID that has time as the first component). Now your tree will arrange itself according to time, and you will only need the sub-trees for the data that has time ranges you frequently look at. Most workloads only reference recent data, so this helps a ton. You’ll still find yourself doing stuff like this elsewhere in other databases to improve locality. What’s really interesting is when you look at MVCC, WAL log systems and how they compare. They get a huge win by not needing locks to read things in a committed state. They also convert a lot of ops that involve random IO into sequential IO, but you pay a price somewhere else, usually needing constant compaction on live, query handling nodes. Rbranson elaborated really nicely on this stuff: http://news.ycombinator.com/item?id=2688848 http://news.ycombinator.com/item?id=2688848 Other DBs also win in some ways by managing their own row cache. Letting the OS do it against disk extents means holes in RAM data. If you handle it yourself, you focus on caching actual rows and not deleted data. Cassandra does this. Once MongoDB addresses compaction, locks, and yielding, it will start to be very competitive. Keep in mind that I think there's something to be said for having native replica sets and sharding. I can say that it definitely does work in 1.8, we use it all over the place. With the recent journaling improvements, durability is also better, but even without it our experience has been more than adequate. Secondary nodes keep up with our primaries just fine. In the rare case where a primary falls over, we lose maybe only a handful of operations, which for our workload is acceptable. Overall, we're happy with our choice of MongoDB, it's already doing what we need. With the improvements coming down the road, I think it'll be a major force in a year from now, so I'm also excited for what the future holds.