26 ms·

A Year with MongoDB

362 points by mitchellh 15y ago

citricsquid 15y ago
Not related to the article but the site: has anyone else been getting "connection interrupted" errors with tumblr recently? If I load a tumblr blog for the first time in ~24 hours the first and second page loads will result in connection interrupted, the 3rd and beyond will all load fine.
jrussbowman 15y ago
From the beginning I've understood mongodb to be built with it's approach for scaling, performance, redundancy and backup to be horizontal scaling. They recently added journaling for single server durability, but before that replication was how you made sure you data was safe. It seems to me when I see complaints about mongodb it's because people don't want to horizontally scale it and instead believe vertical scaling should be more available. Just seems to me people don't like how mongodb is built, but if used as intended I think mongodb performs as advertised. In most cases I don't think it's the tool, rather the one using it.
- LeafStorm 15y ago
  While I agree that you should definitely use your tools in the best way you're capable of, I think for most people there's a baseline expectation that if you save data in a database, that data will be safe (at the very least, recoverable) unless something happens like the server catching on fire. Nearly every other major database has this as the default -- MySQL, PostgreSQL, CouchDB, and Berkeley DB to name a few. (Redis doesn't, but it's also very upfront about it, and does provide this kind of durability as an option from early on.) So when MongoDB breaks this expectation, and when asked to support it as an option, just says, "That's what more hardware is for," it's a pretty big turnoff.
  - antirez 15y ago
    You are wrong about Redis. Cheers
    - justinsb 15y ago
      Care to explain? I believe for Redis, "appendfsync everysec" is the default. The poster's point was that MySQL and Postgres both ship with something like "appendfsync always", and you have to opt-in to the the less safe mode if you want to get more performance. Redis ships with the less safe mode pre-selected, and so has higher performance out-of-the-box.
      - corford 15y ago
        You're right. The Postgres equivalent to "appendfsync always" is "synchronous_commit = on". Which AFAIK is the default. However, one of the nice things about redis is that even if you run "appendfsync everysec" you never run the risk of corruption. You're only risk is losing a maximum of 2 seconds worth of data. If you missed it, there's a wonderful blog post by antirez covering all of this (and a lot more) here: http://antirez.com/post/redis-persistence-demystified.html http://antirez.com/post/redis-persistence-demystified.html
        justinsb 15y ago
        All modern relational databases use this approach (it's called ARIES), so all offer the same guarantees as Redis.
        moe 15y ago
        And to praise redis some more; it also comes with the sanest (async) replication support of any database. By far. It's literally a one-liner in the config file. No bootstrap needed, maintenance free, absolutely no strings attached. 10 seconds and you're done. Which means you'll actually use it from day 1 and never worry about it. Compare that to any other database.
      - deleted 15y ago
        [deleted]
  - mnutt 15y ago
    There's actually two separate concerns: Mongo client's "safe" operator causes the client to see if the database threw an error and throw an error itself. Like someone mentioned, it mostly falls on the mongo clients to implement this. We mostly use fire-and-forget for our application, since it is just logging stats and speed is more important than losing an increment here or there. There should probably be better documentation telling people to always use the safe operations for important data. There is also the durability issue. Early versions shipped with durability turned off by default and required replication to maintain durability. Mongo has had the journal feature since 1.8 and has it enabled by default since 1.9.2. (current version is 2.0) So while mongo has definitely been unsafe in the past, both kinds of safety are now supported, and one is default. The other is either not that big a deal or egregious, depending on the way you view mongo.
- paulsutter 15y ago
  You're right that MongoDB is designed from the start for horizontal scaling, and the author could have better articulated his reluctance to add machines. However, his suggestions would improve mongodb performance in all configurations, not limited to single node. The mongodb guys did a great job simplfying the internal architecture and focusing on a great API to get the product out quickly. I'm confident they can make these sorts of internal improvements to broaden the use of mongodb.
  - heretohelp 15y ago
    >You're right that MongoDB is designed from the start for horizontal scaling, Nope.
- mitchellh 15y ago
  [Note: I wrote the blog post] I'm not at all against horizontally scaling. However, I don't believe that horizontally scaling should be necessary doing a mere 200 updates to per second to a data store that isn't even fsyncing writes to disk. Think of it in terms of cost per ops. Let's just say 200 update ops per second is the point at which you need to shard (not scientific, but let's just use that as a benchmark since that is what we saw at Kiip). MongoDB likes memory, so let's use high-memory AWS instances as a cost benchmark. I think this is fair since MongoDB advertises itself as a DB built for the cloud. The cheapest high-memory instance is around $330/month. That gives you a cost per op of 6.37e-5 cents per update operation. Let's compare this to PostgreSQL, which we've had in production for a couple months at Kiip now. Our PostgreSQL master server has peaked at around 1000 updates per second without issue, and also with the bonus that it doesn't block reads for no reason. The cost per op for PostgreSQL is 1.27e-5 cents. Therefore, if you're swimming in money, then MongoDB seems like a great way to scale. However, we try to be more efficient with our infrastructure expenditures. EDIT: Updated numbers, math is hard.
  - zzzeek 15y ago
    typo in the numbers ? I can get your mongoDB number but not pg: >>> seconds_per_month = 60 * 60 * 24 * 30 >>> ops_per_month_pg = 1000 * seconds_per_month >>> ops_per_month_mg = 200 * seconds_per_month >>> 330.0 / ops_per_month_pg * 100 1.2731481481481482e-05 >>> 330.0 / ops_per_month_mg * 100 6.36574074074074e-05
    - mitchellh 15y ago
      You're absolutely right. Decimal points are hard. Fixed my comment (and noted it).
    - nivertech 15y ago
      You could have skipped all the calculations and just say, that 1000/200=5, I.e PostgreSQL 5 times more cost effective than MongoDB.
  - jabwork 15y ago
    Can you give any sort of indication of the value of a schemaless database and the flexibility it provided as the team fleshed out the data model? Was this a mere convenience over traditional schema migration or something more?
- cheald 15y ago
  I think the article is a fair criticism, and I think your response is likewise fair. Mongo was built with horizontal scaling in mind, and to that end, it tends to suffer noticeably when you overload a single node. Things like the global write lock and single-threaded map/reduce are definitely problems, and shouldn't be pooh-pooh'd away as "oh, just scale horizontally". Uncompacted key names are a real problem, and a consequence of it is that Mongo tends to take more disk/RAM for the same data (+indexes) than a comparable RDBMS does. Maintenance is clunky - you end up having to repair or compact each slave individually, then step down your master and repair or compact it, then step it back up (which does work decently, but is time consuming!) None of these are showstoppers, but they are pain points, especially if you're riding the "Mongo is magical scaling sauce" hype train. It takes a lot of work to really get it humming, and once you do, it's pretty damn good at the things it's good at, but there are gotchas and they will bite you if you aren't looking out for them.
  - ericflo 15y ago
    Mongo was not designed with horizontal scaling in mind. Riak, Cassandra, HBase, Project Voldemort...these are the projects that were designed with horizontal scaling in mind (as evidenced by their architectures.) But not Mongo.
    - jasonmccay 15y ago
      I have to respectfully disagree. Your comment is a bit sweeping and I really don't think that MongoDB's sharding solution is a bad one ... simply the strategies are different. There is a large set of nice features that makes Mongo, for most people, nice to use long before you even need to address sharding. The percentage of people that will need to shard is much lower than the percentage of people that can get considerable benefits of how you can query data in MongoDB, for example, vs. Riak. You are such a Riak-lover. :P
      - ericflo 15y ago
        I am definitely a Riak-lover :) And I also agree that there is a large set of nice features that make Mongo nice to use before sharding.
- saurik 15y ago
  The behavior described in the article, though, is just a fundamental misuse of the hardware resources. If I'm hitting write bottlenecks, I need to shard: that's just a fact of reality. However, write throughput and read throughput should be unrelated (as durable writes are bottlenecks by disk and reads of hot data are bottlenecked by CPU/RAM): if I have saturated five machines with writes I should have five machines worth of read throughput, and that is a /lot/ of throughput... with the behavior in this article, you have no read capacity as you are blocked by writes, even if the writes are to unrelated collections of data (which is downright non-sensical behavior). Even with more machines (I guess, twice as many), the latency of your reads is now being effected horribly by having to wait for the write locks... it is just a bad solution.
paulsutter 15y ago
This article is an excellent articulation of the strengths and (fixable) issues with mongoDB. I like MongoDB a lot, and the improvements suggested would really strengthen the product and could make me more comfortable to use it in more serious applications.
- madworld 15y ago
  How is the global write lock "fixable" without a major rewrite of the codebase? Like the article suggested, it would be one thing if they did it for transaction support. In reality, from looking at the code, it seems like the global write lock came from not wanting to solve the hard problems other people are solving.
  - geoffeg 15y ago
    DB-level locking is planned for MongoDB 2.2 which should be out within a few months. https://jira.mongodb.org/browse/SERVER-4328 https://jira.mongodb.org/browse/SERVER-4328
    - spidaman 15y ago
      Meh, if your other option is to use PostgreSQL and get row level locks, a db level lock is still a fail.
      - j-kidd 15y ago
        And here's a great post that provides some insight on how much effort has been put in by RDBMS vendors to handle locking: http://stackoverflow.com/a/872808 http://stackoverflow.com/a/872808
  - paulsutter 15y ago
    A major rewrite of the core engine is exactly what's needed. Sounds like a fun project. If they don't do it, someone else will. Adoption is hard to replace. Modifying API's is really hard. Rewriting a core engine? Reasonable and in this case probably necessary given the issues. There are lots of people here who could replace that core engine and someone should if the MongoDB guys can't.
rdtsc 15y ago
> Safe off by default I think that is fixed now. But this the single most appalling design decision they could have made while also claiming their product was a "database". (And this has been discussed here before so just do a search if you will). This wasn't a bug, it was a deliberate design decision. Ok, that that would have been alright if they put a bright red warning on their front page. "We disabled durability by default (even though we call this product a database). If you run this configuration as a single server your data could be silently corrupted. Proceed at your own risk". But, they didn't. I call that being "shady". Yap, you read the correctly, you'd issue a write and there would be no result coming back acknowledging that at least the data made into the OS buffer. I can only guess their motive was to look good in benchmarks that everyone like to run and post results on the web. But there were (still are) real cases of people's data being silently corrupted, and they noticed only much later, when backups, and backups of backups were already made.
- jbellis 15y ago
  According to http://www.mongodb.org/display/DOCS/getLastError+Command http://www.mongodb.org/display/DOCS/getLastError+Command it is still unsafe by default.
  - latch 15y ago
    No, the journal file is turned on server-side and you'll get a journal append every 100ms (by default).
    - vsl 15y ago
      That's not what is meant by "safe" here. Safe is when you can be sure that the data were written (into the journal at least) by the time the write call in your code returns. Doing it up to 100ms later leaves a wide window open when the application believes the data are safely stored, while they are in fact in RAM only.
- siavosh 15y ago
  I would bet that there are many apps out there that don't do any reconciliation, and the data will forever be lost unknown. Sadly only 1% of the customers will notice something weird, 0.001% of those will call support saying something is off, and then 99% of those calls will be ignored as customer incompetence. Scary indeed.
axisK 15y ago
Nice article, the write lock has really been making me think about whether it's really the way to go in our own stack.
sirn 15y ago
> We changed the structure of our heaviest used models a couple times in the past year, and instead of going back and updating millions of old documents, we simply added a “version” field to the document and the application handled the logic of reading both the old and new version. This flexibility was useful for both application developers and operations engineers. Ugh, this sounds like a maintenance nightmare. How do you deal with adding extra field to the document? Do you ever feel the need of running on-the-fly migration of old versions? (But when you do, shouldn't running a migration for all documents a better idea?) I'll admit I'm a non-believer, but every time I see "Schemaless" in MongoDB, I think "oh, so you're implementing schema in your application?"
- mitchellh 15y ago
  > Ugh, this sounds like a maintenance nightmare. How do you deal with adding extra field to the document? Do you ever feel the need of running on-the-fly migration of old versions? (But when you do, shouldn't running a migration for all documents a better idea?) Yes, we did on-the-fly migration as we loaded old data in. Doing full data migration was not really an option because querying from MongoDB on un-indexed data is so slow, and paging in all that data would purge hot data, exacerbating the problem. > I'll admit I'm a non-believer, but every time I see "Schemaless" in MongoDB, I think "oh, so you're implementing schema in your application?" That's exactly what happens.
  - judofyr 15y ago
    So it's actually 100% the same as you would do with on-the-fly migrations in SQL: (1) Add column and add code moves the old data when you access it. Deploy. (2) Let it run for a while. Run a background job that migrates the rest (this might be done days or months later). (3) Remove the column and the custom code. The more I hear about "schemaless" the more I realize that it doesn't make any difference at all.
    - scubaguy 15y ago
      For a large table, adding a new column can be non-trivial, each record is updated when the new column is added.
deleted 15y ago
[deleted]
luca_garulli 15y ago
Hey, by reading all the bad things seems that OrientDB would fit better than MongoDB for them: - Non-counting B-Trees: OrientDB uses MVRB-Tree that has the counter. size() requires 0ns - Poor Memory Management: OrientDB uses MMAP too but with many settings to optimize it usage - Uncompressed field names: the same as OrientDB - Global write lock: this kills your concurrency! OrientDB handles read/write locks at segment level so it's really multi-thread under the hood - Safe off by default: the same as OrientDB (turn on synch to stay safe or use good HW/multiple servers) - Offline table compaction: OrientDB compacts at each update/delete so the underlying segments are always well defragmented - Secondaries do not keep hot data in RAM: totally different because OrientDB is multi-master Furthermore you have Transactions, SQL and support for Graphs. Maybe they could avoid to use a RDBMS for some tasks using OrientDB for all. My 0,02.
- deleted 15y ago
  [deleted]
- mtrn 15y ago
  Thanks for writing OrientDB! - I tried it, but I was pressed for time, so I needed something that more or less worked instantly for my requirements - which in the end was elasticsearch. TL; I researched MongoDB and OrientDB for a side-project with a bit heavy data structure (10M+ docs, 800+ fields on two to three levels). MongoDB was blazingly fast, but it segfaulted somewhere in the process (also index creation needs extra time and isn't really ad-hoc). OrientDB wasn't as fast and a little harder to do the initial setup but the inserting speed was ok - for a while (500k docs or so) and then it degraded. I also looked at CouchDB, but I somehow missed the ad-hoc query infrastructure. My current solution, which works nice for the moment is elasticsearch; it's fast - and it's possible to get a prototype from 0 to 10M docs in about 50 minutes - or less, if you load balance the bulk inserts on a cluster - which is so easy to setup it's scary - and then let a full copy of the data settle on each machine in the background. Disclaimer - since this is a side project, I did only minimal research on each of the technologies (call it 5 minute test) and ES clearly won the first round over both MongoDB and OrientDB.
  - AdrianRossouw 15y ago
gregbair 15y ago
If you're going to have a (management|engineering|whatever) blog for your company/project, have a link to your company/project home page prominently somewhere on the blog.
- pearkes 15y ago
  Good point. For the record, we're at http://kiip.me http://kiip.me
Philadelphia 15y ago
The impression I got after hearing some of the 10gen developers speak at a conference is that MongoDB has the same essential problem as PHP. It was written by people without a lot of formal knowledge who, for whatever reason, aren't interested in researching what's been tried before, what works, and what doesn't. Because of that, they're always trying to reinvent the wheel, and make flawed design decisions that keep causing problems.
- brey 15y ago
  which conference? I was at the recent Stockholm conference, had a lengthy 1-1 with one of their kernel devs, and I picked up the opposite impression.
- latch 15y ago
  One of the silliest things I've read on HN, largely because of how insulting it is, yet how easy it is to verify. Rather than making a comment that can be summed up as "i think it's written by people who don't know what they are doing", why not Google them and find out what their background is? The developers at the conference might not have represented the kernel team (it's a surprisingly large company with a lot of different development branches (core, drivers, tools, support)).
- deleted 15y ago
  [deleted]
aschobel 15y ago
We love MongoDB at Catch, it's been our primary backing store for all user data for over 20 months now. > Catch.com > Data Size: 50GB > Total Documents 27,000,000 > Operations per second: 450 (Create, reads, updates, etc.) > Lock % average 0% > CPU load average 0% Global Lock isn't ideal, but Mongo is so fast it hasn't been an issue for us. You need to keep on slow queries and design your schema and indexes correctly. We don't want page faults on indexes, we design them to keep them in memory. I don't get the safety issue, 20 months and we haven't lost any user data. shrug
- bretthoerner 15y ago
  > I don't get the safety issue, 20 months and we haven't lost any user data. shrug Nobody loses any user data until they do.
  - siavosh 15y ago
    This should be a deal breaker for any serious app. Does the performance hit of safe mode negate all other advantages of MongoDB?
    - madworld 15y ago
      That's most people's findings. If your dataset can fit in ram [1] and you don't care about your data being safe then there might be an argument for MongoDB. Once you care about your data, things like Voldemort, Riak, and Cassandra will eat Mongo's lunch on speed. [1] But as Artur Bergman so eloquently points out, if your data can fit in ram, just use a native data-structure (http://youtu.be/oebqlzblfyo?t=13m35s http://youtu.be/oebqlzblfyo?t=13m35s)
    - aschobel 15y ago
      Do you have a citation to back up the claim that you shouldn't use MongoDB for serious apps? We have done billions of ops with Mongo and have never lost any data.
    - aschobel 15y ago
nomoremongo 15y ago
Yyyyyyyup. http://pastebin.com/raw.php?i=FD3xe6Jt http://pastebin.com/raw.php?i=FD3xe6Jt
jalons 15y ago
Am I missing something, or did they say they didn't want to scale mongo horizontally via sharding, then comment that they're doing so with riak, but faulting mongodb for requiring it?
- madworld 15y ago
  What they are doing with Riak isn't sharding. Riak from the ground up was been designed as a distributed database. They didn't want to go horizontal when really they shouldn't have to with their datasize based on Mongo's claims. The problem is, Mongo lies about what their database can do, and the fact that Kiip figured that out is why they didn't want to bother scaling out with mongo as a band-aid for its problems. It was better for them to just use something made to scale. That's how I read it, based on that blog and by his comments on this post.
mrkurt 15y ago
Part of the lesson here is that if you're doing MongoDB on EC2, you should have more than enough RAM for your working set. EBS is pretty bad underlying IO for databases, so you should treat your drives more as a relatively cold storage engine. This is the primary reason we're moving the bulk of our database ops to real hardware with real arrays (and Fusion IO cards for the cool kids). We have a direct connect to Amazon and actual IO performance... it's great.
- mitchellh 15y ago
  > Part of the lesson here is that if you're doing MongoDB on EC2, you should have more than enough RAM for your working set. We had more than enough RAM for our working set. Unfortunately, due to MongoDB's poor memory managed and non-counting B-trees, even our hot data would sometimes be purged out of memory for cold, unused data, causing serious performance degradation.
  - jasonmccay 15y ago
    I understand your point, but the performance issues still stem off of poor IO performance on Amazon EBS. As we continue to use it, we continue to find it to be the source of most people's woes. If you have solid (even reasonable) IO, then moving things in and out of working memory is not painful. We have some customers on non-EBS spindles that have very large working sets (as compared to memory) ... faulting 400-500 times per second, and hardly notice performance slow downs. I think your suggestions are legit, but faulting performance has just as much to do with IO congestion. That applies to insert/update performance as well.
    - ismarc 15y ago
      We are using Mongo in ec2 and raid10 with 6 ebs drives out performs ephemeral disks when the dataset won't fit in RAM in a raw upsert scenario (our actual data, loading in historical data). The use if mmap and relying on the OS to page in/out the appropriate portions is painful, particularly because we end up with a lot of moves (padding factor varies between 1.8 and 1.9 and because of our dataset, using a large field on insert and clearing in update was less performant than upserts and moves). There's really two knobs to turn on Mongo, RAM and disk speed. Our particular cluster doesn't have enough RAM for the dataset to fit in memory, but could double its performance (or more) if each key range was mmapped individually rather than the entire datastore the shard is responsible for just because of how the OS manages pages. We haven't broken down to implement it yet, but with the performance vs. cost tradeoffs, we may have to pretty soon.
deleted 15y ago
[deleted]
lalmalang 15y ago
We had a very similar situation ~300 writes per second on AWS. but I suspect some of this has to do with the fact that most people address scaling by adding a replica set, rather than the much hairier sharding setup (http://www.mongodb.org/display/DOCS/Sharding+Introduction http://www.mongodb.org/display/DOCS/Sharding+Introduction), this seems natural b/c mongodb's 'scalability' is often touted. In reality though, because of the lock, RS dont really address the problem much, and we encountered many of the problems described by the OP. Not to denigrate the work the 10gen guys are doing -- they are obviously working on a hard problem, and were very helpful, and the mms dashboard was nice to pinpoint issues. We decided to switch too though in the end, though i still enjoy using mongo for small stuff here and there
- disbelief 15y ago
  Mind sharing what you switched to? Another schemaless data store, or a more traditional RDBMS?
  - lalmalang 15y ago
    For the moment to Cassandra but very tempted to look into hbase in more detail soon...
- jasonmccay 15y ago
  Again ... this is more an indictment on the poor IO performance of Amazon EBS vs. MongoDB as a solution. MongoDB can scale both vertically and horizontally, but as with anything you scale on Amazon infrastructure, you are going to have to really think through strategies for dealing with the unpredictable performance of EBS. There are blog posts galore addressing this fact. I often think MongoDB has suffered more as a young technology because of the proliferation of the AWS Cloud and the expectations of EBS performance.
  - lalmalang 15y ago
    In fact, I tried on-instance storage too -- this didnt help substantially. The reality is that many (most?) stacks these days need to be able live happily on AWS...
radagaisus 15y ago
Start Up Bloggers listen up! If I click on your blog's logo I want to see your product, not the blog's main page.
tolitius 15y ago
great post. direct and to the point, although there are many more flaws that I am sure you could have shared. we tried MongoDB to consume and analyze market feeds, and it failed miserably. I can add a couple of things to your list: * if there is a pending write due to an fsync lock, all reads are blocked: https://jira.mongodb.org/browse/SERVER-4243 https://jira.mongodb.org/browse/SERVER-4243 * data loss + 10gen's white lies: https://jira.mongodb.org/browse/SERVER-3367?focusedCommentId=66490&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-66490 https://jira.mongodb.org/browse/SERVER-3367?focusedCommentId... * _re_ sharding is hard. shard key is should be chosen once and for all => that alone kills the schemaless advantage * moving chunks between shards [manually or auto] can take hours / days depending on the dataset (but we talking big data, right?) * aggregate (if any complex: e.g. not SUM, COUNT, MIN, MAX) over several gigs of data takes hours (many minutes at best). Not everything can be incremental.. Those are just several. MongoDB has an excellent marketing => Meghan Gill is great at what she does. But besides that, the tech is not quite there (yet?). Nice going with Riak + PostgreSQL. I would also give Redis a try for things that you keep in memory => set theory ftw! :)
- armon 15y ago
  I work at Kiip, and I can confirm that our "non-durable purely in-memory solution" is Redis.
- taligent 15y ago
  MongoDB is successful because of more than just marketing. It has great tool support, decent documentation, books and is accessible. Plus the whole transition from MySQL concept makes it easy to grab onto.
  - chad_walters 15y ago
    That all supports the marketing effort. Mongo is optimized for a good out-of-the-box experience for developers. It is basically the MySQL model -- hook developers first, fix the fundamentals later. Caveat emptor.
    - 15y ago
Jebus 15y ago
Uncompressed field names - If you store 1,000 documents with the key “foo”, then “foo” is stored 1,000 times in your data set Oh my god. I didn't know about this. And I hate short, meaningless and anti-intuitive field names. Please fix it mongodb devs!
- gatesvp1 14y ago
  This is a long-outstanding bug, over 2 years old now: https://jira.mongodb.org/browse/SERVER-863 https://jira.mongodb.org/browse/SERVER-863 Obviously you can vote for them to fix the bug. But it's been two years, so I'm not sure it's really high on the priority list.
pixelmonkey 15y ago
This is a pretty epic troll on MongoDB, and some of their points are important -- particularly global write lock and uncompressed field names, both issues that needlessly afflict large MongoDB clusters and will likely be fixed eventually. However, it's pretty clear from this post that they were not using MongoDB in the best way. For example, in a small part of their criticism of "safe off by default", they write: "We lost a sizable amount of data at Kiip for some time before realizing what was happening and using safe saves where they made sense (user accounts, billing, etc.)." You shouldn't be storing user accounts and billing information in MongoDB. Perhaps MongoDB's marketing made you believe you should store everything in MongoDB, but you should know better. In addition to that data being highly relational, it also requires the transactional semantics present in mature relational databases. When I read "user accounts, billing" here, I cringed. Things that it makes total sense to use MongoDB for: - analytics systems: where server write thorughput, client-side async (unsafe) upserts/inserts, and the atomic $inc operator become very valuable tools. http://blog.mongodb.org/post/171353301/using-mongodb-for-real-time-analytics http://blog.mongodb.org/post/171353301/using-mongodb-for-rea... - content management systems: where schema-free design, avoidance of joins, its query language, and support for arbitrary metadata become an excellent set of tradeoffs vs. tabular storage in an RDBMS. http://www.mongodb.org/display/DOCS/How+MongoDB+is+Used+in+Media+and+Publishing http://www.mongodb.org/display/DOCS/How+MongoDB+is+Used+in+M... - document management systems: I have used MongoDB with great sucess as the canonical store of documents which are then indexed in a full-text search engine like Solr. You can do this kind of storage in an RDBMS, but MongoDB has less administrative overhead, a simpler development workflow, and less impedance mismatch with document-based stores like Solr. Further, with GridFS, you can even use MongoDB as a store for actual files, and leverage MongoDB's replica sets for spreading those files across machines. Is your data relational? Can you benefit from transactional semantics? Can you benefit from on-the-fly data aggregation (SQL aggregates)? Then use a relational database! Using multiple data stores is a reality of all large-scale technology companies. Pick the right tool for the right job. At my company, we use MongoDB, Postgres, Redis, and Solr -- and we use them each on the part of our stack where we leverage their strengths and avoid their weaknesses. This article reads to me like someone who decided to store all of their canonical data for an e-commerce site in Solr, and then complains when they realized that re-indexing their documents takes a long time, index corruption occurs upon Solr/Lucene upgrades, or that referential integrity is not supported. Solr gives you excellent full-text search, and makes a lot of architectural trade-offs to achieve this. Such is the reality of technology tools. What, were you expecting Solr to make your coffee, too? Likewise, MongoDB made a lot of architectural tradeoffs to achieve the goals it set out in its vision, as described here: http://www.mongodb.org/display/DOCS/Philosophy http://www.mongodb.org/display/DOCS/Philosophy It may be a cool technology, but no, it won't make your coffee, too. In the end, the author writes, "Over the past 6 months, we've scaled MongoDB by moving data off of it. [...] we looked at our data access patterns and chose the right tool for the job. For key-value data, we switched to Riak, which provides predictable read/write latencies and is completely horizontally scalable. For smaller sets of relational data where we wanted a rich query layer, we moved to PostgreSQL." Excellent! They ended up in the right place.
DonnyV 15y ago
Please upvote collection level locking for MongoDB here. https://jira.mongodb.org/browse/SERVER-1240 https://jira.mongodb.org/browse/SERVER-1240

362 points by mitchellh 15y ago

Not related to the article but the site: has anyone else been getting "connection interrupted" errors with tumblr recently? If I load a tumblr blog for the first time in ~24 hours the first and second page loads will result in connection interrupted, the 3rd and beyond will all load fine.

jrussbowman 15y ago

From the beginning I've understood mongodb to be built with it's approach for scaling, performance, redundancy and backup to be horizontal scaling. They recently added journaling for single server durability, but before that replication was how you made sure you data was safe. It seems to me when I see complaints about mongodb it's because people don't want to horizontally scale it and instead believe vertical scaling should be more available. Just seems to me people don't like how mongodb is built, but if used as intended I think mongodb performs as advertised. In most cases I don't think it's the tool, rather the one using it.

LeafStorm 15y ago
While I agree that you should definitely use your tools in the best way you're capable of, I think for most people there's a baseline expectation that if you save data in a database, that data will be safe (at the very least, recoverable) unless something happens like the server catching on fire. Nearly every other major database has this as the default -- MySQL, PostgreSQL, CouchDB, and Berkeley DB to name a few. (Redis doesn't, but it's also very upfront about it, and does provide this kind of durability as an option from early on.) So when MongoDB breaks this expectation, and when asked to support it as an option, just says, "That's what more hardware is for," it's a pretty big turnoff.
- antirez 15y ago
  You are wrong about Redis. Cheers
  - justinsb 15y ago
    Care to explain? I believe for Redis, "appendfsync everysec" is the default. The poster's point was that MySQL and Postgres both ship with something like "appendfsync always", and you have to opt-in to the the less safe mode if you want to get more performance. Redis ships with the less safe mode pre-selected, and so has higher performance out-of-the-box.
    - corford 15y ago
      You're right. The Postgres equivalent to "appendfsync always" is "synchronous_commit = on". Which AFAIK is the default. However, one of the nice things about redis is that even if you run "appendfsync everysec" you never run the risk of corruption. You're only risk is losing a maximum of 2 seconds worth of data. If you missed it, there's a wonderful blog post by antirez covering all of this (and a lot more) here: http://antirez.com/post/redis-persistence-demystified.html http://antirez.com/post/redis-persistence-demystified.html
      - justinsb 15y ago
        All modern relational databases use this approach (it's called ARIES), so all offer the same guarantees as Redis.
        moe 15y ago
        And to praise redis some more; it also comes with the sanest (async) replication support of any database. By far. It's literally a one-liner in the config file. No bootstrap needed, maintenance free, absolutely no strings attached. 10 seconds and you're done. Which means you'll actually use it from day 1 and never worry about it. Compare that to any other database.
    - deleted 15y ago
      [deleted]
- mnutt 15y ago
  There's actually two separate concerns: Mongo client's "safe" operator causes the client to see if the database threw an error and throw an error itself. Like someone mentioned, it mostly falls on the mongo clients to implement this. We mostly use fire-and-forget for our application, since it is just logging stats and speed is more important than losing an increment here or there. There should probably be better documentation telling people to always use the safe operations for important data. There is also the durability issue. Early versions shipped with durability turned off by default and required replication to maintain durability. Mongo has had the journal feature since 1.8 and has it enabled by default since 1.9.2. (current version is 2.0) So while mongo has definitely been unsafe in the past, both kinds of safety are now supported, and one is default. The other is either not that big a deal or egregious, depending on the way you view mongo.
paulsutter 15y ago
You're right that MongoDB is designed from the start for horizontal scaling, and the author could have better articulated his reluctance to add machines. However, his suggestions would improve mongodb performance in all configurations, not limited to single node. The mongodb guys did a great job simplfying the internal architecture and focusing on a great API to get the product out quickly. I'm confident they can make these sorts of internal improvements to broaden the use of mongodb.
- heretohelp 15y ago
  >You're right that MongoDB is designed from the start for horizontal scaling, Nope.
mitchellh 15y ago
[Note: I wrote the blog post] I'm not at all against horizontally scaling. However, I don't believe that horizontally scaling should be necessary doing a mere 200 updates to per second to a data store that isn't even fsyncing writes to disk. Think of it in terms of cost per ops. Let's just say 200 update ops per second is the point at which you need to shard (not scientific, but let's just use that as a benchmark since that is what we saw at Kiip). MongoDB likes memory, so let's use high-memory AWS instances as a cost benchmark. I think this is fair since MongoDB advertises itself as a DB built for the cloud. The cheapest high-memory instance is around $330/month. That gives you a cost per op of 6.37e-5 cents per update operation. Let's compare this to PostgreSQL, which we've had in production for a couple months at Kiip now. Our PostgreSQL master server has peaked at around 1000 updates per second without issue, and also with the bonus that it doesn't block reads for no reason. The cost per op for PostgreSQL is 1.27e-5 cents. Therefore, if you're swimming in money, then MongoDB seems like a great way to scale. However, we try to be more efficient with our infrastructure expenditures. EDIT: Updated numbers, math is hard.
- zzzeek 15y ago
  typo in the numbers ? I can get your mongoDB number but not pg: >>> seconds_per_month = 60 * 60 * 24 * 30 >>> ops_per_month_pg = 1000 * seconds_per_month >>> ops_per_month_mg = 200 * seconds_per_month >>> 330.0 / ops_per_month_pg * 100 1.2731481481481482e-05 >>> 330.0 / ops_per_month_mg * 100 6.36574074074074e-05
  - mitchellh 15y ago
    You're absolutely right. Decimal points are hard. Fixed my comment (and noted it).
  - nivertech 15y ago
    You could have skipped all the calculations and just say, that 1000/200=5, I.e PostgreSQL 5 times more cost effective than MongoDB.
- jabwork 15y ago
  Can you give any sort of indication of the value of a schemaless database and the flexibility it provided as the team fleshed out the data model? Was this a mere convenience over traditional schema migration or something more?
cheald 15y ago
I think the article is a fair criticism, and I think your response is likewise fair. Mongo was built with horizontal scaling in mind, and to that end, it tends to suffer noticeably when you overload a single node. Things like the global write lock and single-threaded map/reduce are definitely problems, and shouldn't be pooh-pooh'd away as "oh, just scale horizontally". Uncompacted key names are a real problem, and a consequence of it is that Mongo tends to take more disk/RAM for the same data (+indexes) than a comparable RDBMS does. Maintenance is clunky - you end up having to repair or compact each slave individually, then step down your master and repair or compact it, then step it back up (which does work decently, but is time consuming!) None of these are showstoppers, but they are pain points, especially if you're riding the "Mongo is magical scaling sauce" hype train. It takes a lot of work to really get it humming, and once you do, it's pretty damn good at the things it's good at, but there are gotchas and they will bite you if you aren't looking out for them.
- ericflo 15y ago
  Mongo was not designed with horizontal scaling in mind. Riak, Cassandra, HBase, Project Voldemort...these are the projects that were designed with horizontal scaling in mind (as evidenced by their architectures.) But not Mongo.
  - jasonmccay 15y ago
    I have to respectfully disagree. Your comment is a bit sweeping and I really don't think that MongoDB's sharding solution is a bad one ... simply the strategies are different. There is a large set of nice features that makes Mongo, for most people, nice to use long before you even need to address sharding. The percentage of people that will need to shard is much lower than the percentage of people that can get considerable benefits of how you can query data in MongoDB, for example, vs. Riak. You are such a Riak-lover. :P
    - ericflo 15y ago
      I am definitely a Riak-lover :) And I also agree that there is a large set of nice features that make Mongo nice to use before sharding.
saurik 15y ago
The behavior described in the article, though, is just a fundamental misuse of the hardware resources. If I'm hitting write bottlenecks, I need to shard: that's just a fact of reality. However, write throughput and read throughput should be unrelated (as durable writes are bottlenecks by disk and reads of hot data are bottlenecked by CPU/RAM): if I have saturated five machines with writes I should have five machines worth of read throughput, and that is a /lot/ of throughput... with the behavior in this article, you have no read capacity as you are blocked by writes, even if the writes are to unrelated collections of data (which is downright non-sensical behavior). Even with more machines (I guess, twice as many), the latency of your reads is now being effected horribly by having to wait for the write locks... it is just a bad solution.

paulsutter 15y ago

This article is an excellent articulation of the strengths and (fixable) issues with mongoDB. I like MongoDB a lot, and the improvements suggested would really strengthen the product and could make me more comfortable to use it in more serious applications.

madworld 15y ago
How is the global write lock "fixable" without a major rewrite of the codebase? Like the article suggested, it would be one thing if they did it for transaction support. In reality, from looking at the code, it seems like the global write lock came from not wanting to solve the hard problems other people are solving.
- geoffeg 15y ago
  DB-level locking is planned for MongoDB 2.2 which should be out within a few months. https://jira.mongodb.org/browse/SERVER-4328 https://jira.mongodb.org/browse/SERVER-4328
  - spidaman 15y ago
    Meh, if your other option is to use PostgreSQL and get row level locks, a db level lock is still a fail.
    - j-kidd 15y ago
      And here's a great post that provides some insight on how much effort has been put in by RDBMS vendors to handle locking: http://stackoverflow.com/a/872808 http://stackoverflow.com/a/872808
- paulsutter 15y ago
  A major rewrite of the core engine is exactly what's needed. Sounds like a fun project. If they don't do it, someone else will. Adoption is hard to replace. Modifying API's is really hard. Rewriting a core engine? Reasonable and in this case probably necessary given the issues. There are lots of people here who could replace that core engine and someone should if the MongoDB guys can't.

rdtsc 15y ago

> Safe off by default I think that is fixed now. But this the single most appalling design decision they could have made while also claiming their product was a "database". (And this has been discussed here before so just do a search if you will). This wasn't a bug, it was a deliberate design decision. Ok, that that would have been alright if they put a bright red warning on their front page. "We disabled durability by default (even though we call this product a database). If you run this configuration as a single server your data could be silently corrupted. Proceed at your own risk". But, they didn't. I call that being "shady". Yap, you read the correctly, you'd issue a write and there would be no result coming back acknowledging that at least the data made into the OS buffer. I can only guess their motive was to look good in benchmarks that everyone like to run and post results on the web. But there were (still are) real cases of people's data being silently corrupted, and they noticed only much later, when backups, and backups of backups were already made.

jbellis 15y ago
According to http://www.mongodb.org/display/DOCS/getLastError+Command http://www.mongodb.org/display/DOCS/getLastError+Command it is still unsafe by default.
- latch 15y ago
  No, the journal file is turned on server-side and you'll get a journal append every 100ms (by default).
  - vsl 15y ago
    That's not what is meant by "safe" here. Safe is when you can be sure that the data were written (into the journal at least) by the time the write call in your code returns. Doing it up to 100ms later leaves a wide window open when the application believes the data are safely stored, while they are in fact in RAM only.
siavosh 15y ago
I would bet that there are many apps out there that don't do any reconciliation, and the data will forever be lost unknown. Sadly only 1% of the customers will notice something weird, 0.001% of those will call support saying something is off, and then 99% of those calls will be ignored as customer incompetence. Scary indeed.

axisK 15y ago

Nice article, the write lock has really been making me think about whether it's really the way to go in our own stack.

sirn 15y ago

> We changed the structure of our heaviest used models a couple times in the past year, and instead of going back and updating millions of old documents, we simply added a “version” field to the document and the application handled the logic of reading both the old and new version. This flexibility was useful for both application developers and operations engineers. Ugh, this sounds like a maintenance nightmare. How do you deal with adding extra field to the document? Do you ever feel the need of running on-the-fly migration of old versions? (But when you do, shouldn't running a migration for all documents a better idea?) I'll admit I'm a non-believer, but every time I see "Schemaless" in MongoDB, I think "oh, so you're implementing schema in your application?"

mitchellh 15y ago
> Ugh, this sounds like a maintenance nightmare. How do you deal with adding extra field to the document? Do you ever feel the need of running on-the-fly migration of old versions? (But when you do, shouldn't running a migration for all documents a better idea?) Yes, we did on-the-fly migration as we loaded old data in. Doing full data migration was not really an option because querying from MongoDB on un-indexed data is so slow, and paging in all that data would purge hot data, exacerbating the problem. > I'll admit I'm a non-believer, but every time I see "Schemaless" in MongoDB, I think "oh, so you're implementing schema in your application?" That's exactly what happens.
- judofyr 15y ago
  So it's actually 100% the same as you would do with on-the-fly migrations in SQL: (1) Add column and add code moves the old data when you access it. Deploy. (2) Let it run for a while. Run a background job that migrates the rest (this might be done days or months later). (3) Remove the column and the custom code. The more I hear about "schemaless" the more I realize that it doesn't make any difference at all.
  - scubaguy 15y ago
    For a large table, adding a new column can be non-trivial, each record is updated when the new column is added.

deleted 15y ago

[deleted]

luca_garulli 15y ago

Hey, by reading all the bad things seems that OrientDB would fit better than MongoDB for them: - Non-counting B-Trees: OrientDB uses MVRB-Tree that has the counter. size() requires 0ns - Poor Memory Management: OrientDB uses MMAP too but with many settings to optimize it usage - Uncompressed field names: the same as OrientDB - Global write lock: this kills your concurrency! OrientDB handles read/write locks at segment level so it's really multi-thread under the hood - Safe off by default: the same as OrientDB (turn on synch to stay safe or use good HW/multiple servers) - Offline table compaction: OrientDB compacts at each update/delete so the underlying segments are always well defragmented - Secondaries do not keep hot data in RAM: totally different because OrientDB is multi-master Furthermore you have Transactions, SQL and support for Graphs. Maybe they could avoid to use a RDBMS for some tasks using OrientDB for all. My 0,02.

deleted 15y ago
[deleted]
mtrn 15y ago
Thanks for writing OrientDB! - I tried it, but I was pressed for time, so I needed something that more or less worked instantly for my requirements - which in the end was elasticsearch. TL; I researched MongoDB and OrientDB for a side-project with a bit heavy data structure (10M+ docs, 800+ fields on two to three levels). MongoDB was blazingly fast, but it segfaulted somewhere in the process (also index creation needs extra time and isn't really ad-hoc). OrientDB wasn't as fast and a little harder to do the initial setup but the inserting speed was ok - for a while (500k docs or so) and then it degraded. I also looked at CouchDB, but I somehow missed the ad-hoc query infrastructure. My current solution, which works nice for the moment is elasticsearch; it's fast - and it's possible to get a prototype from 0 to 10M docs in about 50 minutes - or less, if you load balance the bulk inserts on a cluster - which is so easy to setup it's scary - and then let a full copy of the data settle on each machine in the background. Disclaimer - since this is a side project, I did only minimal research on each of the technologies (call it 5 minute test) and ES clearly won the first round over both MongoDB and OrientDB.
- AdrianRossouw 15y ago

gregbair 15y ago

If you're going to have a (management|engineering|whatever) blog for your company/project, have a link to your company/project home page prominently somewhere on the blog.

pearkes 15y ago
Good point. For the record, we're at http://kiip.me http://kiip.me

Philadelphia 15y ago

The impression I got after hearing some of the 10gen developers speak at a conference is that MongoDB has the same essential problem as PHP. It was written by people without a lot of formal knowledge who, for whatever reason, aren't interested in researching what's been tried before, what works, and what doesn't. Because of that, they're always trying to reinvent the wheel, and make flawed design decisions that keep causing problems.

brey 15y ago
which conference? I was at the recent Stockholm conference, had a lengthy 1-1 with one of their kernel devs, and I picked up the opposite impression.
latch 15y ago
One of the silliest things I've read on HN, largely because of how insulting it is, yet how easy it is to verify. Rather than making a comment that can be summed up as "i think it's written by people who don't know what they are doing", why not Google them and find out what their background is? The developers at the conference might not have represented the kernel team (it's a surprisingly large company with a lot of different development branches (core, drivers, tools, support)).
deleted 15y ago
[deleted]

aschobel 15y ago

We love MongoDB at Catch, it's been our primary backing store for all user data for over 20 months now. > Catch.com > Data Size: 50GB > Total Documents 27,000,000 > Operations per second: 450 (Create, reads, updates, etc.) > Lock % average 0% > CPU load average 0% Global Lock isn't ideal, but Mongo is so fast it hasn't been an issue for us. You need to keep on slow queries and design your schema and indexes correctly. We don't want page faults on indexes, we design them to keep them in memory. I don't get the safety issue, 20 months and we haven't lost any user data. shrug

bretthoerner 15y ago
> I don't get the safety issue, 20 months and we haven't lost any user data. shrug Nobody loses any user data until they do.
- siavosh 15y ago
  This should be a deal breaker for any serious app. Does the performance hit of safe mode negate all other advantages of MongoDB?
  - madworld 15y ago
    That's most people's findings. If your dataset can fit in ram [1] and you don't care about your data being safe then there might be an argument for MongoDB. Once you care about your data, things like Voldemort, Riak, and Cassandra will eat Mongo's lunch on speed. [1] But as Artur Bergman so eloquently points out, if your data can fit in ram, just use a native data-structure (http://youtu.be/oebqlzblfyo?t=13m35s http://youtu.be/oebqlzblfyo?t=13m35s)
  - aschobel 15y ago
    Do you have a citation to back up the claim that you shouldn't use MongoDB for serious apps? We have done billions of ops with Mongo and have never lost any data.
  - aschobel 15y ago

nomoremongo 15y ago

Yyyyyyyup. http://pastebin.com/raw.php?i=FD3xe6Jt http://pastebin.com/raw.php?i=FD3xe6Jt

jalons 15y ago

Am I missing something, or did they say they didn't want to scale mongo horizontally via sharding, then comment that they're doing so with riak, but faulting mongodb for requiring it?

madworld 15y ago
What they are doing with Riak isn't sharding. Riak from the ground up was been designed as a distributed database. They didn't want to go horizontal when really they shouldn't have to with their datasize based on Mongo's claims. The problem is, Mongo lies about what their database can do, and the fact that Kiip figured that out is why they didn't want to bother scaling out with mongo as a band-aid for its problems. It was better for them to just use something made to scale. That's how I read it, based on that blog and by his comments on this post.

mrkurt 15y ago

Part of the lesson here is that if you're doing MongoDB on EC2, you should have more than enough RAM for your working set. EBS is pretty bad underlying IO for databases, so you should treat your drives more as a relatively cold storage engine. This is the primary reason we're moving the bulk of our database ops to real hardware with real arrays (and Fusion IO cards for the cool kids). We have a direct connect to Amazon and actual IO performance... it's great.

mitchellh 15y ago
> Part of the lesson here is that if you're doing MongoDB on EC2, you should have more than enough RAM for your working set. We had more than enough RAM for our working set. Unfortunately, due to MongoDB's poor memory managed and non-counting B-trees, even our hot data would sometimes be purged out of memory for cold, unused data, causing serious performance degradation.
- jasonmccay 15y ago
  I understand your point, but the performance issues still stem off of poor IO performance on Amazon EBS. As we continue to use it, we continue to find it to be the source of most people's woes. If you have solid (even reasonable) IO, then moving things in and out of working memory is not painful. We have some customers on non-EBS spindles that have very large working sets (as compared to memory) ... faulting 400-500 times per second, and hardly notice performance slow downs. I think your suggestions are legit, but faulting performance has just as much to do with IO congestion. That applies to insert/update performance as well.
  - ismarc 15y ago
    We are using Mongo in ec2 and raid10 with 6 ebs drives out performs ephemeral disks when the dataset won't fit in RAM in a raw upsert scenario (our actual data, loading in historical data). The use if mmap and relying on the OS to page in/out the appropriate portions is painful, particularly because we end up with a lot of moves (padding factor varies between 1.8 and 1.9 and because of our dataset, using a large field on insert and clearing in update was less performant than upserts and moves). There's really two knobs to turn on Mongo, RAM and disk speed. Our particular cluster doesn't have enough RAM for the dataset to fit in memory, but could double its performance (or more) if each key range was mmapped individually rather than the entire datastore the shard is responsible for just because of how the OS manages pages. We haven't broken down to implement it yet, but with the performance vs. cost tradeoffs, we may have to pretty soon.

deleted 15y ago

[deleted]

lalmalang 15y ago

We had a very similar situation ~300 writes per second on AWS. but I suspect some of this has to do with the fact that most people address scaling by adding a replica set, rather than the much hairier sharding setup (http://www.mongodb.org/display/DOCS/Sharding+Introduction http://www.mongodb.org/display/DOCS/Sharding+Introduction), this seems natural b/c mongodb's 'scalability' is often touted. In reality though, because of the lock, RS dont really address the problem much, and we encountered many of the problems described by the OP. Not to denigrate the work the 10gen guys are doing -- they are obviously working on a hard problem, and were very helpful, and the mms dashboard was nice to pinpoint issues. We decided to switch too though in the end, though i still enjoy using mongo for small stuff here and there

disbelief 15y ago
Mind sharing what you switched to? Another schemaless data store, or a more traditional RDBMS?
- lalmalang 15y ago
  For the moment to Cassandra but very tempted to look into hbase in more detail soon...
jasonmccay 15y ago
Again ... this is more an indictment on the poor IO performance of Amazon EBS vs. MongoDB as a solution. MongoDB can scale both vertically and horizontally, but as with anything you scale on Amazon infrastructure, you are going to have to really think through strategies for dealing with the unpredictable performance of EBS. There are blog posts galore addressing this fact. I often think MongoDB has suffered more as a young technology because of the proliferation of the AWS Cloud and the expectations of EBS performance.
- lalmalang 15y ago
  In fact, I tried on-instance storage too -- this didnt help substantially. The reality is that many (most?) stacks these days need to be able live happily on AWS...

radagaisus 15y ago

Start Up Bloggers listen up! If I click on your blog's logo I want to see your product, not the blog's main page.

tolitius 15y ago

great post. direct and to the point, although there are many more flaws that I am sure you could have shared. we tried MongoDB to consume and analyze market feeds, and it failed miserably. I can add a couple of things to your list: * if there is a pending write due to an fsync lock, all reads are blocked: https://jira.mongodb.org/browse/SERVER-4243 https://jira.mongodb.org/browse/SERVER-4243 * data loss + 10gen's white lies: https://jira.mongodb.org/browse/SERVER-3367?focusedCommentId=66490&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-66490 https://jira.mongodb.org/browse/SERVER-3367?focusedCommentId... * _re_ sharding is hard. shard key is should be chosen once and for all => that alone kills the schemaless advantage * moving chunks between shards [manually or auto] can take hours / days depending on the dataset (but we talking big data, right?) * aggregate (if any complex: e.g. not SUM, COUNT, MIN, MAX) over several gigs of data takes hours (many minutes at best). Not everything can be incremental.. Those are just several. MongoDB has an excellent marketing => Meghan Gill is great at what she does. But besides that, the tech is not quite there (yet?). Nice going with Riak + PostgreSQL. I would also give Redis a try for things that you keep in memory => set theory ftw! :)

armon 15y ago
I work at Kiip, and I can confirm that our "non-durable purely in-memory solution" is Redis.
taligent 15y ago
MongoDB is successful because of more than just marketing. It has great tool support, decent documentation, books and is accessible. Plus the whole transition from MySQL concept makes it easy to grab onto.
- chad_walters 15y ago
  That all supports the marketing effort. Mongo is optimized for a good out-of-the-box experience for developers. It is basically the MySQL model -- hook developers first, fix the fundamentals later. Caveat emptor.
  - 15y ago

Jebus 15y ago

Uncompressed field names - If you store 1,000 documents with the key “foo”, then “foo” is stored 1,000 times in your data set Oh my god. I didn't know about this. And I hate short, meaningless and anti-intuitive field names. Please fix it mongodb devs!

gatesvp1 14y ago
This is a long-outstanding bug, over 2 years old now: https://jira.mongodb.org/browse/SERVER-863 https://jira.mongodb.org/browse/SERVER-863 Obviously you can vote for them to fix the bug. But it's been two years, so I'm not sure it's really high on the priority list.

pixelmonkey 15y ago

This is a pretty epic troll on MongoDB, and some of their points are important -- particularly global write lock and uncompressed field names, both issues that needlessly afflict large MongoDB clusters and will likely be fixed eventually. However, it's pretty clear from this post that they were not using MongoDB in the best way. For example, in a small part of their criticism of "safe off by default", they write: "We lost a sizable amount of data at Kiip for some time before realizing what was happening and using safe saves where they made sense (user accounts, billing, etc.)." You shouldn't be storing user accounts and billing information in MongoDB. Perhaps MongoDB's marketing made you believe you should store everything in MongoDB, but you should know better. In addition to that data being highly relational, it also requires the transactional semantics present in mature relational databases. When I read "user accounts, billing" here, I cringed. Things that it makes total sense to use MongoDB for: - analytics systems: where server write thorughput, client-side async (unsafe) upserts/inserts, and the atomic $inc operator become very valuable tools. http://blog.mongodb.org/post/171353301/using-mongodb-for-real-time-analytics http://blog.mongodb.org/post/171353301/using-mongodb-for-rea... - content management systems: where schema-free design, avoidance of joins, its query language, and support for arbitrary metadata become an excellent set of tradeoffs vs. tabular storage in an RDBMS. http://www.mongodb.org/display/DOCS/How+MongoDB+is+Used+in+Media+and+Publishing http://www.mongodb.org/display/DOCS/How+MongoDB+is+Used+in+M... - document management systems: I have used MongoDB with great sucess as the canonical store of documents which are then indexed in a full-text search engine like Solr. You can do this kind of storage in an RDBMS, but MongoDB has less administrative overhead, a simpler development workflow, and less impedance mismatch with document-based stores like Solr. Further, with GridFS, you can even use MongoDB as a store for actual files, and leverage MongoDB's replica sets for spreading those files across machines. Is your data relational? Can you benefit from transactional semantics? Can you benefit from on-the-fly data aggregation (SQL aggregates)? Then use a relational database! Using multiple data stores is a reality of all large-scale technology companies. Pick the right tool for the right job. At my company, we use MongoDB, Postgres, Redis, and Solr -- and we use them each on the part of our stack where we leverage their strengths and avoid their weaknesses. This article reads to me like someone who decided to store all of their canonical data for an e-commerce site in Solr, and then complains when they realized that re-indexing their documents takes a long time, index corruption occurs upon Solr/Lucene upgrades, or that referential integrity is not supported. Solr gives you excellent full-text search, and makes a lot of architectural trade-offs to achieve this. Such is the reality of technology tools. What, were you expecting Solr to make your coffee, too? Likewise, MongoDB made a lot of architectural tradeoffs to achieve the goals it set out in its vision, as described here: http://www.mongodb.org/display/DOCS/Philosophy http://www.mongodb.org/display/DOCS/Philosophy It may be a cool technology, but no, it won't make your coffee, too. In the end, the author writes, "Over the past 6 months, we've scaled MongoDB by moving data off of it. [...] we looked at our data access patterns and chose the right tool for the job. For key-value data, we switched to Riak, which provides predictable read/write latencies and is completely horizontally scalable. For smaller sets of relational data where we wanted a rich query layer, we moved to PostgreSQL." Excellent! They ended up in the right place.

DonnyV 15y ago

Please upvote collection level locking for MongoDB here. https://jira.mongodb.org/browse/SERVER-1240 https://jira.mongodb.org/browse/SERVER-1240