8 ms·

We've been using MongoDB in production for our main content store at Conversocial for over a year now. We've gone from 1.8 to 2.0 to 2.2 and we're happy. We hav

We've been using MongoDB in production for our main content store at Conversocial for over a year now. We've gone from 1.8 to 2.0 to 2.2 and we're happy. We have ~400gb of data, 220 million documents all on Amazon's SSD instances. We have definitely had our moments where we've screamed that we hate Mongo and are going to rip it out. That's normally where we've overlooked a detail of how it works... and we've had this same experience with every database technology we use - including MySQL and Redis. The day after, when we've cleared our heads... we're happy again. The same as the other technologies. I think that with all technologies you're going to get bitten by some detail you didn't know about or had forgotten. The trick is to mitigate these disasters by thinking about your failure cases.

agscala 14y ago
I wish you would write about some of these issues that you faced so we all can keep them in mind when using MongoDB in the future.
- colinhowe 14y ago
  They're not particularly exciting. Example: we accidentally left detailed logging enabled on a secondary server and weren't monitoring the space left on the drive for logs. The disk got full and the secondary failed. Annoying that it failed but it was our fault for not monitoring that space and also leaving detailed logging on!
  - agscala 14y ago
    Well of course it's not exciting, but that's exactly the kind of examples I like hearing about because now I know I should always have detailed logging off. Anyways, I'm sure a lot of us would appreciate it if you wrote about your experiences, regardless of how mundane they are.
    - colinhowe 14y ago
      Sure! Will try to do it more often :) I don't often want to write about these things as they are silly mistakes... but, if it's helpful, I'll share!
    - krisoft 14y ago
      You mean, detailed logging off? Or I'm confused?
    - mpd 14y ago
      It looks like you want to leave detailed logging off...
      - agscala 14y ago
        Haha whoops. Edited my previous comment.
  - agilord 14y ago
    +1 to share your experience. Kiip and other companies do that, and if you need to decide on a certain technology for your upcoming development, these are really exciting and far-from-boring posts.
steve8918 14y ago
Could you elaborate on the things that MongoDB brings to the table versus a database like MySQL? If you assume that the entire dataset fits in memory for both MongoDB and MySQL, as well as SSD backing for both, what particular advantages does MongoDB have for you in production?
- gregwebs 14y ago
  its not the entire set (use Redis then), it is the working set: data being accessed.
13rules 14y ago
With 400GB of data I'd imagine you are going to be ripping your hair out at some point or another regardless of what database you use ... there is no way to completely avoid tech issues over the lifetime of a product/service.
- Devilboy 14y ago
  At least with a serious RDBMS you'll have loads of awesome tools and solid experts around to help you solve your scaling problems.
23david 14y ago
At our company we have a much bigger MongoDB dataset in production. I can relate well to your comments, since it is the same stuff that I hear from our developers as they defend Mongo after yet another horrific situation. They have a tough time admitting that Mongo is the problem, and instead always find ways to say that it was their fault. This is such a strange thing for me to observe, but I think that it has to do with the fact that MongoDB works so smoothly initially with a default install that it hooks you in, and then later when you have a large dataset and it stops working well, it's hard to understand how what was such an amazing technology can now fail so badly. It's also a huge problem that once you run MongoDB at scale, you desperately need experts to fix things, but it's so hard to find any so-called experts who can help. 10gen did a great job of marketing to developers, but unfortunately they seemed to spit in the face of DBAs and Ops people a long time ago by proclaiming them to be unnecessary and archaic. Running any significant MongoDB database in production requires as much expertise as someone running a big MySQL instance, but there isn't a community of database lovers around MongoDB who you can hire. The DBAs and operations people I know dislike or even hate MongoDB, often for valid reasons that 10gen should address such as the as-yet-unfixed 'write lock' issue, code instability and inaccurate/misleading documentation. 10gen has done a great job at evangelizing to developers and making features that developers love. Now that 10gen has so much money, I hope that they can now afford to start making MongoDB a database that Ops people and DBAs can love. As a final note, this ServerDensity guy is clearly looking at the world through mongo-colored glasses. It makes sense I suppose since he is all-in with MongoDB for his company. But we were using his service up until three months ago, and had a lot of problems with intermittent performance issues that seemed to be database-related since the site was still working and only certain pages would take 30 seconds or so to load. It's possible that the problems were temporary and no longer exist, but it brings home my point. If MongoDB experts can't make their own services 100% reliable, what hope does a regular startup have of getting MongoDB to work well at scale. A developer may feel comfortable making the decision to go with MongoDB, but if they are wrong it won't cost them their job and they won't need to pull their hair out dealing with Ops issues all the time. If a DBA or Ops guy is being hired to manage a company's datastores, I don't see MongoDB (even 2.2) being a contender. At this point there are simply other DBs available that can perform the same or nearly the same without all the fussiness. Developers may be unhappy since nothing yet is as easy to develop on, but they'll be happier in the end when stuff 'just works'.

agscala 14y ago

I wish you would write about some of these issues that you faced so we all can keep them in mind when using MongoDB in the future.

colinhowe 14y ago
They're not particularly exciting. Example: we accidentally left detailed logging enabled on a secondary server and weren't monitoring the space left on the drive for logs. The disk got full and the secondary failed. Annoying that it failed but it was our fault for not monitoring that space and also leaving detailed logging on!
- agscala 14y ago
  Well of course it's not exciting, but that's exactly the kind of examples I like hearing about because now I know I should always have detailed logging off. Anyways, I'm sure a lot of us would appreciate it if you wrote about your experiences, regardless of how mundane they are.
  - colinhowe 14y ago
    Sure! Will try to do it more often :) I don't often want to write about these things as they are silly mistakes... but, if it's helpful, I'll share!
  - krisoft 14y ago
    You mean, detailed logging off? Or I'm confused?
  - mpd 14y ago
    It looks like you want to leave detailed logging off...
    - agscala 14y ago
      Haha whoops. Edited my previous comment.
- agilord 14y ago
  +1 to share your experience. Kiip and other companies do that, and if you need to decide on a certain technology for your upcoming development, these are really exciting and far-from-boring posts.

steve8918 14y ago

Could you elaborate on the things that MongoDB brings to the table versus a database like MySQL? If you assume that the entire dataset fits in memory for both MongoDB and MySQL, as well as SSD backing for both, what particular advantages does MongoDB have for you in production?

gregwebs 14y ago
its not the entire set (use Redis then), it is the working set: data being accessed.

13rules 14y ago

With 400GB of data I'd imagine you are going to be ripping your hair out at some point or another regardless of what database you use ... there is no way to completely avoid tech issues over the lifetime of a product/service.

Devilboy 14y ago
At least with a serious RDBMS you'll have loads of awesome tools and solid experts around to help you solve your scaling problems.

23david 14y ago

At our company we have a much bigger MongoDB dataset in production. I can relate well to your comments, since it is the same stuff that I hear from our developers as they defend Mongo after yet another horrific situation. They have a tough time admitting that Mongo is the problem, and instead always find ways to say that it was their fault. This is such a strange thing for me to observe, but I think that it has to do with the fact that MongoDB works so smoothly initially with a default install that it hooks you in, and then later when you have a large dataset and it stops working well, it's hard to understand how what was such an amazing technology can now fail so badly. It's also a huge problem that once you run MongoDB at scale, you desperately need experts to fix things, but it's so hard to find any so-called experts who can help. 10gen did a great job of marketing to developers, but unfortunately they seemed to spit in the face of DBAs and Ops people a long time ago by proclaiming them to be unnecessary and archaic. Running any significant MongoDB database in production requires as much expertise as someone running a big MySQL instance, but there isn't a community of database lovers around MongoDB who you can hire. The DBAs and operations people I know dislike or even hate MongoDB, often for valid reasons that 10gen should address such as the as-yet-unfixed 'write lock' issue, code instability and inaccurate/misleading documentation. 10gen has done a great job at evangelizing to developers and making features that developers love. Now that 10gen has so much money, I hope that they can now afford to start making MongoDB a database that Ops people and DBAs can love. As a final note, this ServerDensity guy is clearly looking at the world through mongo-colored glasses. It makes sense I suppose since he is all-in with MongoDB for his company. But we were using his service up until three months ago, and had a lot of problems with intermittent performance issues that seemed to be database-related since the site was still working and only certain pages would take 30 seconds or so to load. It's possible that the problems were temporary and no longer exist, but it brings home my point. If MongoDB experts can't make their own services 100% reliable, what hope does a regular startup have of getting MongoDB to work well at scale. A developer may feel comfortable making the decision to go with MongoDB, but if they are wrong it won't cost them their job and they won't need to pull their hair out dealing with Ops issues all the time. If a DBA or Ops guy is being hired to manage a company's datastores, I don't see MongoDB (even 2.2) being a contender. At this point there are simply other DBs available that can perform the same or nearly the same without all the fussiness. Developers may be unhappy since nothing yet is as easy to develop on, but they'll be happier in the end when stuff 'just works'.