18 ms·

What 10gen nailed with MongoDB

109 points by calvinfo 14y ago

bri3d 14y ago
This article is spot-on, but 10gen did one more thing they probably shouldn't have: marketed their not-very-scalable, not-very-durable database as a solution to problems companies were having with scalability and durability.
- bluestix 14y ago
  Marketing geniuses for sure. Success comes with billboards on the 101 in Redwood City.
- whalesalad 14y ago
  I'm very curious about this. I feel like the community at large seems to agree mongo is a dangerous choice. But then I read articles like this. At what scale do you replace Mongo? Should it only be used for bootstrapping something? Is it something that 80% of apps will be fine on? I'm in an interesting spot myself right now. I've been hacking on Django since 0.96. I love the ORM, and hate everything else. Python on the other hand is a wonderful language. Rails... I've been spending 8 hours a day for the past month building a rails project. I love a lot of things about rails, hate the ORM, and dislike a lot of the magic. On the flip side: I've been a Javascript/front-end developer since before jQuery existed, and I love coffeescript. I understand callback-style evented programming and therefore node.js apps make perfect sense to me. My co-founder and I have just finished some heavy duty design/wireframe/conceptual work on a new project of ours. It's time to build the API. I am COMPLETELY torn with what framework/language/database to use for this project. At first I thought mongodb and coffeescript+node would be fun and reliable. But the community seems to think mongo is unreliable and almost dangerous. So I thought, hmm, Riak looks like it will let me throw whatever data I want into it (like mongo) but it's rock solid. I later dismissed that since I figured I'd need to do a lot of heavy lifting on my own. Rails keeps poking me in the back of my mind but I honest-to-god hate ActiveRecord and am afraid to use DataMapper for fear it won't play nice with a lot of the popular ruby goodies out there. I've modeled most of the project now in Django and i'm starting to play with Tastypie for the REST component. It feels too kludgy. I'm a wreck. Mommy?
  - ihsw 14y ago
    One of the main sticking point for serious mongodb infrastructure is the global lock -- yes, write operations cause the entire mongodb instance (every database, every read operation) to block. This is clearly one of the main issues surrounding high-concurrent mongodb development. Thankfully with mongodb-2.2 there is now instead a database lock which mitigates the issue. Personally I feel redis more closely resembles language primitives with simple key-value pairs, hashes, lists (ordered, repeating values allowed), sets (un-ordered, no repeating values), and sorted sets (more interesting).
    - taligent 14y ago
      Redis has no clustering (yet) so is best used when your data is transient and you are able to repopulate.
      - j-kidd 14y ago
        I have no experience with Redis or MongoDB, but your statement is only true if the storage layer is transient. Some of us use servers that don't go away upon shutdown, backed by a SAN.
        mh- 14y ago
        Have you tested how long it would take your redis instance to be usable again after it finishes replaying the commitlog (AOF)?
        deleted 14y ago
        [deleted]
  - WALoeIII 14y ago
    You are thinking too hard. Just pick one, you can always change later. All of the options you have presented are "Very Good" - they do so many things right. To steal a pithy quote from my co-worker: "Its a balloon squeeze." This decision will not determine the success or failure of your business, move along. I pick Rails & Ruby.
- 23david 14y ago
  agreed. however, in the article it praises the mongodb defaults. but time and time again we've been bitten by open files issues that are easily solved by shipping better init.d scripts. I also feel that scaling out mongodb clusters is unnecessarily complicated.
ukd1 14y ago
Nailing community and ease of use (drivers for everything, easy to install) and documentation wouldn't have been much use if the product sucked; it doesn't. A lot of people that I've seen with issues around Mongo haven't understood how it's expected to be used: i.e. it's not a relational database and the schema design is therefore different.
- slurgfest 14y ago
  This might be because a lot of people have no idea what the viable use cases are for all the million different new database-things there are. Because it just isn't that clear unless you have used them all or are a total nerd for the design of database software. In this respect MongoDB is partly a victim of its own success - that means more people using it who aren't sure what it's for, or who are led to believe it's for something it's not really so hot at.
trimbo 14y ago
> Think about a web developer who shows up to a hackathon [...] He just wants a quick way to persist and retrieve data. This is the _worst_ reason to use Mongo. If you are doing a hackathon or just prototyping and need persistence, use pickle. Or store JSON to a text file. Who cares what the solution is. It should: a) Be built into the standard library. b) Not require bringing up another service to work. [Edit.. clarity in the quote]
- nestlequ1k 14y ago
  That's the point. It's actually easier to use MongoDB than to store JSON in a text file. And things move faster, and you still have the dream that you can scale it up (even if 99% of the time it doesnt happen).
TylerBrock 14y ago
To me, the benefits of MongoDB have always been that it makes the two biggest challengers developers are faced with when creating an application easy: Building and Scaling. Being able to efficiently build and scale an app only makes the end product better.
revoltingx 14y ago
Hours to setup a SQL DB? If it takes you hours, you're probably not a good developer.
ndemoor 14y ago
Although I am a heavy user/believer of MongoDB, one caveat that is overlooked in this article is the administration part of Mongo Clusters. Even the simplest replication needs 3 servers, add sharding to the dance for extra performance and the server counter jumps up. For startups this is a major decision to consider as a full-time ops guy isn't always affordable. Luckily, PaaS services as MongoLabs and MongoHQ save the day.
- taligent 14y ago
  I don't understand your point. You could use replication (master/slave) with 2 servers and you don't have to use sharding for anything. How is MongoDB not the same as every other database ? If anything MongoDB is by far the easiest database I've ever used for setting up clustering/sharding.
  - ndemoor 14y ago
    To have a legitimate replication setup, an arbiter node is highly recommended: http://docs.mongodb.org/manual/administration/replication-architectures/#replica-set-arbiter-nodes http://docs.mongodb.org/manual/administration/replication-ar...
  - sauravc 14y ago
    Have you tried CouchBase? The built in admin panel is a dream. Adding and removing servers to a cluster is super easy. http://www.couchbase.com/docs/couchbase-manual-2.0/couchbase-getting-started-setup.html http://www.couchbase.com/docs/couchbase-manual-2.0/couchbase...
  - ukd1 14y ago
    Master / slave is deprecated. Replica sets can run on 2 nodes however, running on an even number of nodes in a set is a very bad idea as a primary node failure will cause the rest of your nodes to go read-only. Always run an odd number. Avoid master / slave.
zacharyvoase 14y ago
> Think about a web developer who shows up to a hackathon, ready to break out his new side project. He doesn’t want to spend hours planning schema or creating databases and tables. He just wants a quick way to persist and retrieve data. When you say that, this is what I hear: > MongoDB is the quickest way to accrue large quantities of technical debt. In this analogy, fast accrual of debt leads to one place: bankruptcy. Which, in software engineering, is the ground-up rewrite.
- batgaijin 14y ago
  Uh, what? What's wrong with using MongoDB to screw around while you are developing it? It's not like you have to marry it. Figure out what schema you are going to be using, and then write your table layouts for Postgres.
  - taligent 14y ago
    Sorry but what you are suggesting is crazy. MongoDB has a completely different data model to PostgreSQL. You can't just build your app around one approach and then trivially move to another. Pick the database for the data model not the other way around.
    - zacharyvoase 14y ago
      I could not fail to disagree with you less. (^ that means I agree). This idea that you can arbitrarily switch between database X and database Y with radically different models is a really harmful fallacy. Code which attempts to ‘abstract away the details’ is often leaky, buggy, complex, and just as prone to tight coupling as any other solution. At the very least, such has been my experience. And I think certain individuals have a policy of downvoting those comments they disagree with rather than explaining their counterpoints in a comment.
      - Morg 14y ago
        But no, it's actually worse my dear. For example, let's say you start with mysql. You have no features whatsoever, implement your own workarounds or simplified model, create your own text search because there is none (except in Myisam but then that's totally fucked up), and then you migrate to postgres, where every feature is available and working as advertised, you still have your workarounds and text search to remove, and your data model is all wrong because it was made on a pitiful dbms. MongoDB is the same but worse, it has even less features than MySQL and is not even particularly fast so ... Those who believe it's slow to design a database are just not DBAs and should leave it to the pros. I write data models for full applications in SQL directly without any document or plan or layout, and it doesn't take long. But then, I know why mysql is a piece of crap, I have truly kick ass instead of triggers on views on information_schema that maintain historifying triggers on any table whatsoever, so not exactly a first-timer. Really, those people are only afraid of learning (the real powerful tools), and will stick to the shittiest ones provided they're just a bit easier to learn. When google does not use an SQL database, it's because for the very simple task at hand, a full-fledged DBMS is overkill. Because what they do is indexed search. not run a database. thinking you can replace a database with indexed search is going back to the dark ages before databases were invented.
runT1ME 14y ago
MongoDB does pretty much everything the opposite of Oracle: Easy to setup, use and program. However, Oracle sure is raking in the dough, so is Mongo succeeding simply because they're fulfilling a niche left by Oracle, or because it's the 'right way' to build a software product company? I hope the answer is the latter, and that Oracle's billions are simply the result of the being entrenched after years and years of doing it the 'old way. That being said, I think it's a little too early to be championing Mongo's business model (even though I'm rooting for it).
zzzeek 14y ago
> For 90% of web development cases, simply storing and retrieving objects from a persistent store is enough of an API. "90%", that's quite a specific number. Most of what I've written for the past fifteen years is web apps, and I'd say about 10% fall into the "all I need is put and get objects" category. Especially once I knew how to use SQL effectively, especially once I knew how to work with ACID, these features became irreplaceable in almost every project I've worked with. IMHO it's all about what you're familiar with. Rich Hickey thinks we're morons for using OOP, as he's a brilliant functional guy. Shrugs.
- pytrin 14y ago
  Completely agree. Web development is simple, but it's not that simple. Relationships between tables, and the ability to normalize and aggregate data are very useful, even for web development.
Morg 14y ago
This article says in many words what can be said in one sentence: Make tech for low-skilled rather than high-skilled kind, there are far more people of the first kind. That's how mysql succeded, that's how PHP succeded, that's how java succeded, that's how mongoDB and node.js are succeding. You want your tech to be used widely ? engineer it to fit perfectly in the hand of unskilled individuals. This is utterly disgusting, there are less good programmers today than there were 30 years ago, and most of that is purely related to the emergence of the "tools for the average dude" that end up plaguing even those who could've learned and become great. Eat your javascript-mvc-cake . code crap . be happy .
sbt 14y ago
I was nailed by MongoDB. It's a great database for prototyping, but all projects that go anywhere run into its limitations: scale, durability, intimate relationship with OS, ease of administration. MongoDB is right for some projects, but in many cases is not. So if you know that you can transition from it at a later point, it gets you off to a flying start. If you foresee that it will be hard to switch later, I would advice spending some time considering future scenarios upfront.
Timshel 14y ago
Interestically someone just released a new async scala drivers : http://news.ycombinator.com/item?id=4454077 http://news.ycombinator.com/item?id=4454077

109 points by calvinfo 14y ago

bri3d 14y ago

This article is spot-on, but 10gen did one more thing they probably shouldn't have: marketed their not-very-scalable, not-very-durable database as a solution to problems companies were having with scalability and durability.

bluestix 14y ago
Marketing geniuses for sure. Success comes with billboards on the 101 in Redwood City.
whalesalad 14y ago
I'm very curious about this. I feel like the community at large seems to agree mongo is a dangerous choice. But then I read articles like this. At what scale do you replace Mongo? Should it only be used for bootstrapping something? Is it something that 80% of apps will be fine on? I'm in an interesting spot myself right now. I've been hacking on Django since 0.96. I love the ORM, and hate everything else. Python on the other hand is a wonderful language. Rails... I've been spending 8 hours a day for the past month building a rails project. I love a lot of things about rails, hate the ORM, and dislike a lot of the magic. On the flip side: I've been a Javascript/front-end developer since before jQuery existed, and I love coffeescript. I understand callback-style evented programming and therefore node.js apps make perfect sense to me. My co-founder and I have just finished some heavy duty design/wireframe/conceptual work on a new project of ours. It's time to build the API. I am COMPLETELY torn with what framework/language/database to use for this project. At first I thought mongodb and coffeescript+node would be fun and reliable. But the community seems to think mongo is unreliable and almost dangerous. So I thought, hmm, Riak looks like it will let me throw whatever data I want into it (like mongo) but it's rock solid. I later dismissed that since I figured I'd need to do a lot of heavy lifting on my own. Rails keeps poking me in the back of my mind but I honest-to-god hate ActiveRecord and am afraid to use DataMapper for fear it won't play nice with a lot of the popular ruby goodies out there. I've modeled most of the project now in Django and i'm starting to play with Tastypie for the REST component. It feels too kludgy. I'm a wreck. Mommy?
- ihsw 14y ago
  One of the main sticking point for serious mongodb infrastructure is the global lock -- yes, write operations cause the entire mongodb instance (every database, every read operation) to block. This is clearly one of the main issues surrounding high-concurrent mongodb development. Thankfully with mongodb-2.2 there is now instead a database lock which mitigates the issue. Personally I feel redis more closely resembles language primitives with simple key-value pairs, hashes, lists (ordered, repeating values allowed), sets (un-ordered, no repeating values), and sorted sets (more interesting).
  - taligent 14y ago
    Redis has no clustering (yet) so is best used when your data is transient and you are able to repopulate.
    - j-kidd 14y ago
      I have no experience with Redis or MongoDB, but your statement is only true if the storage layer is transient. Some of us use servers that don't go away upon shutdown, backed by a SAN.
      - mh- 14y ago
        Have you tested how long it would take your redis instance to be usable again after it finishes replaying the commitlog (AOF)?
      - deleted 14y ago
        [deleted]
- WALoeIII 14y ago
  You are thinking too hard. Just pick one, you can always change later. All of the options you have presented are "Very Good" - they do so many things right. To steal a pithy quote from my co-worker: "Its a balloon squeeze." This decision will not determine the success or failure of your business, move along. I pick Rails & Ruby.
23david 14y ago
agreed. however, in the article it praises the mongodb defaults. but time and time again we've been bitten by open files issues that are easily solved by shipping better init.d scripts. I also feel that scaling out mongodb clusters is unnecessarily complicated.

ukd1 14y ago

Nailing community and ease of use (drivers for everything, easy to install) and documentation wouldn't have been much use if the product sucked; it doesn't. A lot of people that I've seen with issues around Mongo haven't understood how it's expected to be used: i.e. it's not a relational database and the schema design is therefore different.

slurgfest 14y ago
This might be because a lot of people have no idea what the viable use cases are for all the million different new database-things there are. Because it just isn't that clear unless you have used them all or are a total nerd for the design of database software. In this respect MongoDB is partly a victim of its own success - that means more people using it who aren't sure what it's for, or who are led to believe it's for something it's not really so hot at.

trimbo 14y ago

> Think about a web developer who shows up to a hackathon [...] He just wants a quick way to persist and retrieve data. This is the _worst_ reason to use Mongo. If you are doing a hackathon or just prototyping and need persistence, use pickle. Or store JSON to a text file. Who cares what the solution is. It should: a) Be built into the standard library. b) Not require bringing up another service to work. [Edit.. clarity in the quote]

nestlequ1k 14y ago
That's the point. It's actually easier to use MongoDB than to store JSON in a text file. And things move faster, and you still have the dream that you can scale it up (even if 99% of the time it doesnt happen).

TylerBrock 14y ago

To me, the benefits of MongoDB have always been that it makes the two biggest challengers developers are faced with when creating an application easy: Building and Scaling. Being able to efficiently build and scale an app only makes the end product better.

revoltingx 14y ago

Hours to setup a SQL DB? If it takes you hours, you're probably not a good developer.

ndemoor 14y ago

Although I am a heavy user/believer of MongoDB, one caveat that is overlooked in this article is the administration part of Mongo Clusters. Even the simplest replication needs 3 servers, add sharding to the dance for extra performance and the server counter jumps up. For startups this is a major decision to consider as a full-time ops guy isn't always affordable. Luckily, PaaS services as MongoLabs and MongoHQ save the day.

taligent 14y ago
I don't understand your point. You could use replication (master/slave) with 2 servers and you don't have to use sharding for anything. How is MongoDB not the same as every other database ? If anything MongoDB is by far the easiest database I've ever used for setting up clustering/sharding.
- ndemoor 14y ago
  To have a legitimate replication setup, an arbiter node is highly recommended: http://docs.mongodb.org/manual/administration/replication-architectures/#replica-set-arbiter-nodes http://docs.mongodb.org/manual/administration/replication-ar...
- sauravc 14y ago
  Have you tried CouchBase? The built in admin panel is a dream. Adding and removing servers to a cluster is super easy. http://www.couchbase.com/docs/couchbase-manual-2.0/couchbase-getting-started-setup.html http://www.couchbase.com/docs/couchbase-manual-2.0/couchbase...
- ukd1 14y ago
  Master / slave is deprecated. Replica sets can run on 2 nodes however, running on an even number of nodes in a set is a very bad idea as a primary node failure will cause the rest of your nodes to go read-only. Always run an odd number. Avoid master / slave.

zacharyvoase 14y ago

> Think about a web developer who shows up to a hackathon, ready to break out his new side project. He doesn’t want to spend hours planning schema or creating databases and tables. He just wants a quick way to persist and retrieve data. When you say that, this is what I hear: > MongoDB is the quickest way to accrue large quantities of technical debt. In this analogy, fast accrual of debt leads to one place: bankruptcy. Which, in software engineering, is the ground-up rewrite.

batgaijin 14y ago
Uh, what? What's wrong with using MongoDB to screw around while you are developing it? It's not like you have to marry it. Figure out what schema you are going to be using, and then write your table layouts for Postgres.
- taligent 14y ago
  Sorry but what you are suggesting is crazy. MongoDB has a completely different data model to PostgreSQL. You can't just build your app around one approach and then trivially move to another. Pick the database for the data model not the other way around.
  - zacharyvoase 14y ago
    I could not fail to disagree with you less. (^ that means I agree). This idea that you can arbitrarily switch between database X and database Y with radically different models is a really harmful fallacy. Code which attempts to ‘abstract away the details’ is often leaky, buggy, complex, and just as prone to tight coupling as any other solution. At the very least, such has been my experience. And I think certain individuals have a policy of downvoting those comments they disagree with rather than explaining their counterpoints in a comment.
    - Morg 14y ago
      But no, it's actually worse my dear. For example, let's say you start with mysql. You have no features whatsoever, implement your own workarounds or simplified model, create your own text search because there is none (except in Myisam but then that's totally fucked up), and then you migrate to postgres, where every feature is available and working as advertised, you still have your workarounds and text search to remove, and your data model is all wrong because it was made on a pitiful dbms. MongoDB is the same but worse, it has even less features than MySQL and is not even particularly fast so ... Those who believe it's slow to design a database are just not DBAs and should leave it to the pros. I write data models for full applications in SQL directly without any document or plan or layout, and it doesn't take long. But then, I know why mysql is a piece of crap, I have truly kick ass instead of triggers on views on information_schema that maintain historifying triggers on any table whatsoever, so not exactly a first-timer. Really, those people are only afraid of learning (the real powerful tools), and will stick to the shittiest ones provided they're just a bit easier to learn. When google does not use an SQL database, it's because for the very simple task at hand, a full-fledged DBMS is overkill. Because what they do is indexed search. not run a database. thinking you can replace a database with indexed search is going back to the dark ages before databases were invented.

runT1ME 14y ago

MongoDB does pretty much everything the opposite of Oracle: Easy to setup, use and program. However, Oracle sure is raking in the dough, so is Mongo succeeding simply because they're fulfilling a niche left by Oracle, or because it's the 'right way' to build a software product company? I hope the answer is the latter, and that Oracle's billions are simply the result of the being entrenched after years and years of doing it the 'old way. That being said, I think it's a little too early to be championing Mongo's business model (even though I'm rooting for it).

zzzeek 14y ago

> For 90% of web development cases, simply storing and retrieving objects from a persistent store is enough of an API. "90%", that's quite a specific number. Most of what I've written for the past fifteen years is web apps, and I'd say about 10% fall into the "all I need is put and get objects" category. Especially once I knew how to use SQL effectively, especially once I knew how to work with ACID, these features became irreplaceable in almost every project I've worked with. IMHO it's all about what you're familiar with. Rich Hickey thinks we're morons for using OOP, as he's a brilliant functional guy. Shrugs.

pytrin 14y ago
Completely agree. Web development is simple, but it's not that simple. Relationships between tables, and the ability to normalize and aggregate data are very useful, even for web development.

Morg 14y ago

This article says in many words what can be said in one sentence: Make tech for low-skilled rather than high-skilled kind, there are far more people of the first kind. That's how mysql succeded, that's how PHP succeded, that's how java succeded, that's how mongoDB and node.js are succeding. You want your tech to be used widely ? engineer it to fit perfectly in the hand of unskilled individuals. This is utterly disgusting, there are less good programmers today than there were 30 years ago, and most of that is purely related to the emergence of the "tools for the average dude" that end up plaguing even those who could've learned and become great. Eat your javascript-mvc-cake . code crap . be happy .

sbt 14y ago

I was nailed by MongoDB. It's a great database for prototyping, but all projects that go anywhere run into its limitations: scale, durability, intimate relationship with OS, ease of administration. MongoDB is right for some projects, but in many cases is not. So if you know that you can transition from it at a later point, it gets you off to a flying start. If you foresee that it will be hard to switch later, I would advice spending some time considering future scenarios upfront.

Timshel 14y ago

Interestically someone just released a new async scala drivers : http://news.ycombinator.com/item?id=4454077 http://news.ycombinator.com/item?id=4454077