5 ms·

MongoDB Developer 10gen Raises $3.4 Million

48 points by yan 17y ago

stingraycharles 17y ago
MongoDB really is an interesting project. It really does fill a gap between key/value stores and relational databases. At my company, we've been using MongoDB for about half a year: it was the only project that provided auto-sharding, replication, stored procedures, commercial support, and, last but not least, the ability to detect race conditions when multiple clients are writing the same data. Pretty impressive, if you ask me. MongoDB is going to be big.
- jcapote 17y ago
  I would love to see a blog post about your usage of MongoDB
  - stingraycharles 17y ago
    Well, I don't have a blog, and I don't want to get into too much detail, but I can give a quick overview of our main usage. Basically, the core problem we needed to solve was to have a reliable datastore which is able to detect race conditions when multiple webservers are handling the same data (in essense, multiple webservers handling a request from the same visitor). To do this, we use a simple compare-and-swap trick: each object has a version number which is transmitted when an object is read, and this version number is transmitted back when the object is updated: only when the "old" version numbers match, an update is performed, otherwise an error is thrown, and the webserver re-processes the entire request from the start. I believe a similar trick is used in memcached for CAS operations, and it works great in situations where the chance of a race condition occurring is relatively small. Now, for all this to work, we need to have a datastore that lets us atomically compare and swap these objects: this is where, for us, projects based on Amazon Dynamo start to become suboptimal. I believe it's theoretically possible to detect race conditions with such projects if you set the amount of writers to more than half the amount of replicas, but it just starts to give you a feeling you're using it for an unintended purpose. MongoDB, on the other hand, provides the ability to define stored procedures. Furthermore, they guarantee that this happends atomically: only one client can update a single object at the same time, which is a great guarantee. Because of this, we could write a trivial stored procedure that takes 2 arguments (the old version number and the replacement object), and does the compare-and-swapping. I even believe MongoDB can do "update if these fields still match these values" operations nowadays, so it looks like this functionality is even integrated within MongoDB. Of course, in addition to this, we also need the datastore to be highly available and limit the risk of data loss as much as possible: MongoDB has this area covered really well, and automatic failover is implemented within the client drivers. And this is basically all there's to it. Because of the guarantees MongoDB provides, the work we needed to do was reduced to a minimum. Of course, features like ad-hoc querying and their plans to implement map/reduce capabilities help too (we've already invested a lot of time into Hadoop and Map/Reduce applications, so it all fits). All in all, it seems MongoDB just came into existence at exactly the right time for us, and the people developing it seem like a very active, passionate bunch of people. I haven't got regret choosing them.
    - moe 17y ago
      we also need the datastore to be highly available and limit the risk of data loss as much as possible: MongoDB has this area covered really well Ahem, don't want to rain on anyone's parade but I'd be very careful with such claims (which the mongo guys don't make!). The risk of data-loss is pretty high with mongo (when compared to a traditional RDBMS) as there are no defined sync points or a WAL - it only syncs to disk on shutdown. That means when your machine crashes you're almost guaranteed to lose the last n changes where n can be a very large figure depending on your usage. In reality that means you can only rely on the contents of your last mongodump to be truly persisted. Everything else is to be considered volatile.
      - DEinspanjer 17y ago
        I'd also question the validity of claiming limited risk of data loss when using sharding considering it is still considered alpha although I know for a fact that 10gen is kicking butt to try to make it certified production worthy ASAP.
        moe 17y ago
        Yes, I didn't mean to narrow their achievement. MongoDB is shaping up amazingly and I can see it becoming the MySQL of the k/v world within a year. Just as of today I'd still refrain from using it for mission critical stuff. Let it collect a few more battle scars before betting your money on it.
      - stingraycharles 17y ago
        You are correct: a single server indeed has a high risk of losing data for the reasons you describe. I was talking about replication: by making sure all data is replicated, for example, 4 times across 2 datacenters, you limit that risk a lot. There is a small timespan between writing data and all replicas picking it up, but if you make sure your servers aren't overloaded, this usually is very small. Having said that, I agree that there is room for improvement, for example using WAL and sync points like you describe.
  - zefhous 17y ago
    There's a good video about it here: http://vimeo.com/7326668 http://vimeo.com/7326668
- jherdman 17y ago
  I totally agree. I think the big missing piece of the puzzle, though, is full-text searching. I attended a webinar on MongoDB the other day hosted by Dwight Merriman (10gen) and Ian White (Business Insider). Ian mentioned this as a sore spot for his usage of MongoDB and that he'd be coming up with a Lucene solution. It'd be interesting to see the solution he comes up with.
  - _pius 17y ago
    I think the big missing piece of the puzzle, though, is full-text searching. True, but you can emulate it somewhat by indexing on all keywords of the text. MongoDB can index on keys deeply nested in the structure, so you could have a field that's an array of each word in the text and then index on those.
    - jherdman 17y ago
      Indeed, and that's how Business Insider is currently handling search. I'm not sure if this is necessarily a "good" solution though. Thoughts?
      - _pius 17y ago
        Good question, but I don't know enough about this problem to say. It'd be interesting to see some benchmarks comparing a search of a document indexed this way versus, say, a Lucene full-text search.
bravura 17y ago
I like MongoDB because it is very natural to think about databases as storing JSON-style objects. It's also useful for simple text processing pipelines. You iterate over your database and apply the next step of preprocessing, assuming it doesn't exist in the current object (e.g. if no "POS tagged text" field exists, transform "text" into "POS tagged text"). I use MongoDB because it is C++, so it was simple to set up. CouchDB is based upon a Java stack, which I am less familiar at setting up. TokyoDB has the "tables" DB type, which I believe extends the value in "key-value" to more complicated schema-less types, but the Python support for this DB type isn't very good.
- sadiq 17y ago
  Is CouchDB not based on Erlang?
  - jherdman 17y ago
    It is.
  - dminor 17y ago
    Maybe he's thinking of the Lucene, which CouchDB uses.
simonw 17y ago
Can anyone explain the implications of the AGPL license used by MongoDB? I'm pretty confused by it.
- stingraycharles 17y ago
  There's more information here: http://www.mongodb.org/display/DOCS/Licensing http://www.mongodb.org/display/DOCS/Licensing The way I understand it, is that they use AGPL to prevent people from running their own customized mongodb daemons without publishing the patches. This solves the "problem" of web services customizing code and not publishing patches because they're technically not distributing any code. Note that this doesn't apply to the client drivers, so you can still safely link the drivers into your code and/or modify these drivers.
alecco 17y ago
Congratulations! You truly deserve it.

48 points by yan 17y ago

MongoDB really is an interesting project. It really does fill a gap between key/value stores and relational databases. At my company, we've been using MongoDB for about half a year: it was the only project that provided auto-sharding, replication, stored procedures, commercial support, and, last but not least, the ability to detect race conditions when multiple clients are writing the same data. Pretty impressive, if you ask me. MongoDB is going to be big.

jcapote 17y ago
I would love to see a blog post about your usage of MongoDB
- stingraycharles 17y ago
  Well, I don't have a blog, and I don't want to get into too much detail, but I can give a quick overview of our main usage. Basically, the core problem we needed to solve was to have a reliable datastore which is able to detect race conditions when multiple webservers are handling the same data (in essense, multiple webservers handling a request from the same visitor). To do this, we use a simple compare-and-swap trick: each object has a version number which is transmitted when an object is read, and this version number is transmitted back when the object is updated: only when the "old" version numbers match, an update is performed, otherwise an error is thrown, and the webserver re-processes the entire request from the start. I believe a similar trick is used in memcached for CAS operations, and it works great in situations where the chance of a race condition occurring is relatively small. Now, for all this to work, we need to have a datastore that lets us atomically compare and swap these objects: this is where, for us, projects based on Amazon Dynamo start to become suboptimal. I believe it's theoretically possible to detect race conditions with such projects if you set the amount of writers to more than half the amount of replicas, but it just starts to give you a feeling you're using it for an unintended purpose. MongoDB, on the other hand, provides the ability to define stored procedures. Furthermore, they guarantee that this happends atomically: only one client can update a single object at the same time, which is a great guarantee. Because of this, we could write a trivial stored procedure that takes 2 arguments (the old version number and the replacement object), and does the compare-and-swapping. I even believe MongoDB can do "update if these fields still match these values" operations nowadays, so it looks like this functionality is even integrated within MongoDB. Of course, in addition to this, we also need the datastore to be highly available and limit the risk of data loss as much as possible: MongoDB has this area covered really well, and automatic failover is implemented within the client drivers. And this is basically all there's to it. Because of the guarantees MongoDB provides, the work we needed to do was reduced to a minimum. Of course, features like ad-hoc querying and their plans to implement map/reduce capabilities help too (we've already invested a lot of time into Hadoop and Map/Reduce applications, so it all fits). All in all, it seems MongoDB just came into existence at exactly the right time for us, and the people developing it seem like a very active, passionate bunch of people. I haven't got regret choosing them.
  - moe 17y ago
    we also need the datastore to be highly available and limit the risk of data loss as much as possible: MongoDB has this area covered really well Ahem, don't want to rain on anyone's parade but I'd be very careful with such claims (which the mongo guys don't make!). The risk of data-loss is pretty high with mongo (when compared to a traditional RDBMS) as there are no defined sync points or a WAL - it only syncs to disk on shutdown. That means when your machine crashes you're almost guaranteed to lose the last n changes where n can be a very large figure depending on your usage. In reality that means you can only rely on the contents of your last mongodump to be truly persisted. Everything else is to be considered volatile.
    - DEinspanjer 17y ago
      I'd also question the validity of claiming limited risk of data loss when using sharding considering it is still considered alpha although I know for a fact that 10gen is kicking butt to try to make it certified production worthy ASAP.
      - moe 17y ago
        Yes, I didn't mean to narrow their achievement. MongoDB is shaping up amazingly and I can see it becoming the MySQL of the k/v world within a year. Just as of today I'd still refrain from using it for mission critical stuff. Let it collect a few more battle scars before betting your money on it.
    - stingraycharles 17y ago
      You are correct: a single server indeed has a high risk of losing data for the reasons you describe. I was talking about replication: by making sure all data is replicated, for example, 4 times across 2 datacenters, you limit that risk a lot. There is a small timespan between writing data and all replicas picking it up, but if you make sure your servers aren't overloaded, this usually is very small. Having said that, I agree that there is room for improvement, for example using WAL and sync points like you describe.
- zefhous 17y ago
  There's a good video about it here: http://vimeo.com/7326668 http://vimeo.com/7326668
jherdman 17y ago
I totally agree. I think the big missing piece of the puzzle, though, is full-text searching. I attended a webinar on MongoDB the other day hosted by Dwight Merriman (10gen) and Ian White (Business Insider). Ian mentioned this as a sore spot for his usage of MongoDB and that he'd be coming up with a Lucene solution. It'd be interesting to see the solution he comes up with.
- _pius 17y ago
  I think the big missing piece of the puzzle, though, is full-text searching. True, but you can emulate it somewhat by indexing on all keywords of the text. MongoDB can index on keys deeply nested in the structure, so you could have a field that's an array of each word in the text and then index on those.
  - jherdman 17y ago
    Indeed, and that's how Business Insider is currently handling search. I'm not sure if this is necessarily a "good" solution though. Thoughts?
    - _pius 17y ago
      Good question, but I don't know enough about this problem to say. It'd be interesting to see some benchmarks comparing a search of a document indexed this way versus, say, a Lucene full-text search.

bravura 17y ago

I like MongoDB because it is very natural to think about databases as storing JSON-style objects. It's also useful for simple text processing pipelines. You iterate over your database and apply the next step of preprocessing, assuming it doesn't exist in the current object (e.g. if no "POS tagged text" field exists, transform "text" into "POS tagged text"). I use MongoDB because it is C++, so it was simple to set up. CouchDB is based upon a Java stack, which I am less familiar at setting up. TokyoDB has the "tables" DB type, which I believe extends the value in "key-value" to more complicated schema-less types, but the Python support for this DB type isn't very good.

sadiq 17y ago
Is CouchDB not based on Erlang?
- jherdman 17y ago
  It is.
- dminor 17y ago
  Maybe he's thinking of the Lucene, which CouchDB uses.

simonw 17y ago

Can anyone explain the implications of the AGPL license used by MongoDB? I'm pretty confused by it.

stingraycharles 17y ago
There's more information here: http://www.mongodb.org/display/DOCS/Licensing http://www.mongodb.org/display/DOCS/Licensing The way I understand it, is that they use AGPL to prevent people from running their own customized mongodb daemons without publishing the patches. This solves the "problem" of web services customizing code and not publishing patches because they're technically not distributing any code. Note that this doesn't apply to the client drivers, so you can still safely link the drivers into your code and/or modify these drivers.

alecco 17y ago

Congratulations! You truly deserve it.