4 ms·

I would love to see a blog post about your usage of MongoDB

by jcapote 17y ago

stingraycharles 17y ago
Well, I don't have a blog, and I don't want to get into too much detail, but I can give a quick overview of our main usage. Basically, the core problem we needed to solve was to have a reliable datastore which is able to detect race conditions when multiple webservers are handling the same data (in essense, multiple webservers handling a request from the same visitor). To do this, we use a simple compare-and-swap trick: each object has a version number which is transmitted when an object is read, and this version number is transmitted back when the object is updated: only when the "old" version numbers match, an update is performed, otherwise an error is thrown, and the webserver re-processes the entire request from the start. I believe a similar trick is used in memcached for CAS operations, and it works great in situations where the chance of a race condition occurring is relatively small. Now, for all this to work, we need to have a datastore that lets us atomically compare and swap these objects: this is where, for us, projects based on Amazon Dynamo start to become suboptimal. I believe it's theoretically possible to detect race conditions with such projects if you set the amount of writers to more than half the amount of replicas, but it just starts to give you a feeling you're using it for an unintended purpose. MongoDB, on the other hand, provides the ability to define stored procedures. Furthermore, they guarantee that this happends atomically: only one client can update a single object at the same time, which is a great guarantee. Because of this, we could write a trivial stored procedure that takes 2 arguments (the old version number and the replacement object), and does the compare-and-swapping. I even believe MongoDB can do "update if these fields still match these values" operations nowadays, so it looks like this functionality is even integrated within MongoDB. Of course, in addition to this, we also need the datastore to be highly available and limit the risk of data loss as much as possible: MongoDB has this area covered really well, and automatic failover is implemented within the client drivers. And this is basically all there's to it. Because of the guarantees MongoDB provides, the work we needed to do was reduced to a minimum. Of course, features like ad-hoc querying and their plans to implement map/reduce capabilities help too (we've already invested a lot of time into Hadoop and Map/Reduce applications, so it all fits). All in all, it seems MongoDB just came into existence at exactly the right time for us, and the people developing it seem like a very active, passionate bunch of people. I haven't got regret choosing them.
- moe 17y ago
  we also need the datastore to be highly available and limit the risk of data loss as much as possible: MongoDB has this area covered really well Ahem, don't want to rain on anyone's parade but I'd be very careful with such claims (which the mongo guys don't make!). The risk of data-loss is pretty high with mongo (when compared to a traditional RDBMS) as there are no defined sync points or a WAL - it only syncs to disk on shutdown. That means when your machine crashes you're almost guaranteed to lose the last n changes where n can be a very large figure depending on your usage. In reality that means you can only rely on the contents of your last mongodump to be truly persisted. Everything else is to be considered volatile.
  - DEinspanjer 17y ago
    I'd also question the validity of claiming limited risk of data loss when using sharding considering it is still considered alpha although I know for a fact that 10gen is kicking butt to try to make it certified production worthy ASAP.
    - moe 17y ago
      Yes, I didn't mean to narrow their achievement. MongoDB is shaping up amazingly and I can see it becoming the MySQL of the k/v world within a year. Just as of today I'd still refrain from using it for mission critical stuff. Let it collect a few more battle scars before betting your money on it.
  - stingraycharles 17y ago
    You are correct: a single server indeed has a high risk of losing data for the reasons you describe. I was talking about replication: by making sure all data is replicated, for example, 4 times across 2 datacenters, you limit that risk a lot. There is a small timespan between writing data and all replicas picking it up, but if you make sure your servers aren't overloaded, this usually is very small. Having said that, I agree that there is room for improvement, for example using WAL and sync points like you describe.
zefhous 17y ago
There's a good video about it here: http://vimeo.com/7326668 http://vimeo.com/7326668

I would love to see a blog post about your usage of MongoDB

by jcapote 17y ago

I would love to see a blog post about your usage of MongoDB

Well, I don't have a blog, and I don't want to get into too much detail, but I can give a quick overview of our main usage. Basically, the core problem we needed to solve was to have a reliable datastore which is able to detect race conditions when multiple webservers are handling the same data (in essense, multiple webservers handling a request from the same visitor). To do this, we use a simple compare-and-swap trick: each object has a version number which is transmitted when an object is read, and this version number is transmitted back when the object is updated: only when the "old" version numbers match, an update is performed, otherwise an error is thrown, and the webserver re-processes the entire request from the start. I believe a similar trick is used in memcached for CAS operations, and it works great in situations where the chance of a race condition occurring is relatively small. Now, for all this to work, we need to have a datastore that lets us atomically compare and swap these objects: this is where, for us, projects based on Amazon Dynamo start to become suboptimal. I believe it's theoretically possible to detect race conditions with such projects if you set the amount of writers to more than half the amount of replicas, but it just starts to give you a feeling you're using it for an unintended purpose. MongoDB, on the other hand, provides the ability to define stored procedures. Furthermore, they guarantee that this happends atomically: only one client can update a single object at the same time, which is a great guarantee. Because of this, we could write a trivial stored procedure that takes 2 arguments (the old version number and the replacement object), and does the compare-and-swapping. I even believe MongoDB can do "update if these fields still match these values" operations nowadays, so it looks like this functionality is even integrated within MongoDB. Of course, in addition to this, we also need the datastore to be highly available and limit the risk of data loss as much as possible: MongoDB has this area covered really well, and automatic failover is implemented within the client drivers. And this is basically all there's to it. Because of the guarantees MongoDB provides, the work we needed to do was reduced to a minimum. Of course, features like ad-hoc querying and their plans to implement map/reduce capabilities help too (we've already invested a lot of time into Hadoop and Map/Reduce applications, so it all fits). All in all, it seems MongoDB just came into existence at exactly the right time for us, and the people developing it seem like a very active, passionate bunch of people. I haven't got regret choosing them.

moe 17y ago
we also need the datastore to be highly available and limit the risk of data loss as much as possible: MongoDB has this area covered really well Ahem, don't want to rain on anyone's parade but I'd be very careful with such claims (which the mongo guys don't make!). The risk of data-loss is pretty high with mongo (when compared to a traditional RDBMS) as there are no defined sync points or a WAL - it only syncs to disk on shutdown. That means when your machine crashes you're almost guaranteed to lose the last n changes where n can be a very large figure depending on your usage. In reality that means you can only rely on the contents of your last mongodump to be truly persisted. Everything else is to be considered volatile.
- DEinspanjer 17y ago
  I'd also question the validity of claiming limited risk of data loss when using sharding considering it is still considered alpha although I know for a fact that 10gen is kicking butt to try to make it certified production worthy ASAP.
  - moe 17y ago
    Yes, I didn't mean to narrow their achievement. MongoDB is shaping up amazingly and I can see it becoming the MySQL of the k/v world within a year. Just as of today I'd still refrain from using it for mission critical stuff. Let it collect a few more battle scars before betting your money on it.
- stingraycharles 17y ago
  You are correct: a single server indeed has a high risk of losing data for the reasons you describe. I was talking about replication: by making sure all data is replicated, for example, 4 times across 2 datacenters, you limit that risk a lot. There is a small timespan between writing data and all replicas picking it up, but if you make sure your servers aren't overloaded, this usually is very small. Having said that, I agree that there is room for improvement, for example using WAL and sync points like you describe.

zefhous 17y ago

There's a good video about it here: http://vimeo.com/7326668 http://vimeo.com/7326668