6 ms·
12 Months with MongoDB
- jrosoff 16y agoGreat writeup! Couple questions: - I'm curious why querying before a write makes such a big difference. I would have guessed that updating a document that's not in RAM would first load it into RAM, then perform the update. Does the write get applied to disk without loading the page into RAM first? If you do an update to a document that is not in RAM, is it in RAM following the update? - Can you elaborate on the corruption that occurred to both the master & the slave during a DAS failure? We have seen something similar in our deployment (high write volume leading to corruption in both master & slave. required repair to recover. ran on a partially functioning slave during the repair), but were unable to identify the root cause.
- fehguy 16y agoQuerying before the writes solved a lot of problems. It gets the object in the working RAM set. When doing an update, the database gets LOCKED when the statement hits the server--that means if your document is not in memory, you have to wait while it gets looked up. This was an easy, easy win for us. Regarding the corruption, I got an "invalido BSON object" or something on repair, which tells me some object was only partially flushed to disk when the DAS went down. The slave actually worked fine for simple lookups by ID, but there was some issue with the index and I was unable to run filters against it. Luckily the huge collections are only accessed via unique identifier, so this wasn't a huge issue.
- danudey 16y agoThis seems like the sort of optimization that should be occurring in MongoDB itself - instead of acquiring the lock, loading the record into memory (if it's not already), then making the change and releasing the lock, acquire the lock after the record has been loaded into memory (if it's not already). Have you spoken with any of the MongoDB developers about why it's currently the way it is, vs. a more efficient update path?
- fehguy 16y agoI think there are some possible timing issues with making that a general behavior in the server. 10gen did make it the default behavior on slaves, where the inserts are controlled by the oplog (http://jira.mongodb.org/browse/SERVER-1646 http://jira.mongodb.org/browse/SERVER-1646). For us, our DB abstraction layer made this behavior so simple to add that we didn't make much fuss about it.
- weixiyen 16y agoWhere are the "MongoDB is Web Scale" jokes? crickets. If you are not using MongoDB, you are missing out badly and are probably developing at a much slower rate than someone who is.
- invertedlambda 16y agoMongoDB is web scale. No downvotes this time, please! http://www.xtranormal.com/watch/6995033/ http://www.xtranormal.com/watch/6995033/
- mkramlich 16y agothat's hilarious, thank you. and very much on topic!
- jasonjei 16y agoI think it's a bit short-sighted to assume everyone can use MongoDB if you're dealing with ACID type apps, or anything that deals with money. It's silly to say that they're developing at a much slower rate than someone that is. Use the right tool, or a combination of right tools, for the job.
- weixiyen 16y agoYes everyone, upvote the strawman argument.
- pierrefar 16y agoWhy is it wrong to upvote the argument that says "use the tool that suits the job"? MongoDB fits some use cases really well, and even then we're still learning how to use it (read the post about the problems they had and how they fixed them). It's basically like any other database: it's good for some things, not so good for others, and when you use, learn it like you would learn any other part of your technology stack.
- weixiyen 16y ago
- megaman821 16y agoIt is kind of odd that speed is the main motivation to switch from MySQL. Horizontal scaling is the usually given reason. From what I have seen Mongo achieves most of its speed by not using fsync by default. There were some slides floating around a while ago that showed Postgres at about the same speed by turning off fsync.
- kchodorow 16y agoA lot of people find MongoDB is faster for queries because they avoid a lot of joins.
- fehguy 16y agoYou could run mysql with the memory storage engine and with our schema--which would require multiple outer joins OR subqueries--mongodb will still be much faster. So I think it's much more than an fsync issue.
- Devilboy 16y agoIf the multiple outer joins or subqueries are too slow you can always denormalise a bit, you don't have to give up on your SQL database if you don't want to.
- megaman821 16y agoI still think it is amazing the the free SQL databases don't have materialized views yet. MySQL's deficiencies aren't inherent to SQL databases. Other databases have faster query parsers and better query planners. It seems that with all the time and money invested into NoSQL solutions, Postgres could be improved to the level of Oracle or DB2.
- saurik 16y agoThis comment, which I completely agree with, really depresses me. People really like competing with each other rather than working together to build awesome solutions (and users often even /encourage/ this behavior with a pro-competition bias). The idea that we now have a million "sort of crummy" storage solutions rather than a couple, or even one, really good one--and mostly due to superficial differences in syntax of usage or specification of deployment--makes me want to cry.
- michaelchisari 16y agoAlright, so, I fully understand the scalability reasons for using MongoDB, but I need someone to clearly explain to me when NoSQL would be a better solution than SQL from a development standpoint. Because like someone pointed out, Postgres without fsync can be just as fast. What is the advantage of giving up the ability to use SQL and the associated relational algebra that has long been established in that query language? I'm not asking to start a flame war, I'm sincerely interested. Can someone give me a use case for when NoSQL would have a clear advantage over SQL?
- thibaut_barrere 16y agoHi Michael, from a development standpoint only, I like the fact that I don't have to write database migrations (as defined in Rails). It means I can iterate more quickly during the development. As well for data aggregation kind of jobs (such as http://www.toutpourmonipad.com/ http://www.toutpourmonipad.com/ where I munge different-formatted data streams), it's really convenient to be able to mix datas that are partly equal, partly different, when it's relevant to you. Edit: forgot to mention that MongoDB comes with a built-in geographical index (MySQL doesn't have it, I'm not sure for PostGres - I believe it's via some extension). Edit2: forgot to mention I really appreciate the upsert abilities for what I do (http://www.mongodb.org/display/DOCS/Updating#Updating-UpsertswithModifiers http://www.mongodb.org/display/DOCS/Updating#Updating-Upsert...) Edit3: anyone with some curiosity for MongoDB will appreciate this book: http://www.amazon.com/MongoDB-Definitive-Guide-Kristina-Chodorow/dp/1449381561 http://www.amazon.com/MongoDB-Definitive-Guide-Kristina-Chod... - well-written and concise
- saurik 16y agoThis is because you are simply never updating your schema: if you actually want to rename a field, change a datatype, or reorganize your content, you are still going to need to run a migration, and now it won't even be possible to transanction lock the upgrade (better database servers, like PostgreSQL, can do multiple whole-daabase schema modifications within a transaction while still allowing non-conflicting access). In essence, your underlying schema is now "id->blob". ;(
- wslh 16y agoOnly had 5 days with MongoDB and I found it a good alternative for persistence of basic data structure in Python, my main concern was something that can $set individual elements of a JSON instead of retrieving the whole doc and modify it.
- mkramlich 16y ago> my main concern was something that can $set individual elements of a JSON instead of retrieving the whole doc and modify it. i ran into the same question
- nl 16y agoNot quite sure what you mean, but Atomic Modifications might be what you want: http://www.mongodb.org/display/DOCS/Atomic+Operations http://www.mongodb.org/display/DOCS/Atomic+Operations You can update any field in a document. Obviously you'll need the ID of the document.
- j_baker 16y agoDumb question: the author talks about not having to use caching because Mongo has built-in caching. Don't most RDBMSes also have built-in caching?
- andrewjshults 16y agoThey might be referring to Mongo writing data into memory rather than requiring that it be written to disk before responding success (in its default configuration) or that Mongo uses memory mapped files (and tries to uses as much memory as it can) to hold the active records in memory.
- ntoshev 16y agoYes, RDBMDSes have built-in cache, but they are not as fast as memcached. So why the difference? I haven't really checked, but you have to populate and invalidate memcached explicitly, and it doesn't honor database isolation (if you invalidate cache after writing an object, some instances may read stale data after you already updated the db, etc). MongoDB's cache may cut corners in a similar way, I'd love to know.
- semipermeable 16y agoHas anyone had experience comparing MongoDB and HiveDB from apache? I briefly considered both of them over a year ago before realizing that they weren't quite yet prime time for my application, and I've not had time to look at them since. I'm curious to see how they've evolved in practice.
- sigzero 16y agoFrom the tutorial: SELECT * FROM things WHERE name="mongo" ================================================ > db.things.find({name:"mongo"}).forEach(printjson); { "_id" : ObjectId("4c2209f9f3924d31102bd84a"), "name" : "mongo" } I am having a hard time finding the benefit of that except you can do it programmatically and not step out of your language of choice and into SQL. I am just looking into this...so maybe the lightbulb will get brighter as I go through the docs for MongoDB.