5 ms·
Does MongoDB 1.3.x (dev) silently lose your data?
- vain 16y agohasnt for me, so far ... been running three heavily written to mongo installs for more than a month...
- smokinn 16y agoI hope you have backups and have tested restoring them.
- digitallogic 16y agoAgreed, but that has nothing to do with the use of MongoDB. Any data that "can't be lost" needs to be backed up.
- andrewvc 16y agoTitle is misleading, he was using the unstable dev branch. There's no evidence here that the stable branch does this.
- luigi 16y agoHis excuse at the bottom of his post is way lame. It's very clear what is and isn't stable on MongoDB's main download page: http://www.mongodb.org/display/DOCS/Downloads http://www.mongodb.org/display/DOCS/Downloads
- benatkin 16y agoFlagged, but a title change would be acceptable, too.
- bravura 16y agoI have updated the title in response to this criticism
- benatkin 16y agoThanks! I unflagged it. Come to think of it, the article is still a good cautionary tale. What it really teaches is: a) do your research b) use the production version if you're using code in production, unless you have a really good reason not to and have some safeguards in place
- piotrSikora 16y agoThe fact that it was "unstable" branch doesn't justify data loss or corruption. It can be used as an excuse to explain why server crashed, but not why data was lost. This is big no-no in my book. Also, if the 500k documents limit on 32-bit architectures is true then it's just sad.
- runT1ME 16y agoso one class of bugs is OK, but another is not? scratches head. Have you ever done software development?
- piotrSikora 16y agoI never said that bugs are "OK", but yeah... Some bugs are acceptable and some are not. Data loss is never acceptable, especially not with databases. And yes, I did my share of software development...
- cheald 16y agoExcept in this case, it's not a bug, it's a design limitation. The MongoDB folks make no bones about it - the software is very limited on 32-bit systems. It warns you about it. It tells you to use 64-bit servers. If you insist on using the tool beyond the bounds for which it was designed to be used, that's your problem, as the developer. Is it a "bug" if you redline your car's engine and cause it to throw a rod? Or is that simply you pushing the machine past its design limits, despite the warnings not to do so?
- piotrSikora 16y agoData loss / corruption is design limitation?
- cheald 16y agoYes. MongoDB doesn't ensure a write to disk before returning on an insert operation (it is what is called "eventually consistent") in order to maintain high write throughput. Because it uses memory-mapped files, it can't address anything larger than a 32-bit pointer on a 32-bit system, and so you would execute an insert, and when the DB eventually tries to write it to disk, it would fail due to lack of resources. Because the write is asynchronous, the client doesn't realize that the data has not been serialized to disk, and you end up with effective data loss. Fixing this would require either a) not using memory-mapped files, or b) ensuring full durability on writes, both of which would massively slow down the data store as a whole.
- CitizenKane 16y agoNot only that, but he fails to grok the reason for there being a limit with the database size on 32-bit systems. It's directly a result of MongoDB using memory mapped file IO (http://en.wikipedia.org/wiki/Memory-mapped_file http://en.wikipedia.org/wiki/Memory-mapped_file) which gets better performance, but on a 32-bit system you're constrained to the amount of memory addressable by a 32-bit process. So rather than do somewhat hacky (and complex) code they opted to be forward looking and just leave the limitation in place. Thankfully, with a 64-bit OS the upper limit goes up to 256 terabytes.
- oldgregg 16y agoWahwahwah. This guy sounds like he has never used open source software before. I have a complex mongodb install in production with no problems. Every time I "bundle install" it breaks something because everything is running edge and the APIs are changing all the time. If you are not competent to deal with that then maybe you shouldn't be playing with shiny new toys.
- bobx11 16y agoI use MongoDB, collecting just under 200k documents a day for months and I haven't lost any data yet so far that we've found. Sqlite looks pretty speedy though - may try going back to something like that for smaller jobs. The one thing I found funny is that deleting documents from a database leaves the space allocated and you have to manually run the repair command to reclaim the space, but that requires 100% additional space because it creates a new db and copies the data there.... little bit of a problem since we were already a 99% for the device :-| other than that - been really happy with it.
- kscaldef 16y agoThat's not terribly uncommon. PostgreSQL also does not reclaim space from deleted rows until you vacuum. Many hash table implementations also only mark items deleted rather than actually removing them.
- selenamarie 16y agoAlthough - for further clarification, PostgreSQL doesn't require 2x the total amount of disk to reclaim/reuse space under normal circumstances. As of version 8.1 (out since 2005), VACUUMing is an automatic function inside the server. (autovacuum)
- cheald 16y agoSQLite is really nice for small projects, but it rapidly degrades once it hits large amounts of data. It's wonderfully useful for many things, but managing large datasets is not one of those things.
- henning 16y agoUsing MongoDB was a mistake to begin with even if he didn't select the wrong branch to use. His project isn't about databases, it's about machine learning. For his requirements, he just needs a simple datastore and it sounds like SQLite would have worked fine. He procrastinated by trying out cool new software (a classic symptom of a project that is not going properly) and lost a lot of time because of the mistaken manner in which he did it. He should get back on track by focusing on the real meat of his project and use the simplest bitbucket he can find for it.
- bjclark 16y agoYes, the fact that MongoDB was losing his data was all his fault because he chose MongoDB. Sound logic.
- mikeklaas 16y agoNo project is "about databases" [1]: they about something else, and databases are a tool use to achieve that end. Besides, whether he was procrastinating or not is irrelevant to his complaint. [1] ok, some are.
- bravura 16y agoIf you are doing document processing, a document database is actually the right choice. I disagree with your comment. I do machine learning and NLP, and I prefer mongodb as a data-store to RDBMs. SQLite would be overkill. MongoDB is a more convenient and simple choice than an RDBM. Using a schema-less document database is natural in NLP, where you might add columns to your fields on the fly, e.g. you are experimenting with a new preprocessing step and want to cache the result.
- luigi 16y agoIt should be noted that this post is over two months old.
- rit 16y agoEr, wow. I don't get the impression he has any idea what's going on. The title is incredibly misleading, especially given that this is a ~2 month old post. As of this writing, 1.4.x is the stable branch, with 1.5.x as the unstable. I should note that I've been running MongoDB in Production since last August. Development, deployment and go Live occurred on a pre-GA 0.9.x version and I never encountered ANY issues. With almost a year of uptime I've had no data loss or anything else. At the same time I understood I was using an unstable version and was careful to also understand what was going on under the covers. So here's the REALLY important thing you should understand if you're using MongoDB because it probably has to do with his data "loss": Data operations in Mongo, viz. insert/update are asynchronous - from both a API client and a disk-persistence standpoint. One of the things that gives you the speed boost you see from MongoDB is this concept. When your language driver sends new rows, it does NOT by default wait around to see if Mongo saved it correctly. It sends it off, makes sure MongoDB got the data and returns to you. You can force it to wait, and you can ask it for the lastError - but normally you don't wait for a "I got it in correctly" answer. Additionally, once MongoDB receives it, it goes into memory... it writes to disk lazily, rather than immediately (if you sent 1000 increment commands between the last disk write and the next, it would batch them into a single increment, etc). Yes, these things can lead to data "loss" - or the appearance thereof. If you're sending in a bad update or insert statement and not checking for errors, data will "disappear" - by which I mean it was bad data and never accepted for write.
- deleted 16y ago[deleted]
- cheald 16y agoMongoDB VERY CLEARLY warns you about the limitations of the 32-bit install when you spin it up. When I invoke mongod on my 32-bit development box, I get a nice big message: ** NOTE: when using MongoDB 32 bit, you are limited to about 2 gigabytes of data ** see http://blog.mongodb.org/post/137788967/32-bit-limitations for more If you manage to miss that, it's your own fault. Combined with the fact that he was using an unstable development branch, he has absolutely zero room whatsoever to complain. He hit a documented limit and got less-than-friendly behavior on an unstable build of the software. Going nuclear over it is just silly, and makes him look inattentive and/or naive.
- bad_user 16y agoThat seems like a brutal limitation done for non-obvious reasons ... I mean, wtf? > makes him look inattentive and/or naive Oh I wish you gave me more options. This is not a proper ad hominem.
- cheald 16y agoIt's a brutal limitation if you're expecting MongoDB to be "MySQL, except with more awesome because it's new and shiny!", sure, but once you read the blog post, it's quite apparent why - memory-mapped files are blisteringly fast, and raw speed at the cost of other features (like relational constraints, transactions, and single-server durability) is a part of the MongoDB setup. If those are acceptable sacrifices, then great, enjoy a wicked fast data store. If they're not, then it's not the tool for you, no matter how much the cool kids chat it up.
- grasshoper 16y agoSpot on. I came here to say just this. Dev branch or not, there is no way he could have missed this. I don't know how you can better advertise a limitation than this. Maybe he didn't realize he was on a 32-bit box. Maybe he didn't pay attention to how much data he would be using. If you want to use Mongo for something serious, make sure you're on a 64-bit box. Memory-mapped file I/O is an important part of what makes Mongo so fast and attractive, but it has obvious limitations on a 32-bit box.