7 ms·

Why is a database that fails so easily and most of the time even loses data so popular? Is it really all just a huge marketing budget?

by nomongo 15y ago

viraptor 15y ago
If I never expect the dataset to grow past 1GB and a single server, why would I use anything else? It doesn't really fail - none of the issues described were "failures" really. [edit: just to be clear, it didn't crash and burn, I don't think performance issue == failure] The data loss was not confirmed either: "There appears to be some data loss occurring" and in small deployments you can just use transaction log. There's no other project I know of, which provides: schemaless json documents, indexing on any part of them, server-side mapreduce, lots of connectors for different languages, atomic updates on part of the document. If there is one and it's better than mongo, I'd switch any moment.
- deleted 15y ago
  [deleted]
- cscotta 15y ago
  >> "It doesn't really fail - none of the issues described were "failures" really." These absolutely were failures. The author listed several instances in which the database became unavailable, the vendor-supplied client drivers refused to communicate with it, or both. Some of these scenarios included the primary database daemon crashing, secondaries failing to return from a "repairing" to an "online" state after a failure (and unable to serve operations in the cluster), and configuration servers failing to propagate shard config to the rest of the cluster -- which required taking down the entire database cluster to repair. Each of the issues described above would result in extended application downtime (or at best highly degraded availability), the full attention of an operations team, and potential lost revenue. The data loss concern is also unnerving. In a rapidly-moving distributed system, it can be difficult to pin down and identify the root cause of data loss. However, many techniques such as implementing counters at the application level and periodically sanity-checking them against the database can at minimum indicate that data is missing or corrupted. The issues described do not appear to be related to a journal or lack thereof. Further, the fact that the database's throughput is limited to utilizing a single core of a 16-way box due to a global write lock demonstrates that even when ample IO throughput is available, writes will be stuck contending for the global lock, while all reads are blocked. Being forced to run multiple instances of the daemon behind a sharding service on the same box to achieve any reasonable level of concurrency is embarrassing. On the "1GB / small dataset" point, keep in mind that Mongo does not permit compactions and read/write operations to occur concurrently. As documents are inserted, updated, and deleted, what may be 1GB of data will grow without bound in size, past 10GB, 16GB, 32GB, and so on until it is compacted in a write-heavy scenario. Unfortunately, compaction also requires that nodes be taken out of service. Even with small datasets, the fact that they will continue to grow without bound in write/update/delete-heavy scenarios until the node is taken out of service to be compacted further compromises the availability of the system. What's unfortunate is that many of these issues aren't simply "bugs" that can be fixed with a JIRA ticket, a patch, and a couple rounds of code review -- instead, they reach to the core of the engine itself. Even with small datasets, there are very good reasons to pause and carefully consider whether or not your application and operations team can tolerate these tradeoffs.
  - rbranson 15y ago
    Just to be 100% clear -- so people don't misunderstand your explanation of Mongo's compaction: Mongo does have a free space map that it uses to attempt to fit new data or resized documents into "holes" left by deleted data. However, compaction will still eventually have to be ran as the data will continue to fragment and eventually things get bad.
- rhizome 15y ago
  The data loss was not confirmed either: "There appears to be some data loss occurring" Oh, this mystery is a failure all right, and even the most charitable interpretation would call it a misfeature.
- japherwocky 15y ago
  what do you think of redis? I feel the same way about Mongo for the most part, but have been considering switching.
  - rkalla 15y ago
    If you can model your data in redis data structures it is excellent. Keep in mind that there is no preferred mechanism for operating redis when data is larger than ram. There is vm and diskstore, both deprecated by antirez, and a focus on data sets that fit in ram. If you can do both of those things, it is awesome.
BarkMore 15y ago
The the data model contributes to it's popularity. A document store with indexes on document fields is very convenient for several types of applications.
- jethroalias97 15y ago
  It's interesting that couchdb gets little love (as evidenced by google trends), but it has document storage by index, easy enough to install, copy on write so has no global lock, sharding with bigcouch, and all client access is entirely REST... it may be couch is a little hard to grok, I dunno.
  - wisty 15y ago
    Couch doesn't have great documentation, and doesn't have official native client drivers. Oh, and it's slow (though you can tune it, and it doesn't crumble under load).
  - rkalla 15y ago
    It isnt packaged and marketed as cleanly. It takes hunting and learning on your own to get good with couch, and map reduce is no where as easy to get started with as mongo queries (although the addition of unql may improve things next year). Also there is no single, central steward and authority on couch. All of this stymies traction and confidence even though the tech is great.
    - olefoo 15y ago
      Which all sounds rather like Postgres. It's harder to use because you have to know what you're doing, and it's not as popular even though it's better on some axes that are significant if you're building something solid.
    - jethroalias97 15y ago
      This is true. I think couchdb tries to be all things to all people rather than just focusing on being a great data store. It's a database like mongo, it's a mobile database like sqlite, you can use it to host apps with couchapp, you can use it as a map-reduce cluster like hadoop, etc. I would rather have just a solid no-sql database that solves that problem. I like couch better than mongo for the reasons I mentioned, but I know couch also has problems and still haven't found a good third option.
hello_moto 15y ago
There are tons of reasons for that. Let me pull some of them from my butt: Reason #1: Devs aren't Ops. Reason #2: Devs need something new on their resume. Reason #3: Certain type of Devs would read blogs and get excited and skipping scientific mumbo-jumbo and directly take the blogs as _the_ source of truth. Reason #4: It's easy to bootstrap (schemaless, etc) your weekend project. Dealing with DB apparently is tedious for devs. I'm sure others can add more... Let me feel your love HN-ers ;)
- viraptor 15y ago
  Why do I get the feeling that you're an op and look down on development people? If that's really true, try to start developing some project and see how you like frequent schema changes, trying to synchronise schemas with peers, resolving relation issues when merging features, etc. On the other hand if you abstract your interaction with data enough, you can change the whole backend later once it's stable and not care about it up-front. What I hear you saying is unfortunately - it's worse for ops, so noone should use it.
  - hello_moto 15y ago
    I've been dev since I graduated. I had been everything else during my intern (dev, qa, test-automation developer, tools dev, build engineer, integration engineer, etc). Are you saying that Rails schema migration can only solve 10% of your migration needs? That kinda suck bro.
    - viraptor 15y ago
      Migration tools are great once you stop changing the schema very often like it happens when a project starts. This of course depends on the project... if you can have a full design from the start, it's probably going to work too. If you don't know the exact requirements or way to get there - not so much.
      - hello_moto 15y ago
        If you change your schema once every 2-3 days, something wrong with whomever leading the software project. That's like writing a software with zero planning or lack of knowledge for the problem domain. I don't care if it is a startup or not. Come up with a very simple idea, draw the models in ER diagram, implement that stuff. It's very hard to imagine that tomorrow suddenly all relationships need to be changed. Even if that is the case, scrap your Repository/Entity model and start from the beginning. Nothing can help you much if the fundamentals are wrong.
rbranson 15y ago
10gen has focused strongly on ease of adoption, which seems like the highest priority of MongoDB at this point. From what I can tell, the idea is to get everyone using it, and then "scale" it once you've got people willing to pay out $ for fixes, but sometimes bad decisions made early on (like the global locks and in-place updates) are harder to change than originally thought.
- hello_moto 15y ago
  Not a bad business strategy. Kinda like MySQL back then right?
  - cperciva 15y ago
    Sadly, it seems that "give them free crap and then charge for fixes" is a very common business model in the open source world.
    - hello_moto 15y ago
      Better than asking for big pile of money and give back crap and charge way more for fixes isn't it? :)
      - cperciva 15y ago
        Yes. But I think asking happy users to donate ultimately produces better code than asking unhappy users to pay for support.
        hello_moto 15y ago
        So true. So true. I wish happy users would donate more as well. Unfortunately when happy users get used to the culture of free and good quality software, they started to have a sense of entitlement (instead of donating). That if the software didn't provide exactly what they wanted, they starting to swear and whine instead of being... calm and helpful.
FooBarWidget 15y ago
Not all data is important enough that small losses are unacceptable. Analytics data that can be inferred from other sources, for example. Furthermore MongoDB supports autosharding while most (all?) SQL databases do not.
rdtsc 15y ago
I have been asking that too and I concluded that it is due to dishonest marketing. Up until a couple of months ago they basically shipped a database product with disabled singe server durability. That fact should have been written in bright flashing red letter warning on their front page, it wasn't. So it made for very fast benchmarks, because everyone benchmarks for speed, not many benchmark for failure.
InclinedPlane 15y ago
So what's the preferred alternative noSQL wise? MongoDB is flaky. CouchDB is a maintainability nightmare, so I hear. Riak? Cassandra? Or does everything else have some other equally huge down-side?
- rkalla 15y ago
  They all have their warts. For every story like this, there are petabyte deployments of your favorite datastore that work fine. For every X sucks article, ther is Y is awesome. In the nosql world the only way to choose is around the problems they solve... They are each specializing and optimizing for certain nitches. mongo is the most mysql-esque, but dosnt do things that redis, couch or cassandra do that you may need. There is no clear winner (fortunately or unfortunately dependng on what you were hoping for)
  - InclinedPlane 15y ago
    Have there been any new entrants in the last few years? Seems like innovation has stalled a bit and stabilization / improvement hasn't caught up yet.
    - rkalla 15y ago
      Good question -- as far as new, mature NoSQL solutions on the scene, I just became aware of OrientDB which sort of baffles me with it's functionality. It looks like this amazingly functional blend of MySQL and NoSQL: http://www.orientechnologies.com/orient-db.htm http://www.orientechnologies.com/orient-db.htm Other than that, I actually think these solutions have been stabilizing exactly because of what you say: innovation is slowing down/stalling. 1-3 years ago the cool thing to do was store data different ways, now that we have all these solutions that people are ready to use in production, they are demanding more and more secure/safe functionality from them. In the last year Redis added the append log and flushing to disk. CouchDB rewrote the replication code in the last release and has always had a wonderfully redundant and safe file mutation model (can can copy the DB file while in use and still get a safe snapshot) and MongoDB has been responding aggressively to crashes and corruption since 1.7 after all the single-server durability fiasco around 1.5/6 that had everyone up in arms. These data stores are really brilliant pieces of code with some wonderful deployments to prove their worth. There is still work to be done, sure, but I am not aware of glaring deficiencies in these systems like I used to be a year or more ago where you could point at "Oh, the XYZ bug might get you" -- that just doesn't seem to be happening anymore. I don't know a whole hell of a lot about Cassandra (I am one of the few humans that still doesn't grok the data model easily) but I remember data recovery bugs from a year ago in the issue tracker that all got knocked out to the point that 1.x is looking like a really awesome release for them. At this point, I think it just depends on what you need.
fdr 15y ago
It has a pretty good user experience, except for all the details. But the model isn't bad; it should be learned from. On the other hand, there is no trade-off made by Mongo that I'm aware of that is not fundamentally unavailable to more mature projects in a tractable amount of engineering time, so the question comes down to "does Mongo shed its reputation for lulz soon enough" vs "do other projects witness and adapt". Yet we've also seen in the past that shedding such a reputation is not strictly required to be popular. And marketing budgets do matter.
dextorious 15y ago
Probably because the quality of CS graduates has been so low at recent years. MongoDB = oh, shiny, fast.
vannevar 15y ago
Why is a database that fails so easily and most of the time even loses data so popular? Perhaps because both of your premises are wrong? I've used Mongo for over a year now with ~1000 writes/sec and haven't seen any of these problems. I'm not saying they don't exist (some are confirmed bugs that have been fixed), but they're not nearly as prevalent as your 'Do you still beat your wife?'-style question implies.

Why is a database that fails so easily and most of the time even loses data so popular? Is it really all just a huge marketing budget?

by nomongo 15y ago

Why is a database that fails so easily and most of the time even loses data so popular? Is it really all just a huge marketing budget?

viraptor 15y ago

If I never expect the dataset to grow past 1GB and a single server, why would I use anything else? It doesn't really fail - none of the issues described were "failures" really. [edit: just to be clear, it didn't crash and burn, I don't think performance issue == failure] The data loss was not confirmed either: "There appears to be some data loss occurring" and in small deployments you can just use transaction log. There's no other project I know of, which provides: schemaless json documents, indexing on any part of them, server-side mapreduce, lots of connectors for different languages, atomic updates on part of the document. If there is one and it's better than mongo, I'd switch any moment.

deleted 15y ago
[deleted]
cscotta 15y ago
>> "It doesn't really fail - none of the issues described were "failures" really." These absolutely were failures. The author listed several instances in which the database became unavailable, the vendor-supplied client drivers refused to communicate with it, or both. Some of these scenarios included the primary database daemon crashing, secondaries failing to return from a "repairing" to an "online" state after a failure (and unable to serve operations in the cluster), and configuration servers failing to propagate shard config to the rest of the cluster -- which required taking down the entire database cluster to repair. Each of the issues described above would result in extended application downtime (or at best highly degraded availability), the full attention of an operations team, and potential lost revenue. The data loss concern is also unnerving. In a rapidly-moving distributed system, it can be difficult to pin down and identify the root cause of data loss. However, many techniques such as implementing counters at the application level and periodically sanity-checking them against the database can at minimum indicate that data is missing or corrupted. The issues described do not appear to be related to a journal or lack thereof. Further, the fact that the database's throughput is limited to utilizing a single core of a 16-way box due to a global write lock demonstrates that even when ample IO throughput is available, writes will be stuck contending for the global lock, while all reads are blocked. Being forced to run multiple instances of the daemon behind a sharding service on the same box to achieve any reasonable level of concurrency is embarrassing. On the "1GB / small dataset" point, keep in mind that Mongo does not permit compactions and read/write operations to occur concurrently. As documents are inserted, updated, and deleted, what may be 1GB of data will grow without bound in size, past 10GB, 16GB, 32GB, and so on until it is compacted in a write-heavy scenario. Unfortunately, compaction also requires that nodes be taken out of service. Even with small datasets, the fact that they will continue to grow without bound in write/update/delete-heavy scenarios until the node is taken out of service to be compacted further compromises the availability of the system. What's unfortunate is that many of these issues aren't simply "bugs" that can be fixed with a JIRA ticket, a patch, and a couple rounds of code review -- instead, they reach to the core of the engine itself. Even with small datasets, there are very good reasons to pause and carefully consider whether or not your application and operations team can tolerate these tradeoffs.
- rbranson 15y ago
  Just to be 100% clear -- so people don't misunderstand your explanation of Mongo's compaction: Mongo does have a free space map that it uses to attempt to fit new data or resized documents into "holes" left by deleted data. However, compaction will still eventually have to be ran as the data will continue to fragment and eventually things get bad.
rhizome 15y ago
The data loss was not confirmed either: "There appears to be some data loss occurring" Oh, this mystery is a failure all right, and even the most charitable interpretation would call it a misfeature.
japherwocky 15y ago
what do you think of redis? I feel the same way about Mongo for the most part, but have been considering switching.
- rkalla 15y ago
  If you can model your data in redis data structures it is excellent. Keep in mind that there is no preferred mechanism for operating redis when data is larger than ram. There is vm and diskstore, both deprecated by antirez, and a focus on data sets that fit in ram. If you can do both of those things, it is awesome.

BarkMore 15y ago

The the data model contributes to it's popularity. A document store with indexes on document fields is very convenient for several types of applications.

jethroalias97 15y ago
It's interesting that couchdb gets little love (as evidenced by google trends), but it has document storage by index, easy enough to install, copy on write so has no global lock, sharding with bigcouch, and all client access is entirely REST... it may be couch is a little hard to grok, I dunno.
- wisty 15y ago
  Couch doesn't have great documentation, and doesn't have official native client drivers. Oh, and it's slow (though you can tune it, and it doesn't crumble under load).
- rkalla 15y ago
  It isnt packaged and marketed as cleanly. It takes hunting and learning on your own to get good with couch, and map reduce is no where as easy to get started with as mongo queries (although the addition of unql may improve things next year). Also there is no single, central steward and authority on couch. All of this stymies traction and confidence even though the tech is great.
  - olefoo 15y ago
    Which all sounds rather like Postgres. It's harder to use because you have to know what you're doing, and it's not as popular even though it's better on some axes that are significant if you're building something solid.
  - jethroalias97 15y ago
    This is true. I think couchdb tries to be all things to all people rather than just focusing on being a great data store. It's a database like mongo, it's a mobile database like sqlite, you can use it to host apps with couchapp, you can use it as a map-reduce cluster like hadoop, etc. I would rather have just a solid no-sql database that solves that problem. I like couch better than mongo for the reasons I mentioned, but I know couch also has problems and still haven't found a good third option.

hello_moto 15y ago

There are tons of reasons for that. Let me pull some of them from my butt: Reason #1: Devs aren't Ops. Reason #2: Devs need something new on their resume. Reason #3: Certain type of Devs would read blogs and get excited and skipping scientific mumbo-jumbo and directly take the blogs as _the_ source of truth. Reason #4: It's easy to bootstrap (schemaless, etc) your weekend project. Dealing with DB apparently is tedious for devs. I'm sure others can add more... Let me feel your love HN-ers ;)

viraptor 15y ago
Why do I get the feeling that you're an op and look down on development people? If that's really true, try to start developing some project and see how you like frequent schema changes, trying to synchronise schemas with peers, resolving relation issues when merging features, etc. On the other hand if you abstract your interaction with data enough, you can change the whole backend later once it's stable and not care about it up-front. What I hear you saying is unfortunately - it's worse for ops, so noone should use it.
- hello_moto 15y ago
  I've been dev since I graduated. I had been everything else during my intern (dev, qa, test-automation developer, tools dev, build engineer, integration engineer, etc). Are you saying that Rails schema migration can only solve 10% of your migration needs? That kinda suck bro.
  - viraptor 15y ago
    Migration tools are great once you stop changing the schema very often like it happens when a project starts. This of course depends on the project... if you can have a full design from the start, it's probably going to work too. If you don't know the exact requirements or way to get there - not so much.
    - hello_moto 15y ago
      If you change your schema once every 2-3 days, something wrong with whomever leading the software project. That's like writing a software with zero planning or lack of knowledge for the problem domain. I don't care if it is a startup or not. Come up with a very simple idea, draw the models in ER diagram, implement that stuff. It's very hard to imagine that tomorrow suddenly all relationships need to be changed. Even if that is the case, scrap your Repository/Entity model and start from the beginning. Nothing can help you much if the fundamentals are wrong.

rbranson 15y ago

10gen has focused strongly on ease of adoption, which seems like the highest priority of MongoDB at this point. From what I can tell, the idea is to get everyone using it, and then "scale" it once you've got people willing to pay out $ for fixes, but sometimes bad decisions made early on (like the global locks and in-place updates) are harder to change than originally thought.

hello_moto 15y ago
Not a bad business strategy. Kinda like MySQL back then right?
- cperciva 15y ago
  Sadly, it seems that "give them free crap and then charge for fixes" is a very common business model in the open source world.
  - hello_moto 15y ago
    Better than asking for big pile of money and give back crap and charge way more for fixes isn't it? :)
    - cperciva 15y ago
      Yes. But I think asking happy users to donate ultimately produces better code than asking unhappy users to pay for support.
      - hello_moto 15y ago
        So true. So true. I wish happy users would donate more as well. Unfortunately when happy users get used to the culture of free and good quality software, they started to have a sense of entitlement (instead of donating). That if the software didn't provide exactly what they wanted, they starting to swear and whine instead of being... calm and helpful.

FooBarWidget 15y ago

Not all data is important enough that small losses are unacceptable. Analytics data that can be inferred from other sources, for example. Furthermore MongoDB supports autosharding while most (all?) SQL databases do not.

rdtsc 15y ago

I have been asking that too and I concluded that it is due to dishonest marketing. Up until a couple of months ago they basically shipped a database product with disabled singe server durability. That fact should have been written in bright flashing red letter warning on their front page, it wasn't. So it made for very fast benchmarks, because everyone benchmarks for speed, not many benchmark for failure.

InclinedPlane 15y ago

So what's the preferred alternative noSQL wise? MongoDB is flaky. CouchDB is a maintainability nightmare, so I hear. Riak? Cassandra? Or does everything else have some other equally huge down-side?

rkalla 15y ago
They all have their warts. For every story like this, there are petabyte deployments of your favorite datastore that work fine. For every X sucks article, ther is Y is awesome. In the nosql world the only way to choose is around the problems they solve... They are each specializing and optimizing for certain nitches. mongo is the most mysql-esque, but dosnt do things that redis, couch or cassandra do that you may need. There is no clear winner (fortunately or unfortunately dependng on what you were hoping for)
- InclinedPlane 15y ago
  Have there been any new entrants in the last few years? Seems like innovation has stalled a bit and stabilization / improvement hasn't caught up yet.
  - rkalla 15y ago
    Good question -- as far as new, mature NoSQL solutions on the scene, I just became aware of OrientDB which sort of baffles me with it's functionality. It looks like this amazingly functional blend of MySQL and NoSQL: http://www.orientechnologies.com/orient-db.htm http://www.orientechnologies.com/orient-db.htm Other than that, I actually think these solutions have been stabilizing exactly because of what you say: innovation is slowing down/stalling. 1-3 years ago the cool thing to do was store data different ways, now that we have all these solutions that people are ready to use in production, they are demanding more and more secure/safe functionality from them. In the last year Redis added the append log and flushing to disk. CouchDB rewrote the replication code in the last release and has always had a wonderfully redundant and safe file mutation model (can can copy the DB file while in use and still get a safe snapshot) and MongoDB has been responding aggressively to crashes and corruption since 1.7 after all the single-server durability fiasco around 1.5/6 that had everyone up in arms. These data stores are really brilliant pieces of code with some wonderful deployments to prove their worth. There is still work to be done, sure, but I am not aware of glaring deficiencies in these systems like I used to be a year or more ago where you could point at "Oh, the XYZ bug might get you" -- that just doesn't seem to be happening anymore. I don't know a whole hell of a lot about Cassandra (I am one of the few humans that still doesn't grok the data model easily) but I remember data recovery bugs from a year ago in the issue tracker that all got knocked out to the point that 1.x is looking like a really awesome release for them. At this point, I think it just depends on what you need.

fdr 15y ago

It has a pretty good user experience, except for all the details. But the model isn't bad; it should be learned from. On the other hand, there is no trade-off made by Mongo that I'm aware of that is not fundamentally unavailable to more mature projects in a tractable amount of engineering time, so the question comes down to "does Mongo shed its reputation for lulz soon enough" vs "do other projects witness and adapt". Yet we've also seen in the past that shedding such a reputation is not strictly required to be popular. And marketing budgets do matter.

dextorious 15y ago

Probably because the quality of CS graduates has been so low at recent years. MongoDB = oh, shiny, fast.

vannevar 15y ago

Why is a database that fails so easily and most of the time even loses data so popular? Perhaps because both of your premises are wrong? I've used Mongo for over a year now with ~1000 writes/sec and haven't seen any of these problems. I'm not saying they don't exist (some are confirmed bugs that have been fixed), but they're not nearly as prevalent as your 'Do you still beat your wife?'-style question implies.