4 ms·

I have to say that I did not experience a single data loss that was a result of the database misbehaving. Memory leaks weren't a huge issue for us as well. Aft

I have to say that I did not experience a single data loss that was a result of the database misbehaving. Memory leaks weren't a huge issue for us as well. After stabilizing the setup I have to say it was basically a fire and forget part of the stack for us. The role of MongoDB was to act as a fast-insert and aggregation framework for other parts of the system. So, we would insert BIG amounts of data at a time and aggregate it into a K/V store where we pulled the data from. After a while, the setup became to expensive to run, at this point we turned it off for a cheaper solution, but in terms of functionality, it functioned pretty well.

mack73 10y ago
Fire-and-forget is awesome for when you do not care at all about your data. I'm sure those types of writes are super (duper) fast. What exactly is the use case for that? Serious question.
- vidarh 10y ago
  Don't know about the guy you replied to, but e.g. consider any application that regularly crawl feeds, api's etc. where the data is rapidly changing and only a portion of the data is necessary to give good output. There are lots of applications like that where you just need "enough" data to give good results and/or where any loss will auto-heal next time you crawl the original source.
  - mack73 10y ago
    That makes sense to me. If your MongoDB cluster under preasure will only actually persist 90% of your writes and this is something you anticipate, then MongoDB seems like a good choice, if writing in this style is faster than other nosql systems (that make grander promises about persistance) that is.
    - vidarh 10y ago
      Exactly - the important thing is you need to actually understand the risk, and make an informed decision what level of loss is ok to you (and you should understand whether or not it's actually saving you anything - as you say, it makes sense if it is faster; there's no point losing data if you don't gain something from accepting the risk). This is also perhaps the biggest problem with MongoDB: It's fast but unsafe "out of the box", and not everyone will know that when they use it. I think that's a large part of the problem a lot of people have with it.
- devishard 10y ago
  I understand that some data loss is acceptable sometimes, but you can get the super-fast writes with a store that doesn't drop your data at random (such as Cassandra) so why would you choose the store that does?
nollbit 10y ago
> I have to say that I did not experience a single data loss that was a result of the database misbehaving. How do you know? Was your data checksummed? Did you read back changes after writes to verify what was written?
devishard 10y ago
> I have to say that I did not experience a single data loss that was a result of the database misbehaving. Okay, but numerous people have experienced data loss using MongoDB. You can spend 10 minutes on Twitter and find someone talking about it. > Memory leaks weren't a huge issue for us as well. After stabilizing the setup I have to say it was basically a fire and forget part of the stack for us. "After stabilizing the setup"? So basically you worked around MongoDB's stability problems instead of choosing a solution that didn't have stability problems? > The role of MongoDB was to act as a fast-insert and aggregation framework for other parts of the system. Sure, but there are other solutions that can do this without MongoDB's issues (i.e. Cassandra). > After a while, the setup became to expensive to run, at this point we turned it off for a cheaper solution, but in terms of functionality, it functioned pretty well. Sure, you can build something stable on sand, but it's going to be a lot harder than just building on a solid foundation.
- avitzurel 10y ago
  > Okay, but numerous people have experienced data loss using MongoDB. You can spend 10 minutes on Twitter and find someone talking about it. Never said people didn't. > "After stabilizing the setup"? So basically you worked around MongoDB's stability problems instead of choosing a solution that didn't have stability problems? No, I haven't worked around the limitations, after tweaking heap sizes, machine sizes and disk speeds the setup was very stable. > Sure, but there are other solutions that can do this without MongoDB's issues (i.e. Cassandra). Again, I DO NOT disagree, there are other solutions and today we are running a completely different setup to replace the same component. > Sure, you can build something stable on sand, but it's going to be a lot harder than just building on a solid foundation. I'm trying to ignore the snarky/attacking nature of your comment. Not sure if this is addressable.

mack73 10y ago

Fire-and-forget is awesome for when you do not care at all about your data. I'm sure those types of writes are super (duper) fast. What exactly is the use case for that? Serious question.

vidarh 10y ago
Don't know about the guy you replied to, but e.g. consider any application that regularly crawl feeds, api's etc. where the data is rapidly changing and only a portion of the data is necessary to give good output. There are lots of applications like that where you just need "enough" data to give good results and/or where any loss will auto-heal next time you crawl the original source.
- mack73 10y ago
  That makes sense to me. If your MongoDB cluster under preasure will only actually persist 90% of your writes and this is something you anticipate, then MongoDB seems like a good choice, if writing in this style is faster than other nosql systems (that make grander promises about persistance) that is.
  - vidarh 10y ago
    Exactly - the important thing is you need to actually understand the risk, and make an informed decision what level of loss is ok to you (and you should understand whether or not it's actually saving you anything - as you say, it makes sense if it is faster; there's no point losing data if you don't gain something from accepting the risk). This is also perhaps the biggest problem with MongoDB: It's fast but unsafe "out of the box", and not everyone will know that when they use it. I think that's a large part of the problem a lot of people have with it.
devishard 10y ago
I understand that some data loss is acceptable sometimes, but you can get the super-fast writes with a store that doesn't drop your data at random (such as Cassandra) so why would you choose the store that does?

nollbit 10y ago

> I have to say that I did not experience a single data loss that was a result of the database misbehaving. How do you know? Was your data checksummed? Did you read back changes after writes to verify what was written?

devishard 10y ago

> I have to say that I did not experience a single data loss that was a result of the database misbehaving. Okay, but numerous people have experienced data loss using MongoDB. You can spend 10 minutes on Twitter and find someone talking about it. > Memory leaks weren't a huge issue for us as well. After stabilizing the setup I have to say it was basically a fire and forget part of the stack for us. "After stabilizing the setup"? So basically you worked around MongoDB's stability problems instead of choosing a solution that didn't have stability problems? > The role of MongoDB was to act as a fast-insert and aggregation framework for other parts of the system. Sure, but there are other solutions that can do this without MongoDB's issues (i.e. Cassandra). > After a while, the setup became to expensive to run, at this point we turned it off for a cheaper solution, but in terms of functionality, it functioned pretty well. Sure, you can build something stable on sand, but it's going to be a lot harder than just building on a solid foundation.

avitzurel 10y ago
> Okay, but numerous people have experienced data loss using MongoDB. You can spend 10 minutes on Twitter and find someone talking about it. Never said people didn't. > "After stabilizing the setup"? So basically you worked around MongoDB's stability problems instead of choosing a solution that didn't have stability problems? No, I haven't worked around the limitations, after tweaking heap sizes, machine sizes and disk speeds the setup was very stable. > Sure, but there are other solutions that can do this without MongoDB's issues (i.e. Cassandra). Again, I DO NOT disagree, there are other solutions and today we are running a completely different setup to replace the same component. > Sure, you can build something stable on sand, but it's going to be a lot harder than just building on a solid foundation. I'm trying to ignore the snarky/attacking nature of your comment. Not sure if this is addressable.