5 ms·

You sound like a "database system design" is just one specific thing that has a well-defined definition which is known to and agreed on by everyone. Ughh... fin

You sound like a "database system design" is just one specific thing that has a well-defined definition which is known to and agreed on by everyone. Ughh... fine, not arguing. But in that case MongoDB is not a "database system" then. How about I call it a tool that holds your data for you and gives it back later. This tool has certain properties that make it work (or not work) for certain applications. It is just that simple. I do not work with/for 10gen but we have been using MongoDB since version 1.5 and found it incredibly useful addition to our toolbox. It always helps to own more than one hammer.

DenisM 15y ago
Sure it might be useful, I just don't think it has longevity in it. In other words it will be surpassed by another system, aiming to solve the same set of problems, but designed with concurrency in mind from the start. MongoDB developers will end up spending all their efforts to retrofit concurrency, and the community of users will simply move on.
- xxqs 15y ago
  I guess some other team will eventually spin off an alternative server engine, still keeping the API compatibility, but improving or re-designing the poor parts. Let's wait another year or two. Pity is that as an application developer, I cannot rely on Mongo as a long-term backend solution.
- milkshakes 15y ago
  In other words it will be surpassed by another system, aiming to solve the same set of problems, but designed with concurrency in mind from the start. MongoDB developers will end up spending all their efforts to retrofit concurrency, and the community of users will simply move on. Fine, I'll bite. Your assertion that MongoDB was not designed "with concurrency in mind" is simply wrong. They have in fact put a lot of thought into it, they simply made decisions that you either do not understand, or happen to disagree with. Nobody at 10gen is arguing that a global write lock is the best idea in the world. They are acutely aware of the drawbacks. They initially implemented it that way because it significantly reduces the complexity of offering concurrency in the first place. There's a reason that a NoSQL solution with more granular locking doesn't exist yet. It's much more difficult to construct. There is an ongoing, documented, transparent effort to migrate toward more granular locking, but that is hard and takes time. In the meantime, they have been working on mitigating the drawbacks of their decision by aggressively yielding whenever possible during long running operations. Did you even read the article? Did you look at the benchmarks? The entire point of it was it's not as bad as you would think, and it's getting better.
  - timr 15y ago
    "Did you even read the article? Did you look at the benchmarks? The entire point of it was it's not as bad as you would think, and it's getting better." It's still quite bad -- the 2.0 benchmark shows a dropoff of 1000 read qps with as few as 60 faults per second. The 1.8 metrics drop to damn near zero reads under the same scenario. That's obviously worse, but a 33% drop in read capacity for 60 faults/sec is bad, no matter how you look at it. Another way of thinking about it: if you're throwing 3,000 qps at a mongo instance, and expecting only 60 faults/s, then you need to ensure that 2940/3000 (98%) of your working set fits in RAM. Maybe there are toy problems where that's true as the DB grows to terabytes of persistent storage, but there aren't very many.
    - DenisM 15y ago
      To be fair, there is a class of problems that fits in RAM, e.g. financial systems. (For more detailed examples, consider the YC startup MemSQL). However these scenarios are write-intensive and demand very high concurrency.
  - DenisM 15y ago
    >that is hard and takes time. And I'm arguing that's going to be a hell of ride, which is more likely to break their backs than yield success. I read up on the link you posted elsewhere (thanks for that, by the way) and well, it's just as bad as I thought. Imagine, in pre-2.0 world, there's a greedy reader, and a subsequent writer queued up on the lock. All subsequent readers are blocked until writer quits, which can't quit before greedy reader does. This is a nightmare. That was before 2.0, now the greedy reader will yield, writer will finish, and all the pending readers will be unblocked. This is only an improvement if you consider the nightmarish previous situation. There are still two problems: 1) single writer blocks all readers on a shard while in progress 2) as soon as a new writer is queued up behind the reader lock, all subsequent readers are queued up again. Does this look like optimal resource use to you? It does not to me. Let's contrast this with "legacy" engines: Sybase: readers/writer lock has granularity of a 8kb page. If you're not touching the same page someone else is writing you're fine. (*) they might have moved on since the 1990-s, I haven't looked. Microsoft: reader/writer lock has granularity of a row. If you're not reading a row someone is writing, you're fine. That was in the 1990s, they have since moved on to snapshots, but have not yet made them default option, I think. PostgreSQL or Oracle: 1) readers are reading a snapshot and never block writers 2) writers block each other, and granularity of locking is single row. If you're not writing the same row someone else is writing you're fine. SQL Lite - readers do not block writers, there is a database-wide writer/writer lock. Note that this is a very lightweight desktop-oriented database, not a cloud solution. MongoDB - reader/writer lock granularity is a shard, the part of the database apportioned to a single CPU core. If you happen to read data on the same shard someone is writing, or is planning to write you're not fine at all. Their plans are "collection level locking". So I get it, you're saying they planned to add serious concurrency later. I agree on that - they planned. Where you and I disagree is that they will likely fail, because retrofitting concurrency is exceptionally hard. I just can't believe that anyone who knows what he's getting into would actually agree to get into this. I understand you need to compromise something when you start out, but I think concurrency is the worst possible choice.
- yxhuvud 15y ago
  Sqlite is also not designed with concurrency in mind. That doesn't mean it doesn't fill a purpose or that it won't be with us for a long time.

DenisM 15y ago

Sure it might be useful, I just don't think it has longevity in it. In other words it will be surpassed by another system, aiming to solve the same set of problems, but designed with concurrency in mind from the start. MongoDB developers will end up spending all their efforts to retrofit concurrency, and the community of users will simply move on.

xxqs 15y ago
I guess some other team will eventually spin off an alternative server engine, still keeping the API compatibility, but improving or re-designing the poor parts. Let's wait another year or two. Pity is that as an application developer, I cannot rely on Mongo as a long-term backend solution.
milkshakes 15y ago
In other words it will be surpassed by another system, aiming to solve the same set of problems, but designed with concurrency in mind from the start. MongoDB developers will end up spending all their efforts to retrofit concurrency, and the community of users will simply move on. Fine, I'll bite. Your assertion that MongoDB was not designed "with concurrency in mind" is simply wrong. They have in fact put a lot of thought into it, they simply made decisions that you either do not understand, or happen to disagree with. Nobody at 10gen is arguing that a global write lock is the best idea in the world. They are acutely aware of the drawbacks. They initially implemented it that way because it significantly reduces the complexity of offering concurrency in the first place. There's a reason that a NoSQL solution with more granular locking doesn't exist yet. It's much more difficult to construct. There is an ongoing, documented, transparent effort to migrate toward more granular locking, but that is hard and takes time. In the meantime, they have been working on mitigating the drawbacks of their decision by aggressively yielding whenever possible during long running operations. Did you even read the article? Did you look at the benchmarks? The entire point of it was it's not as bad as you would think, and it's getting better.
- timr 15y ago
  "Did you even read the article? Did you look at the benchmarks? The entire point of it was it's not as bad as you would think, and it's getting better." It's still quite bad -- the 2.0 benchmark shows a dropoff of 1000 read qps with as few as 60 faults per second. The 1.8 metrics drop to damn near zero reads under the same scenario. That's obviously worse, but a 33% drop in read capacity for 60 faults/sec is bad, no matter how you look at it. Another way of thinking about it: if you're throwing 3,000 qps at a mongo instance, and expecting only 60 faults/s, then you need to ensure that 2940/3000 (98%) of your working set fits in RAM. Maybe there are toy problems where that's true as the DB grows to terabytes of persistent storage, but there aren't very many.
  - DenisM 15y ago
    To be fair, there is a class of problems that fits in RAM, e.g. financial systems. (For more detailed examples, consider the YC startup MemSQL). However these scenarios are write-intensive and demand very high concurrency.
- DenisM 15y ago
  >that is hard and takes time. And I'm arguing that's going to be a hell of ride, which is more likely to break their backs than yield success. I read up on the link you posted elsewhere (thanks for that, by the way) and well, it's just as bad as I thought. Imagine, in pre-2.0 world, there's a greedy reader, and a subsequent writer queued up on the lock. All subsequent readers are blocked until writer quits, which can't quit before greedy reader does. This is a nightmare. That was before 2.0, now the greedy reader will yield, writer will finish, and all the pending readers will be unblocked. This is only an improvement if you consider the nightmarish previous situation. There are still two problems: 1) single writer blocks all readers on a shard while in progress 2) as soon as a new writer is queued up behind the reader lock, all subsequent readers are queued up again. Does this look like optimal resource use to you? It does not to me. Let's contrast this with "legacy" engines: Sybase: readers/writer lock has granularity of a 8kb page. If you're not touching the same page someone else is writing you're fine. (*) they might have moved on since the 1990-s, I haven't looked. Microsoft: reader/writer lock has granularity of a row. If you're not reading a row someone is writing, you're fine. That was in the 1990s, they have since moved on to snapshots, but have not yet made them default option, I think. PostgreSQL or Oracle: 1) readers are reading a snapshot and never block writers 2) writers block each other, and granularity of locking is single row. If you're not writing the same row someone else is writing you're fine. SQL Lite - readers do not block writers, there is a database-wide writer/writer lock. Note that this is a very lightweight desktop-oriented database, not a cloud solution. MongoDB - reader/writer lock granularity is a shard, the part of the database apportioned to a single CPU core. If you happen to read data on the same shard someone is writing, or is planning to write you're not fine at all. Their plans are "collection level locking". So I get it, you're saying they planned to add serious concurrency later. I agree on that - they planned. Where you and I disagree is that they will likely fail, because retrofitting concurrency is exceptionally hard. I just can't believe that anyone who knows what he's getting into would actually agree to get into this. I understand you need to compromise something when you start out, but I think concurrency is the worst possible choice.
yxhuvud 15y ago
Sqlite is also not designed with concurrency in mind. That doesn't mean it doesn't fill a purpose or that it won't be with us for a long time.