6 ms·

MongoDB CTO: How our new storage engine will earn its stripes

71 points by lucasjans 12y ago

qwerta 12y ago
I am glad they finally introduced LSM trees and per document locking. But I think thats too little too late :-)
- samneilz 12y ago
  Agreed
- remon 12y ago
  Why on earth would it be too late? MongoDB is comfortably in the top 3 in terms of traction and they just fixed what many consider the primary technical argument against using it. Too late for what?
  - davidw 12y ago
    Here's my guess about some of the most utilized/popular databases: * SQLite - it's everywhere thanks to mobile phones. * Oracle. * Microsoft's thing has to have a big following. * Mysql. * DB2 is not as big as Oracle, but IBM isn't some lightweight, either. MongoDB gets talked about a lot, and probably used among people trying new stuff out, but I think a lot of mostly silent people wouldn't touch something like that for their important data with a 10 foot pole.
    - bane 12y ago
      > SQLite - it's everywhere thanks to mobile phones. and browsers...and tons of other pieces of software
    - arethuza 12y ago
      "Microsoft's thing" SQL Server is definitely one of Microsoft's best products - as well as being an excellent database engine it also is used by SharePoint, the various Dynamics applications (CRM & ERP) as well as being the "natural" choice for application development in .Net If I had to guess I would suspect that, at least in terms of revenue, SQL Server probably tops Oracle and DB2.
    - dkhenry 12y ago
      I think you overestimate the reach of Oracle. They are no where outside fortuine 500 and thats not a lot of deployments. MySQL is most likely the most deployed thanks to the LAMP stack's proliferation across all sorts of small and medium sites. After that it really would be a toss up between MS-SQL who has traction in the SMB space due to it being used for sharepoint and exchange, and PostgreSQL but lets be honest the most utilized database is Access.
      - dagw 12y ago
        I do a lot of work with geo-spatial analysis and often work with with various counties and municipalities in my experience everybody has an Oracle database stuffed away somewhere storing the regions canonical geo-spatial data.
        dkhenry 12y ago
        Even if each town had an oracle database squirreled away ( which they don't ) That still would not even come close to just the wordpress deployments (60 Million).
      - collyw 12y ago
        Sadly many more people use Excel as their database when they should be using Access.
  - johngd 12y ago
    I am generally curious about the source for your claim. Would you mind sharing, please?
  - jeltz 12y ago
    MongoDB is placed 5th at DB-Engines, and I would suspect SQLite should actually be ranked higher since it used virtually everywhere. http://db-engines.com/en/ranking http://db-engines.com/en/ranking
    - davidw 12y ago
      Cool site, and I like their methodology: http://db-engines.com/en/ranking_definition http://db-engines.com/en/ranking_definition It's kind of how I did http://langpop.com/ http://langpop.com/ back in the day (subsequently sold and it looks like they don't maintain it).
  - qwerta 12y ago
    Reputation is hard to fix. Mysql today is still considered 'toy' db, even trough it is very stable and has good features. I think they lost industries with most money (financial and enterprise) to Cassandra and other competitors. Plus there are now solutions with aim to reimplement MongoDB API and be drop in replacement.
    - dragonwriter 12y ago
      > Mysql today is still considered 'toy' db, even trough it is very stable and has good features. I'd actually say that its feature set relative to other equally stable open-source DBs has gotten worse over the years, not better. Not because its gotten absolutely worse, but because other open source databases have gotten better faster.
      - qwerta 12y ago
        Mysql has good selection of engines and good clustering.
  - jandrewrogers 12y ago
    MongoDB had a ton of momentum a few years ago but I think the argument being made is that it lost most of it due to the growing perception that it had fundamental design defects. Just anecdotally, a large number of companies I deal with had MongoDB at one time but I can't think of a one that uses it today. It was not replaced with a single thing but it was replaced for a variety of reasons. As a long time observer of the database market with no skin in this particular game, in terms of regaining market momentum it is difficult to get past the reputation of being technically defective once that perception is broadly in place. To make matters worse, needing to completely replace the storage engine with a proper design will be viewed by some as an admission of that rather than an upgrade per se. It may be too little, too late for its long term prospects. By contrast, PostgreSQL is doing great right now; just about every company I talk to is using it somewhere. Putting on my database engine designer hat, MongoDB did need to start from scratch with that storage engine; the original one was a naive design with a lot of problems. The sharding/replication design is still quite dodgy, which will hinder practical scalability. A new storage engine is a start but not enough if MongoDB wants to regain their momentum.
fleitz 12y ago
BDB... seriously? At this point why not just: apt-get install postgres psql < "CREATE TABLE mongodb (key varchar[1024] NOT NULL PRIMARY KEY,value jsonb);" then you get the best of both worlds.
- eknkc 12y ago
  Easy replication & sharding is appealing.
  - eddd 12y ago
    And replication for postgres is not easy? Sharding in postgres requires some plproxy/pgbouncer knowledge - but they don't cause as many problems in the future as mongodb. Me personally, when it comes to databases i go for the reliable first - then i can play with scalability.
    - remon 12y ago
      That almost certainly implies you're not working on anything that requires horizontal scalability. That's a luxury not everyone can afford. Scalability, in almost all cases that actually require it, is the first priority. It's like saying it's more important that the car drives reliably rather than everyone that needs to travel fitting into the car in the first place.
      - davidw 12y ago
        I think there are probably way too many people out there doing scaling before they're doing "getting things right". It's premature optimization that causes a lot of problems. Naturally, there are plenty of people who do need to scale a lot, too, but presumably by the time they need to, they have a good idea of the problem and exactly how best to scale it.
        remon 12y ago
        That may be generally true but those two things are not mutually exclusive. Scalability doesn't come at the expense of getting things right, it just might come at the expense of certain features which applies to most distributed system problems.
dschiptsov 12y ago
But the old one is crap, right?
wiremine 12y ago
I'm not a huge MongoDB fan, but I think it is great to see competition and evolution in this space. From the article: "In addition to compression and record-level locking, WiredTiger also gives MongoDB multi-version concurrency control (MVCC), multi-document transactions, and support for log-structured merge-trees, or LSM trees, for very high insert workloads" I'd also love to see them add document joins at some point. That's a a big differentiating feature for RethinkDB, although I've found Rethink's syntax to be difficult to reason through.
- remon 12y ago
  Why do you consider joins a big differentiating feature for NoSQL storage solutions?
  - tinco 12y ago
    I can't speak for everyone, but I recently started on a project I chose RethinkDB for because of the support for joins. It's very nice to store documents, but it's a pain to have to resort to an SQL database every time you have relational data, or digress and issue multiple queries to patch in your relations. Just basic join support allows you to have both documents and relational data in the same database, made the sell for me anyway.
  - personZ 12y ago
    Most real world data is some mix of relational and unique / big bag of properties. The best solution is often somewhere in between (which is, for me, probably pgsql and its support of JSON, though I've achieved the same versatility with xml data types).
jitix 12y ago
I have been waiting for this since a long time. Having implemented an OLTP application using MongoDB (luckily their write loads were low), I stopped using Mongo 2 years back because of the write bottleneck (it was still good for OLAP though). Good decision by the MongoDB team since they already have the market traction.
t1m 12y ago
Here is a link to some more 'thorough' benchmarks that compare WiredTiger to many other database back-ends. http://symas.com/mdb/inmem/ http://symas.com/mdb/inmem/
- hyc_symas 12y ago
  I'd say the MongoDB folks made a pretty safe choice here, since WiredTiger offers both LSM and Btree engines. I've recently completed some more benchmarks on all of the above-tested engines, and will be publishing the report soon (as soon as it's done being written...). Testing on-disk, with DBs 5x larger than RAM, at data sizes of 24, 96, 384, 768, 2000, and 4000 bytes per record. The important result to note is that LSM's high write speed advantage only exists for small record sizes, and for short durations. On long duration tests the compaction overhead stops them in their tracks, and at data sizes above 2000 bytes per record, LSM write amplification is too expensive (regardless of test duration). I.e., above 2000 byte records, Btrees are always faster than LSMs. This result is particularly relevant for MongoDB, since a document store tends to have large records containing multiple fields, as opposed to individual values being stored as separate small records. And in cases where that applies (large records), nothing comes anywhere close to LMDB's performance. (A preview of the on-disk test results I mentioned was presented at BuildStuff.LT last week http://symas.com/mdb/20141120-BuildStuff-Lightning.pdf http://symas.com/mdb/20141120-BuildStuff-Lightning.pdf pages 103-on)
  - eis 12y ago
    Howard I always found your benchmarks as well as your comments and explanations of the results to be very thorough and easy to understand. So thanks for that. I wonder if you had a look at ForestDB which is also based on a B+-tree like structure instead of LSM and claims to have very low write amplification. They have published benchmarks[1] but there is no LMDB or WiredTiger included. [1] https://github.com/couchbaselabs/forestdb/wiki/Performance-Results https://github.com/couchbaselabs/forestdb/wiki/Performance-R...
    - hyc_symas 12y ago
      I attempted to test it a while ago; the test driver is included along with all the others https://github.com/hyc/leveldb/tree/benches/doc/bench https://github.com/hyc/leveldb/tree/benches/doc/bench It crashed in all of my test invocations though, so I haven't looked at it since.

71 points by lucasjans 12y ago

qwerta 12y ago

I am glad they finally introduced LSM trees and per document locking. But I think thats too little too late :-)

samneilz 12y ago
Agreed
remon 12y ago
Why on earth would it be too late? MongoDB is comfortably in the top 3 in terms of traction and they just fixed what many consider the primary technical argument against using it. Too late for what?
- davidw 12y ago
  Here's my guess about some of the most utilized/popular databases: * SQLite - it's everywhere thanks to mobile phones. * Oracle. * Microsoft's thing has to have a big following. * Mysql. * DB2 is not as big as Oracle, but IBM isn't some lightweight, either. MongoDB gets talked about a lot, and probably used among people trying new stuff out, but I think a lot of mostly silent people wouldn't touch something like that for their important data with a 10 foot pole.
  - bane 12y ago
    > SQLite - it's everywhere thanks to mobile phones. and browsers...and tons of other pieces of software
  - arethuza 12y ago
    "Microsoft's thing" SQL Server is definitely one of Microsoft's best products - as well as being an excellent database engine it also is used by SharePoint, the various Dynamics applications (CRM & ERP) as well as being the "natural" choice for application development in .Net If I had to guess I would suspect that, at least in terms of revenue, SQL Server probably tops Oracle and DB2.
  - dkhenry 12y ago
    I think you overestimate the reach of Oracle. They are no where outside fortuine 500 and thats not a lot of deployments. MySQL is most likely the most deployed thanks to the LAMP stack's proliferation across all sorts of small and medium sites. After that it really would be a toss up between MS-SQL who has traction in the SMB space due to it being used for sharepoint and exchange, and PostgreSQL but lets be honest the most utilized database is Access.
    - dagw 12y ago
      I do a lot of work with geo-spatial analysis and often work with with various counties and municipalities in my experience everybody has an Oracle database stuffed away somewhere storing the regions canonical geo-spatial data.
      - dkhenry 12y ago
        Even if each town had an oracle database squirreled away ( which they don't ) That still would not even come close to just the wordpress deployments (60 Million).
    - collyw 12y ago
      Sadly many more people use Excel as their database when they should be using Access.
- johngd 12y ago
  I am generally curious about the source for your claim. Would you mind sharing, please?
- jeltz 12y ago
  MongoDB is placed 5th at DB-Engines, and I would suspect SQLite should actually be ranked higher since it used virtually everywhere. http://db-engines.com/en/ranking http://db-engines.com/en/ranking
  - davidw 12y ago
    Cool site, and I like their methodology: http://db-engines.com/en/ranking_definition http://db-engines.com/en/ranking_definition It's kind of how I did http://langpop.com/ http://langpop.com/ back in the day (subsequently sold and it looks like they don't maintain it).
- qwerta 12y ago
  Reputation is hard to fix. Mysql today is still considered 'toy' db, even trough it is very stable and has good features. I think they lost industries with most money (financial and enterprise) to Cassandra and other competitors. Plus there are now solutions with aim to reimplement MongoDB API and be drop in replacement.
  - dragonwriter 12y ago
    > Mysql today is still considered 'toy' db, even trough it is very stable and has good features. I'd actually say that its feature set relative to other equally stable open-source DBs has gotten worse over the years, not better. Not because its gotten absolutely worse, but because other open source databases have gotten better faster.
    - qwerta 12y ago
      Mysql has good selection of engines and good clustering.
- jandrewrogers 12y ago
  MongoDB had a ton of momentum a few years ago but I think the argument being made is that it lost most of it due to the growing perception that it had fundamental design defects. Just anecdotally, a large number of companies I deal with had MongoDB at one time but I can't think of a one that uses it today. It was not replaced with a single thing but it was replaced for a variety of reasons. As a long time observer of the database market with no skin in this particular game, in terms of regaining market momentum it is difficult to get past the reputation of being technically defective once that perception is broadly in place. To make matters worse, needing to completely replace the storage engine with a proper design will be viewed by some as an admission of that rather than an upgrade per se. It may be too little, too late for its long term prospects. By contrast, PostgreSQL is doing great right now; just about every company I talk to is using it somewhere. Putting on my database engine designer hat, MongoDB did need to start from scratch with that storage engine; the original one was a naive design with a lot of problems. The sharding/replication design is still quite dodgy, which will hinder practical scalability. A new storage engine is a start but not enough if MongoDB wants to regain their momentum.

fleitz 12y ago

BDB... seriously? At this point why not just: apt-get install postgres psql < "CREATE TABLE mongodb (key varchar[1024] NOT NULL PRIMARY KEY,value jsonb);" then you get the best of both worlds.

eknkc 12y ago
Easy replication & sharding is appealing.
- eddd 12y ago
  And replication for postgres is not easy? Sharding in postgres requires some plproxy/pgbouncer knowledge - but they don't cause as many problems in the future as mongodb. Me personally, when it comes to databases i go for the reliable first - then i can play with scalability.
  - remon 12y ago
    That almost certainly implies you're not working on anything that requires horizontal scalability. That's a luxury not everyone can afford. Scalability, in almost all cases that actually require it, is the first priority. It's like saying it's more important that the car drives reliably rather than everyone that needs to travel fitting into the car in the first place.
    - davidw 12y ago
      I think there are probably way too many people out there doing scaling before they're doing "getting things right". It's premature optimization that causes a lot of problems. Naturally, there are plenty of people who do need to scale a lot, too, but presumably by the time they need to, they have a good idea of the problem and exactly how best to scale it.
      - remon 12y ago
        That may be generally true but those two things are not mutually exclusive. Scalability doesn't come at the expense of getting things right, it just might come at the expense of certain features which applies to most distributed system problems.

dschiptsov 12y ago

But the old one is crap, right?

wiremine 12y ago

I'm not a huge MongoDB fan, but I think it is great to see competition and evolution in this space. From the article: "In addition to compression and record-level locking, WiredTiger also gives MongoDB multi-version concurrency control (MVCC), multi-document transactions, and support for log-structured merge-trees, or LSM trees, for very high insert workloads" I'd also love to see them add document joins at some point. That's a a big differentiating feature for RethinkDB, although I've found Rethink's syntax to be difficult to reason through.

remon 12y ago
Why do you consider joins a big differentiating feature for NoSQL storage solutions?
- tinco 12y ago
  I can't speak for everyone, but I recently started on a project I chose RethinkDB for because of the support for joins. It's very nice to store documents, but it's a pain to have to resort to an SQL database every time you have relational data, or digress and issue multiple queries to patch in your relations. Just basic join support allows you to have both documents and relational data in the same database, made the sell for me anyway.
- personZ 12y ago
  Most real world data is some mix of relational and unique / big bag of properties. The best solution is often somewhere in between (which is, for me, probably pgsql and its support of JSON, though I've achieved the same versatility with xml data types).

jitix 12y ago

I have been waiting for this since a long time. Having implemented an OLTP application using MongoDB (luckily their write loads were low), I stopped using Mongo 2 years back because of the write bottleneck (it was still good for OLAP though). Good decision by the MongoDB team since they already have the market traction.

t1m 12y ago

Here is a link to some more 'thorough' benchmarks that compare WiredTiger to many other database back-ends. http://symas.com/mdb/inmem/ http://symas.com/mdb/inmem/

hyc_symas 12y ago
I'd say the MongoDB folks made a pretty safe choice here, since WiredTiger offers both LSM and Btree engines. I've recently completed some more benchmarks on all of the above-tested engines, and will be publishing the report soon (as soon as it's done being written...). Testing on-disk, with DBs 5x larger than RAM, at data sizes of 24, 96, 384, 768, 2000, and 4000 bytes per record. The important result to note is that LSM's high write speed advantage only exists for small record sizes, and for short durations. On long duration tests the compaction overhead stops them in their tracks, and at data sizes above 2000 bytes per record, LSM write amplification is too expensive (regardless of test duration). I.e., above 2000 byte records, Btrees are always faster than LSMs. This result is particularly relevant for MongoDB, since a document store tends to have large records containing multiple fields, as opposed to individual values being stored as separate small records. And in cases where that applies (large records), nothing comes anywhere close to LMDB's performance. (A preview of the on-disk test results I mentioned was presented at BuildStuff.LT last week http://symas.com/mdb/20141120-BuildStuff-Lightning.pdf http://symas.com/mdb/20141120-BuildStuff-Lightning.pdf pages 103-on)
- eis 12y ago
  Howard I always found your benchmarks as well as your comments and explanations of the results to be very thorough and easy to understand. So thanks for that. I wonder if you had a look at ForestDB which is also based on a B+-tree like structure instead of LSM and claims to have very low write amplification. They have published benchmarks[1] but there is no LMDB or WiredTiger included. [1] https://github.com/couchbaselabs/forestdb/wiki/Performance-Results https://github.com/couchbaselabs/forestdb/wiki/Performance-R...
  - hyc_symas 12y ago
    I attempted to test it a while ago; the test driver is included along with all the others https://github.com/hyc/leveldb/tree/benches/doc/bench https://github.com/hyc/leveldb/tree/benches/doc/bench It crashed in all of my test invocations though, so I haven't looked at it since.