3 ms·

A few comments on the problems: * CPU bottleneck. The mongod is no usually bound by CPU except for building indexes on existing data (which shouldn't really ha

by dmytton 13y ago

A few comments on the problems: * CPU bottleneck. The mongod is no usually bound by CPU except for building indexes on existing data (which shouldn't really happen in production). The issue he's talking about is contention between the web server (or workers) and the mongos. This isn't anything unexpected. It's recommended to put the mongos onto the application server and then you scale this by adding CPUs initially but then by adding multiple application nodes and using a load balancer. Or perhaps splitting the mongos onto dedicated nodes? * Virtualisation: VMs are notorious for having variable performance because you have the overhead of the hypervisor but more importantly, are sharing resources with others. We run performance critical apps on dedicated servers and reserve VMs for tools or things which aren't high throughput e.g. a MongoDB arbiter. * EBS: This has had known performance issues for years. It's fine as a basic file store but should never be used for databases. PIOPs are the way around this but local instance storage is also an option. * No transactions: MongoDB has never had them. This is known. * Schemaless != no schema design. It makes it easy to play around but you still need to think through things carefully. See http://blog.serverdensity.com/mongodb-schema-design-pitfalls/ http://blog.serverdensity.com/mongodb-schema-design-pitfalls... * No joins. Again, it has never had joins. This is known.

ericcholis 13y ago
Agreed on all counts. Sounds like he is complaining more about issues with VM performance than MongoDB performance on VMs. The joins statement kills me. If somehow the database design requires joins, Mongo is fast enough to run two queries and then let you work with them in code.
- thedufer 13y ago
  > If somehow the database design requires joins, Mongo is fast enough to run two queries and then let you work with them in code. That's definitely not true. What if you were planning on filtering after the join? You may find yourself pulling millions of records. The bandwidth alone would bring you down. I work with MongoDB, and once in awhile I really miss joins. You can't emulate joins in any reasonable amount of time in Mongo.
  - Xorlev 13y ago
    +1 joins are efficiently done by databases, not by applications. At least in any reasonable dataset. Joining in application works for a few thousand records.
    - pkolaczk 13y ago
      Depends on what you are joining. Selecting five rows from one (sharded) table and the looking up matching five rows from another (sharded) table by primary key is almost just as fast from the application as it would be done by the database itself. Additionally many NoSQL databases let you store arbitrary number of data in a row, so you don't need (big) intermediate join tables. However, I agree that a database can be much faster when you want to join two huge datasets, because then it has all the fancy ways of fast joining with sort-merge-join or hash-join, etc. But if you need to join huge datasets, you're screwed anyway, because such joins don't scale at all, and are still really, really slow - they need to do at least one sequential scan (and typically 3) over each full dataset. Definitely not something you want to do in your OLTP app. I saw an OLTP app where joining a few thousand rows (yeah, thousand, not million!) killed the app performance totally, up to the point that a single user using the app had to wait >10 seconds for a page refresh. The app was probably done by someone thinking that joins are free.
  - k_bx 13y ago
    > That's definitely not true. What if you were planning on filtering after the join? You may find yourself pulling millions of records. The bandwidth alone would bring you down. I work with MongoDB, and once in awhile I really miss joins. You can't emulate joins in any reasonable amount of time in Mongo. I agree that joins should be done by mongo (as a feature, similar to RethinkDB), but they can get you in trouble when you convert your collection from local-to-joined-one into sharded-into-multiple-servers. You'd then gain huge network load that can potentially bring your system down. If you decided that data in some particular collection is potentially huge -- just don't use joins on it (unless REALLY once in a while).
  - pkolaczk 13y ago
    "What if you were planning on filtering after the join? You may find yourself pulling millions of records." Then you always filter before doing the join. Problem solved.
jeffdavis 13y ago
"[EBS is] fine as a basic file store but should never be used for databases." Doesn't Heroku use EBS for all of their postgres databases? It may be the case that postgres works better on EBS than mongo does. Postgres has a traditional write-ahead log that minimizes (and spreads out) block writes and hides latencies. Mongo does not.
- jbellis 13y ago
  WAL only helps so much, since you also need seeks for reads. Cassandra has a WAL + log-structured storage + no read-before-update design, so it basically eliminates seeks on writes entirely, and EBS is still ass for workloads that don't fit in cache. Which, if you're bothering to use C*, is almost all of them.
  - jeffdavis 13y ago
    That's ideal, but many kinds of updates require some kind of read. I agree in general though.
davidw 13y ago
> PIOPs are the way around this but local instance storage is also an option. PIOP is not ringing a bell - I'll stick my neck out and ask what it is.
- zerd 13y ago
  Amazon EBS Provisioned IOPS. Standard EBS gives around 100 IOPS (Input/Output Operations Per Second). With PIOP you can get more if you pay more.
nasalgoat 13y ago
There are a number of very large bottlenecks in MongoDB, and one of the worst is the mongos process. I can't run more than 100 processes on a server talking to a mongos process, because any more than that, mongos uses 70% of the CPU power on a 32-core box. 70% of all CPU for database transaction overhead! After that, it simply stops working. The next one is the insane 20K connection limit per mongod process that is hardcoded into the binary. When asked about it, 10gen says they just decided years ago that that was the limit because no one would ever have more than 20GB in a server. So, I can't just run more mongos processes on other boxes - it hits the 20K limit. I could go on but the point is, MongoDB is really badly designed.

by dmytton 13y ago

ericcholis 13y ago

Agreed on all counts. Sounds like he is complaining more about issues with VM performance than MongoDB performance on VMs. The joins statement kills me. If somehow the database design requires joins, Mongo is fast enough to run two queries and then let you work with them in code.

thedufer 13y ago
> If somehow the database design requires joins, Mongo is fast enough to run two queries and then let you work with them in code. That's definitely not true. What if you were planning on filtering after the join? You may find yourself pulling millions of records. The bandwidth alone would bring you down. I work with MongoDB, and once in awhile I really miss joins. You can't emulate joins in any reasonable amount of time in Mongo.
- Xorlev 13y ago
  +1 joins are efficiently done by databases, not by applications. At least in any reasonable dataset. Joining in application works for a few thousand records.
  - pkolaczk 13y ago
    Depends on what you are joining. Selecting five rows from one (sharded) table and the looking up matching five rows from another (sharded) table by primary key is almost just as fast from the application as it would be done by the database itself. Additionally many NoSQL databases let you store arbitrary number of data in a row, so you don't need (big) intermediate join tables. However, I agree that a database can be much faster when you want to join two huge datasets, because then it has all the fancy ways of fast joining with sort-merge-join or hash-join, etc. But if you need to join huge datasets, you're screwed anyway, because such joins don't scale at all, and are still really, really slow - they need to do at least one sequential scan (and typically 3) over each full dataset. Definitely not something you want to do in your OLTP app. I saw an OLTP app where joining a few thousand rows (yeah, thousand, not million!) killed the app performance totally, up to the point that a single user using the app had to wait >10 seconds for a page refresh. The app was probably done by someone thinking that joins are free.
- k_bx 13y ago
  > That's definitely not true. What if you were planning on filtering after the join? You may find yourself pulling millions of records. The bandwidth alone would bring you down. I work with MongoDB, and once in awhile I really miss joins. You can't emulate joins in any reasonable amount of time in Mongo. I agree that joins should be done by mongo (as a feature, similar to RethinkDB), but they can get you in trouble when you convert your collection from local-to-joined-one into sharded-into-multiple-servers. You'd then gain huge network load that can potentially bring your system down. If you decided that data in some particular collection is potentially huge -- just don't use joins on it (unless REALLY once in a while).
- pkolaczk 13y ago
  "What if you were planning on filtering after the join? You may find yourself pulling millions of records." Then you always filter before doing the join. Problem solved.

jeffdavis 13y ago

"[EBS is] fine as a basic file store but should never be used for databases." Doesn't Heroku use EBS for all of their postgres databases? It may be the case that postgres works better on EBS than mongo does. Postgres has a traditional write-ahead log that minimizes (and spreads out) block writes and hides latencies. Mongo does not.

jbellis 13y ago
WAL only helps so much, since you also need seeks for reads. Cassandra has a WAL + log-structured storage + no read-before-update design, so it basically eliminates seeks on writes entirely, and EBS is still ass for workloads that don't fit in cache. Which, if you're bothering to use C*, is almost all of them.
- jeffdavis 13y ago
  That's ideal, but many kinds of updates require some kind of read. I agree in general though.

davidw 13y ago

> PIOPs are the way around this but local instance storage is also an option. PIOP is not ringing a bell - I'll stick my neck out and ask what it is.

zerd 13y ago
Amazon EBS Provisioned IOPS. Standard EBS gives around 100 IOPS (Input/Output Operations Per Second). With PIOP you can get more if you pay more.

nasalgoat 13y ago

There are a number of very large bottlenecks in MongoDB, and one of the worst is the mongos process. I can't run more than 100 processes on a server talking to a mongos process, because any more than that, mongos uses 70% of the CPU power on a 32-core box. 70% of all CPU for database transaction overhead! After that, it simply stops working. The next one is the insane 20K connection limit per mongod process that is hardcoded into the binary. When asked about it, 10gen says they just decided years ago that that was the limit because no one would ever have more than 20GB in a server. So, I can't just run more mongos processes on other boxes - it hits the 20K limit. I could go on but the point is, MongoDB is really badly designed.