4 ms·

I'm not trying to critique the company's engineering, but the workloads and challenges they are facing are not really high in QPS. The company says they get abo

by diziet 10y ago

I'm not trying to critique the company's engineering, but the workloads and challenges they are facing are not really high in QPS. The company says they get about 600 QPS. These are database QPS, not even service QPS! Now no matter what kind of operations you're doing, correct query and index planning should serve you ok. Looking at the graph, there are a lot of updates going on, so I would say GridFS is out of the question. MongoDB WT (and 3.2 especially) really moved the Mongo to the next level, and it's really weird to hear that it took so long to upgrade to WT. We moved a system to WT from MMAPv1 in the early 3.0 days (and to 3.2 later) and saw improvements in read/write 99.9th percentile latency, general ROI on instance utilization, QPS, and especially snappy compression bringing an X0 TB deployment to more reasonable scale as far as storage goes. This is not a crazy system, but peaks of 100k while jobs calculate and constant load of 50k QPS is very straightforward and WT responds beautifully to. As mentioned before, this is a bug that got a number of fixes upstream (though 3.2.10 came out just only yesterday), but one of the only reasonable way to hit that load was to insert/update a lot of documents on a very slow underlying disk. Test your workloads before upgrading -- but there is something severely wrong going on with your deployment/query planning if 600 QPS per server is bringing cache issues, especially given r3.4xlarge underlying hardware.

spimmy 10y ago
I have plenty of rants about the company's engineering, priorities, etc. I just hate people saying dumb shit. :) Workload QPS is a meaningless number unless you know what the workload is. At Parse, we used RocksDB (a LSM tree write optimized storage engine) to get something stupid like 9x faster writes. I don't know where you're seeing 600 QPS ... (we got 100x that if we were optimizing for writes) ... but it still doesn't mean anything unless you know the size of the document, whether it's inserts, queries, updates, whether it's hot spotting on a row, whether the lock is at the collection, document, database, or global, etc. If you're going to switch storage engines, you have to use some sort of capture replay technology or you can not have any confidence in the change. Use what we wrote at Parse: https://github.com/ParsePlatform/flashback https://github.com/ParsePlatform/flashback. We spent ~6 months teasing out bugs in rocksdb, mongodb, the 2.6 query planner rewrite, etc before we even started to make the switch. If you don't do that, you don't get to complain about the results. Sorry.
- diziet 10y ago
  Spimmy, I am not sure about the context of what you're saying though I agree with the context and feedback of your comments, but I am basing the 600 QPS based on this comment on the blog post linked in the article: >To put things in context, we do approximately ~18.07K operations/minute on one primary server. We have two shards in the cluster so two primary server making it ~36.14K operations/minute. 36,140 / 60 is roughly 600. Confusingly, the opcounter screenshot at the top of the blog shows something close to 3k updates / second with a peak of about 4.5k, though it is not explained. I left a more detailed comment on the clevertap blog regarding the kind of operations they are using. When you talk about benchmarking and high level metrics like QPS, there is a level of thresholds where you can say "you're most likely doing something wrong as you should not be hitting cache limit issues". The reason you will move to WT is because of snappy, generally higher 99.9th percentile performance, higher throughput, much better document based locks, etc. Replaying real data is wonderful and great for preparing for a migration like that. MMAPv1 to WT was a much smoother transition for us than what I imagine MMAPv1 to Rocks would be, though the team at FB including Mark Callaghan has made incredible progress on it.
  - spimmy 10y ago
    I'm agreeing with your comment, not arguing with it. The problems we had moving from mmapv1 to rocksdb were almost entirely based on the rewrite of the query planner in 2.6, not the storage engine itself. The WT storage engine is far less mature than RocksDB. Only a very tiny amount of code in the Mongo+Rocks integration is actually new code, it's just the thin layer of glue between mongo and the storage engine API. It was a tricky migration because we made the decision to do it early while so much code was still new in mongo for that version. We wanted to help give feedback on the storage engine API because we had pushed so hard for that feature, and because we were pushing the boundaries of what the mmap engine could ever reasonably be expected to do. (Side note: OMG, anyone who measures their request rate in anything other than per-second is putting up a giant red flag that they don't know what they're doing.)
    - diziet 10y ago

by diziet 10y ago

spimmy 10y ago

I have plenty of rants about the company's engineering, priorities, etc. I just hate people saying dumb shit. :) Workload QPS is a meaningless number unless you know what the workload is. At Parse, we used RocksDB (a LSM tree write optimized storage engine) to get something stupid like 9x faster writes. I don't know where you're seeing 600 QPS ... (we got 100x that if we were optimizing for writes) ... but it still doesn't mean anything unless you know the size of the document, whether it's inserts, queries, updates, whether it's hot spotting on a row, whether the lock is at the collection, document, database, or global, etc. If you're going to switch storage engines, you have to use some sort of capture replay technology or you can not have any confidence in the change. Use what we wrote at Parse: https://github.com/ParsePlatform/flashback https://github.com/ParsePlatform/flashback. We spent ~6 months teasing out bugs in rocksdb, mongodb, the 2.6 query planner rewrite, etc before we even started to make the switch. If you don't do that, you don't get to complain about the results. Sorry.

diziet 10y ago
Spimmy, I am not sure about the context of what you're saying though I agree with the context and feedback of your comments, but I am basing the 600 QPS based on this comment on the blog post linked in the article: >To put things in context, we do approximately ~18.07K operations/minute on one primary server. We have two shards in the cluster so two primary server making it ~36.14K operations/minute. 36,140 / 60 is roughly 600. Confusingly, the opcounter screenshot at the top of the blog shows something close to 3k updates / second with a peak of about 4.5k, though it is not explained. I left a more detailed comment on the clevertap blog regarding the kind of operations they are using. When you talk about benchmarking and high level metrics like QPS, there is a level of thresholds where you can say "you're most likely doing something wrong as you should not be hitting cache limit issues". The reason you will move to WT is because of snappy, generally higher 99.9th percentile performance, higher throughput, much better document based locks, etc. Replaying real data is wonderful and great for preparing for a migration like that. MMAPv1 to WT was a much smoother transition for us than what I imagine MMAPv1 to Rocks would be, though the team at FB including Mark Callaghan has made incredible progress on it.
- spimmy 10y ago
  I'm agreeing with your comment, not arguing with it. The problems we had moving from mmapv1 to rocksdb were almost entirely based on the rewrite of the query planner in 2.6, not the storage engine itself. The WT storage engine is far less mature than RocksDB. Only a very tiny amount of code in the Mongo+Rocks integration is actually new code, it's just the thin layer of glue between mongo and the storage engine API. It was a tricky migration because we made the decision to do it early while so much code was still new in mongo for that version. We wanted to help give feedback on the storage engine API because we had pushed so hard for that feature, and because we were pushing the boundaries of what the mmap engine could ever reasonably be expected to do. (Side note: OMG, anyone who measures their request rate in anything other than per-second is putting up a giant red flag that they don't know what they're doing.)
  - diziet 10y ago