28 ms·

It seems that the main criticisms are: a) Lack of online compaction (causing data set to be much bigger than necessary) b) Broken auto sharding (complicating

It seems that the main criticisms are: a) Lack of online compaction (causing data set to be much bigger than necessary) b) Broken auto sharding (complicating horizontal scaling) c) "Scary" durability and scaling stories with auto sharding Looks like auto-sharding has had a lot of work done, while online compaction still isn't there. But in particular I'm curious about (c). What exactly is so scary durability wise with auto-sharding? Is it just the fact that it didn't work in 1.5/1.6?

schmichael 15y ago
"But in particular I'm curious about (c). What exactly is so scary durability wise with auto-sharding? Is it just the fact that it didn't work in 1.5/1.6?" Yes. During the 1.5 dev cycle we ran out of time. Our main database server couldn't vertically scale any further, so we had to do something. Despite 1.6.0 being released before our migration finishing, auto-sharding still didn't seem to be anywhere near the stability we needed. I've heard some horror stories regarding auto-sharding & replica sets (in 1.8) since then that sadly I'm not at liberty to share. I think you missed a number of finer points as well: * We were not guided to make the best schema decisions initially, and migrating schemas is tricky and manual. * Some schema decisions exposed us to short-comings in MongoDB (the double updating issue). * The global write lock is severely limiting. * It's unsafe by default (drivers default to not checking for errors when writing, among other things) * The ops load is greater for auto-sharding+replica-sets than for manually sharded master/slave setups (granted the former has features the latter does not). Perhaps the most frustrating part was the constant unfortunate surprises. There are stories that didn't even make it into the slides. Many you could blame on us. Some we could blame on EC2. However, strong benefits to using MongoDB over PgSQL never materialized, so we opted to switch to a system with far fewer unknowns for our fairly modest needs. If you think fully utilizing MongoDB's architecture will give your business a competitive advantage: go nuts. That was far from our experience.
- thadeus_venture 15y ago
  Thank you, that's very interesting. "Going nuts" with MongoDB is exactly what I'm contemplating right now at my start up, and this is something I haven't quite heard before. Is there more info on the double update issue anywhere? It sounds like a really bad race condition.

schmichael 15y ago

"But in particular I'm curious about (c). What exactly is so scary durability wise with auto-sharding? Is it just the fact that it didn't work in 1.5/1.6?" Yes. During the 1.5 dev cycle we ran out of time. Our main database server couldn't vertically scale any further, so we had to do something. Despite 1.6.0 being released before our migration finishing, auto-sharding still didn't seem to be anywhere near the stability we needed. I've heard some horror stories regarding auto-sharding & replica sets (in 1.8) since then that sadly I'm not at liberty to share. I think you missed a number of finer points as well: * We were not guided to make the best schema decisions initially, and migrating schemas is tricky and manual. * Some schema decisions exposed us to short-comings in MongoDB (the double updating issue). * The global write lock is severely limiting. * It's unsafe by default (drivers default to not checking for errors when writing, among other things) * The ops load is greater for auto-sharding+replica-sets than for manually sharded master/slave setups (granted the former has features the latter does not). Perhaps the most frustrating part was the constant unfortunate surprises. There are stories that didn't even make it into the slides. Many you could blame on us. Some we could blame on EC2. However, strong benefits to using MongoDB over PgSQL never materialized, so we opted to switch to a system with far fewer unknowns for our fairly modest needs. If you think fully utilizing MongoDB's architecture will give your business a competitive advantage: go nuts. That was far from our experience.

thadeus_venture 15y ago
Thank you, that's very interesting. "Going nuts" with MongoDB is exactly what I'm contemplating right now at my start up, and this is something I haven't quite heard before. Is there more info on the double update issue anywhere? It sounds like a really bad race condition.