6 ms·

While the details are very interesting, there are still many questions to be answered (on both sides): - how difficult would be to bring up read-only replicas?

While the details are very interesting, there are still many questions to be answered (on both sides): - how difficult would be to bring up read-only replicas? (hopefully that should take much less than 11 hours + 6hours) - why the 3rd shard could accommodate only 5% of the data? - how can you plan capacity when using the "wrong" sharding? (basically leading to unpredictable distributions) I have posted the rest of the questions here: http://nosql.mypopescu.com/post/1265191137/foursquare-mongodb-outage-post-mortem http://nosql.mypopescu.com/post/1265191137/foursquare-mongod... as I hope to get some more answers.

harryh 16y ago
> how difficult would be to bring up read-only replicas? > (hopefully that should take much less than 11 hours > 6hours) Bringing up read only replicas would have been easy, but our appservers are not currently designed to read data from multiple replicas so it wouldn't have helped. We hope to make architectural changes to allow for this sort of thing in the future but aren't there yet. > - why the 3rd shard could accommodate only 5% of the data? The issue wasn't the amount of data the 3rd shard could accomodate, but the rate at which data could be transfered off the 1st (overloaded) shard onto the 3rd shard. > how can you plan capacity when using the "wrong" > sharding? (basically leading to unpredictable > distributions) We weren't really using the "wrong" sharding. And even the uneven distribution we saw (about 60%/40%) wasn't totally horrible. Not 100% understanding your question here.
- alexpopescu 16y ago
  I didn't mean "wrong" in the full sense of the word. What I actually mean is that sharding based on user id will most probably not give/guarantee predictable distributions of the writes generated by users (basically there can be no guarantee that chunk#1 of users will always generate the aprox. same amount of data as chunk#2 of users).
- dirkgadsden 16y ago
  What exactly is the rate like for transferring data across MongoDB instances on EBS? Was the overhead of going across EBS volumes the major factor in the length of the downtime? The 60%/40% sharding distribution doesn't seem too bad in the big picture; are there any plans to be evening that out?

harryh 16y ago

> how difficult would be to bring up read-only replicas? > (hopefully that should take much less than 11 hours > 6hours) Bringing up read only replicas would have been easy, but our appservers are not currently designed to read data from multiple replicas so it wouldn't have helped. We hope to make architectural changes to allow for this sort of thing in the future but aren't there yet. > - why the 3rd shard could accommodate only 5% of the data? The issue wasn't the amount of data the 3rd shard could accomodate, but the rate at which data could be transfered off the 1st (overloaded) shard onto the 3rd shard. > how can you plan capacity when using the "wrong" > sharding? (basically leading to unpredictable > distributions) We weren't really using the "wrong" sharding. And even the uneven distribution we saw (about 60%/40%) wasn't totally horrible. Not 100% understanding your question here.

alexpopescu 16y ago
I didn't mean "wrong" in the full sense of the word. What I actually mean is that sharding based on user id will most probably not give/guarantee predictable distributions of the writes generated by users (basically there can be no guarantee that chunk#1 of users will always generate the aprox. same amount of data as chunk#2 of users).
dirkgadsden 16y ago
What exactly is the rate like for transferring data across MongoDB instances on EBS? Was the overhead of going across EBS volumes the major factor in the length of the downtime? The 60%/40% sharding distribution doesn't seem too bad in the big picture; are there any plans to be evening that out?