4 ms·
I would have used sharding, and then normal failure handling and resynchronisation between nodes. If you have the requirement that two adjacent versions have t
by jpollock 14y ago
I would have used sharding, and then normal failure handling and resynchronisation between nodes. If you have the requirement that two adjacent versions have to be able to resynchronise after failure upgrades (and rollback) become equivalent to normal node failure.
As it stands, during your upgrade you've lost all of your fault tolerance and can't meet your performance requirements - you've gone from 5 nodes able to process the traffic to 1!
- msherry 14y agoYou make a good point re: fault tolerance. As it happens, I've simplified the diagram quite a bit to make it simpler to visualise. We have more than 2 Nginx instances, and more than one shard of our app was running the new code.
- jpollock 14y agoIt sounds like a shard collapsed down to a single database instance? Otherwise, you wouldn't have had to turn off requests to the shard, you could have quiesced one of the nodes, taken it out of service and brought it back up with the new release?
- jpollock 14y agoI'm sounding really negative, I apologize. :) I love that you've got your upgrades to 15s, that's a powerful place to be!