4 ms·
This is a great write up - making these public with all the gory technical details helps us all be better at our jobs. I’m curious whether shutting down the ea
by itsdrewmiller 8y ago
This is a great write up - making these public with all the gory technical details helps us all be better at our jobs.
I’m curious whether shutting down the east coast apps entirely and running off of the west coast was considered? Not enough capacity or some other problem?
edit: I guess the third technical initiative strongly implies that it was just a capacity issue: "This project has the goal of supporting N+1 redundancy at the facility level. The goal of that work is to tolerate the full failure of a single data center failure without user impact."
- collinf 8y agoIn the article they said that both regions consumed writes and got into an inconsistent state during the process. > This effort was challenging because by this point the West Coast database cluster had ingested writes from our application tier for nearly 40 minutes. Additionally, there were the several seconds of writes that existed in the East Coast cluster that had not been replicated to the West Coast and prevented replication of new writes back to the East Coast.
- itsdrewmiller 8y agoAs I read it their solution to the east coast write conflict was to just put those aside and reconcile them later (and they are still working on that) - I don't think that would impact their ability to turn off east coast app servers too.