3 ms·
Isn’t that circular dependency kind of risky? What if a bad deploy knocks offline something required to complete the rollback process for that bad deploy? I’m
by orf 5y ago
Isn’t that circular dependency kind of risky? What if a bad deploy knocks offline something required to complete the rollback process for that bad deploy?
I’m sure there are multiple layers of redundancy, but it seems like you could get into a Facebook style situation where you need to go open a (virtual) cage somewhere to flip a switch.
- dekhn 5y agoYes, it's risky but given all the alternatives, it makes the least nonsense. It works because there is massive redundancy and datacenters aren't all updated at the same time. When I worked at Google I worked in hardware platforms and I spent a lot of time working with hwops people, who have to go physically out to some server and press buttons (whilst chatting the results to me). If corp gmail, chat, and everything else was down, I might have some trouble getting in touch with those folks and verifying the right fixes (which is why the FB situation is so crazy).
- ramraj07 5y agoUntil something goes bad in a silly place like it did in Facebook and everything disappears everywhere for a good while. Because google is everything.