3 ms·
Someone will add safeguards for all that stuff and it ends up making it way harder to get real work done. I know in theory all of it can be done well, but in pr
by traderj0e 6mo ago
Someone will add safeguards for all that stuff and it ends up making it way harder to get real work done. I know in theory all of it can be done well, but in practice it's harder than it might sound.
I've seen this at work the most with slow rollouts. They said it was for prod only, then it became applied to staging and dev somehow. They said you can force push in emergencies, but approximately 0 people on any given team know how to do this reliably, and it still takes way longer even in --force --now --breakglass --yesimeanit mode. So the end result is longer MTTR. It maybe prevents some kinds of outages, but also you're less likely to manually monitor a rollout when it takes longer.
- lokar 6mo agoIf you automate it all it’s fine. The automation has no problem waiting around for traffic to drain out of something before decommissioning it.
- traderj0e 6mo agoThen you're back to square 1. The only way to win here is to require user supervision and make it simple and easy to use.
- lokar 6mo agoExpanding to dev, and not actually having people available on-call to deal with emergencies seems like an implementation problem. The basic setup has worked well for me at several companies with many large teams on-call.