3 ms·
> Teams generally learn these things as object lessons (as in “we had downtime, wrote a postmortem, and it won’t happen again”) but that’s not the way things ne
by ryanbrunner 7y ago
> Teams generally learn these things as object lessons (as in “we had downtime, wrote a postmortem, and it won’t happen again”) but that’s not the way things need to be. As the team evolves and the product matures, the release tooling gets better. You read articles like this and you can make the tooling better without the object lesson. No one-size-fits-all solution, obviously, but… painless, one-button rollbacks are one-size-fits-most for sure!
One advantage of 'learning as you go' is that oftentimes you can afford to be messy and cut some corners when you're just starting out without it impacting people too much.
I don't think anyone would disagree that having a rollout process that can be easily rolled back, utilizes things like feature flags, incremental rollout, and care around schema changes is a better process, but it's undeniably slower than "just roll it out and deal with the crashes", particularly when there's not plug and play standards for a lot of those processes and you need to build them yourself.
And if you have a tiny amount of users who are the early adopter type, messiness around rollouts and the occasional outage might not be the worst thing if you can move much faster.
- klodolph 7y ago> …messiness around rollouts and the occasional outage might not be the worst thing if you can move much faster. Unfortunately, my personal experience with software engineers pushing to “roll forward” on failure is that they want to meet arbitrary deadlines. Metrics that are good for the SWE (e.g. if we roll forward, I complete my Q4 OKRs) but not good for the team—total velocity isn’t negatively impacted by a single rollback, at least not much, but reliability is positively impacted by rollback policy.