3 ms·
Every developer makes a rollback-worthy mistake every x changes. This means your probability of having a clean release is (1-1/x)^NumDevs EDIT - Assuming everyo
by jpollock 5y ago
Every developer makes a rollback-worthy mistake every x changes. This means your probability of having a clean release is (1-1/x)^NumDevs EDIT - Assuming everyone submits 1 change/release.
Let's give it 1/100 bad code change, and 100 devs. The probability of a clean push is:
(1-1/100)^100 = 0.36. That's worse than a coin flip.
for 10 devs = 0.90. Still not great, one push in 10 will be bad.
When you're trying to push daily (or even more frequently), this will kill the team's velocity. It's even worse when there are interleaving changes making rollback and re-release impossible before the next release. Rollback and freeze is a problem when you're trying to meet deadlines (either marketing or regulatory).
Feature flags allow the code to go in and each change to be rolled back independently. This lets the rest of the team to make progress while the bug is debugged.
Short release pipelines could be a solution, but they're aren't necessarily sufficient.
Time to recovery when there isn't a feature flag is also problem.
For example, some bugs take days/weeks to reveal themselves (think month end), so a "30 min build and full push" is not enough of a guarantee. Time to recovery needs to consider the time to find the change and roll it back, including dealing with any interleaving changes that happened in the meantime.