4 ms·

Because if you do it in two rollouts, you don't find out that the first one is broken until you do the second one... which means the first isn't necessarily a s

by aiiane 7y ago

Because if you do it in two rollouts, you don't find out that the first one is broken until you do the second one... which means the first isn't necessarily a safe rollback target for the second. But the one before the first isn't safe either once the second has been rolled out because it doesn't have any logic to not explode upon seeing new values. The third can be a flag flip instead of a code push, but it still needs to be a discrete event to start generating the new value that triggers new behavior.

Thorrez 7y ago
So you're saying with the 3 step rollout if rollout 3/3 goes bad you should rollback straight to 1/3? I guess that does make a bit of sense in the case of a database where the newly created values can stick around. But also there could have been a bug in 1/3 that wasn't exposed until step 3/3. Meaning even rolling back to 1/3 won't solve the problem. The 2 step rollout is safe if we assume there are no bugs in the code, but problematic if there are bugs. But the 3 step rollout is also problematic if there are bugs. I guess the 3 step rollout has the advantage that it partitions off a section that might be buggier than other sections so that it can be individually disabled. But that'll only help sometimes, and I'm not sure if the additional complexity is worth it. Flag flip vs code push doesn't seem to make a difference to me. All 3 rollouts could be flag flips enabling code that was written much earlier but hidden behind flags. Another problem is doing a double rollback like that seems a bit risky to me, because other features that were being deployed simultaneously might not have been designed to handle a double rollback, only a single rollback, so they could break. If we want to allow double rollbacks, we must require that all development not just handle single rollbacks smoothly but also double rollbacks.
- dillonmckay 7y ago
  You shouldn’t be deploying other features simultaneously.

by aiiane 7y ago

Thorrez 7y ago

So you're saying with the 3 step rollout if rollout 3/3 goes bad you should rollback straight to 1/3? I guess that does make a bit of sense in the case of a database where the newly created values can stick around. But also there could have been a bug in 1/3 that wasn't exposed until step 3/3. Meaning even rolling back to 1/3 won't solve the problem. The 2 step rollout is safe if we assume there are no bugs in the code, but problematic if there are bugs. But the 3 step rollout is also problematic if there are bugs. I guess the 3 step rollout has the advantage that it partitions off a section that might be buggier than other sections so that it can be individually disabled. But that'll only help sometimes, and I'm not sure if the additional complexity is worth it. Flag flip vs code push doesn't seem to make a difference to me. All 3 rollouts could be flag flips enabling code that was written much earlier but hidden behind flags. Another problem is doing a double rollback like that seems a bit risky to me, because other features that were being deployed simultaneously might not have been designed to handle a double rollback, only a single rollback, so they could break. If we want to allow double rollbacks, we must require that all development not just handle single rollbacks smoothly but also double rollbacks.

dillonmckay 7y ago
You shouldn’t be deploying other features simultaneously.