3 ms·

>In this case, you need to make sure you can recognize the new value and not explode just from seeing it, then get that shipped everywhere. Then you have to be

>In this case, you need to make sure you can recognize the new value and not explode just from seeing it, then get that shipped everywhere. Then you have to be able to do something reasonable when it occurs, and get that shipped everywhere. Finally, you can flip whatever flag lets you start actually emitting that new value and see what happens. Can't those first 2 steps be combined together? Why do they need to be shipped separately?

akuji1993 7y ago
Depends on your updating architecture.. If you have a central backend service, but scheduled patches or the user has to manually update, that might become a problem. There could also be the case that client and backend can be updated independently and one might not be done before the other. The issue might only exist for a few minutes, or an hour, but would still be a huge impact on the software.
- Thorrez 7y ago
  Sorry, I still don't understand. The post say you need to do 1. Have the code recognize the new value. Get that shipped everywhere. 2. Have the code do something reasonable when the new value appears. Get that shipped everywhere. 3. Start emitting the new value. My proposal is 1. Have the code recognize the new value. Have the code do something reasonable when the new value appears. Get that shipped everywhere. 2. Start emitting the new value. If the user has to manually update, and might refuse to do so, the problem exists under both the 3 step and the 2 step process. So nothing is gained by using the 3 step process. If the client and backend are updated independently, that should still be fine with the 2 step process. The 2 step process says "Get that shipped everywhere", meaning shipped to both the client and the backend.
  - joshuamorton 7y ago
    I would rephrase this as 1. Support the new value in your schema 2. Support the new value in the client 3. Emit the new value from the server Combining 1 and 2 is probably possible, but not a great idea. Imagine if you end up rolling back 2 and 3, but there are still potentially new values in flight. If 1 is still there, you're good. If 1 and 2 are combined, you rollback all 3, but the new value is still in flight and your client crashes.
    - aeorgnoieang 7y ago
      The advice seemed to be to, first, support (i.e. correctly ignore) any possibly new value in both the client and server (i.e. any client of the 'schema').
    - deleted 7y ago
      [deleted]
  - dkersten 7y ago
    If 2. breaks when you do 3., you roll back to 1. If you rolled back to the one before 1, then the values that 3 emitted may still be in the system, causing it to break. When you do 3, you're really testing 2 and 1 is the fallback. If you did 1 and 2 together, then you can't roll back just 2 on its own.
perlgeek 7y ago
The important part was somewhat implied in the OP. You have to * Ship the new version * Wait until you are sure you don't need a rollback to an old version * flip the switch If you don't wait, you might need to roll back to an old version that doesn't support the new state, which then blows up.
- Thorrez 7y ago
  I get that there need to be 2 rollouts. What I don't get is that the post says there need to be 3 rollouts.
  - aiiane 7y ago
    Because if you do it in two rollouts, you don't find out that the first one is broken until you do the second one... which means the first isn't necessarily a safe rollback target for the second. But the one before the first isn't safe either once the second has been rolled out because it doesn't have any logic to not explode upon seeing new values. The third can be a flag flip instead of a code push, but it still needs to be a discrete event to start generating the new value that triggers new behavior.
    - Thorrez 7y ago
      So you're saying with the 3 step rollout if rollout 3/3 goes bad you should rollback straight to 1/3? I guess that does make a bit of sense in the case of a database where the newly created values can stick around. But also there could have been a bug in 1/3 that wasn't exposed until step 3/3. Meaning even rolling back to 1/3 won't solve the problem. The 2 step rollout is safe if we assume there are no bugs in the code, but problematic if there are bugs. But the 3 step rollout is also problematic if there are bugs. I guess the 3 step rollout has the advantage that it partitions off a section that might be buggier than other sections so that it can be individually disabled. But that'll only help sometimes, and I'm not sure if the additional complexity is worth it. Flag flip vs code push doesn't seem to make a difference to me. All 3 rollouts could be flag flips enabling code that was written much earlier but hidden behind flags. Another problem is doing a double rollback like that seems a bit risky to me, because other features that were being deployed simultaneously might not have been designed to handle a double rollback, only a single rollback, so they could break. If we want to allow double rollbacks, we must require that all development not just handle single rollbacks smoothly but also double rollbacks.

akuji1993 7y ago

Depends on your updating architecture.. If you have a central backend service, but scheduled patches or the user has to manually update, that might become a problem. There could also be the case that client and backend can be updated independently and one might not be done before the other. The issue might only exist for a few minutes, or an hour, but would still be a huge impact on the software.

Thorrez 7y ago
Sorry, I still don't understand. The post say you need to do 1. Have the code recognize the new value. Get that shipped everywhere. 2. Have the code do something reasonable when the new value appears. Get that shipped everywhere. 3. Start emitting the new value. My proposal is 1. Have the code recognize the new value. Have the code do something reasonable when the new value appears. Get that shipped everywhere. 2. Start emitting the new value. If the user has to manually update, and might refuse to do so, the problem exists under both the 3 step and the 2 step process. So nothing is gained by using the 3 step process. If the client and backend are updated independently, that should still be fine with the 2 step process. The 2 step process says "Get that shipped everywhere", meaning shipped to both the client and the backend.
- joshuamorton 7y ago
  I would rephrase this as 1. Support the new value in your schema 2. Support the new value in the client 3. Emit the new value from the server Combining 1 and 2 is probably possible, but not a great idea. Imagine if you end up rolling back 2 and 3, but there are still potentially new values in flight. If 1 is still there, you're good. If 1 and 2 are combined, you rollback all 3, but the new value is still in flight and your client crashes.
  - aeorgnoieang 7y ago
    The advice seemed to be to, first, support (i.e. correctly ignore) any possibly new value in both the client and server (i.e. any client of the 'schema').
  - deleted 7y ago
    [deleted]
- dkersten 7y ago
  If 2. breaks when you do 3., you roll back to 1. If you rolled back to the one before 1, then the values that 3 emitted may still be in the system, causing it to break. When you do 3, you're really testing 2 and 1 is the fallback. If you did 1 and 2 together, then you can't roll back just 2 on its own.

perlgeek 7y ago

The important part was somewhat implied in the OP. You have to * Ship the new version * Wait until you are sure you don't need a rollback to an old version * flip the switch If you don't wait, you might need to roll back to an old version that doesn't support the new state, which then blows up.

Thorrez 7y ago
I get that there need to be 2 rollouts. What I don't get is that the post says there need to be 3 rollouts.
- aiiane 7y ago
  Because if you do it in two rollouts, you don't find out that the first one is broken until you do the second one... which means the first isn't necessarily a safe rollback target for the second. But the one before the first isn't safe either once the second has been rolled out because it doesn't have any logic to not explode upon seeing new values. The third can be a flag flip instead of a code push, but it still needs to be a discrete event to start generating the new value that triggers new behavior.
  - Thorrez 7y ago
    So you're saying with the 3 step rollout if rollout 3/3 goes bad you should rollback straight to 1/3? I guess that does make a bit of sense in the case of a database where the newly created values can stick around. But also there could have been a bug in 1/3 that wasn't exposed until step 3/3. Meaning even rolling back to 1/3 won't solve the problem. The 2 step rollout is safe if we assume there are no bugs in the code, but problematic if there are bugs. But the 3 step rollout is also problematic if there are bugs. I guess the 3 step rollout has the advantage that it partitions off a section that might be buggier than other sections so that it can be individually disabled. But that'll only help sometimes, and I'm not sure if the additional complexity is worth it. Flag flip vs code push doesn't seem to make a difference to me. All 3 rollouts could be flag flips enabling code that was written much earlier but hidden behind flags. Another problem is doing a double rollback like that seems a bit risky to me, because other features that were being deployed simultaneously might not have been designed to handle a double rollback, only a single rollback, so they could break. If we want to allow double rollbacks, we must require that all development not just handle single rollbacks smoothly but also double rollbacks.