3 ms·

My problem with this is that now every single production release is either pushed back by at least a day (multiple days if issues are found), OR there's code ru

by Sodman 6y ago

My problem with this is that now every single production release is either pushed back by at least a day (multiple days if issues are found), OR there's code running in production that hasn't run through e2e tests yet. Plus it adds a layer of complexity to your version control. I'm assuming you're running these e2e tests against a branch that's up to date with the latest master. What happens if I have 4 PRs I need to merge - they all might pass tests when run against the current master but as soon as I merge one, the other 3 haven't technically been tested with master anymore, so I'll have to re-run e2e, which pushes everything back by another day (per PR)?

asdfasgasdgasdg 6y ago
> is either pushed back by at least a day (multiple days if issues are found), Which is better than issues making it to production -- presumably what would have happened if you didn't have the tests. > OR there's code running in production that hasn't run through e2e tests yet. This situation should never occur except if there's a production emergency causing immediate loss of revenue. Even then, the proper answer is usually to roll back to a known-good version. > I'm assuming you're running these e2e tests against a branch that's up to date with the latest master. What happens if I have 4 PRs I need to merge - they all might pass tests when run against the current master but as soon as I merge one, the other 3 haven't technically been tested with master anymore, so I'll have to re-run e2e, which pushes everything back by another day (per PR)? The way releases should work is that periodically you cut a release in a new release branch. After that, further changes to master don't affect the release. All the e2e tests need to run against the release branch before it can be deployed. It should be deployed immediately to your test realm and given time to soak. If bugs are found, minimal fixes should be made against master and then cherry-picked into the release branch, followed by a rerun of tests and redeployment to the test realm. You should aim for a zero-cherry-pick situation. Do this by enabling features and changes via configuration rather than via binary changes. Once the release branch reaches the appropriate level of stability, it should be gradually rolled out, zone by zone, to production. Zones should have some soak time before further zones are touched. The response to production anomalies should be a roll back to the previous version. This applies to big, complex services that are mission-critical. For small, less critical services, a daily/push-on-green type of approach might be fine. But for such services testing should be much easier and it shouldn't be a huge imposition to run all tests before each release.
- Sodman 6y ago
  My problem with the "release-branch" approach is mostly that your prod releases end up being big-bang multi-feature releases. Any bug in one feature can delay the entire release. More changes usually means more room for things to go wrong. It's also harder to debug issues when they do happen, and they will happen eventually, regardless of how thorough your test cycle is. Your developers may also have lost context if they time between writing the code and deploying it is so long (eg if this process forces you to monthly or quarterly releases). > Once the release branch reaches the appropriate level of stability, it should be gradually rolled out, zone by zone, to production. Zones should have some soak time before further zones are touched. The response to production anomalies should be a roll back to the previous version. I 100% agree with you on this. Canary releases, followed by gradual rollouts with the option to roll back are ideal. > Do this by enabling features and changes via configuration rather than via binary changes. If your CI/CD pipeline is fast enough that you can deploy on each commit to master (multiple times daily), then you can roll forward in the rare situations that rolling back isn't an option. Feature flag frameworks certainly have some useful features, but they add a layer of complexity to your deployments which feels like a band-aid to help get around slow deployments. Totally valid for eg an iOS app that takes 2 weeks for Apple to review, but if you're deploying to the web where you can push new releases whenever you want, I don't think the added complexity is worth it. I think your thorough, managed release approach is absolutely appropriate when the consequences of failure are life-threatening, or heavy revenue losses or brand damage. But the worst case scenario for most of the software most of us work on are closer to "1% of users could potentially see an error message during a 5-60s window before we detect the increased error rate and roll back the canary release"
  - deleted 6y ago
    [deleted]

by Sodman 6y ago

asdfasgasdgasdg 6y ago

> is either pushed back by at least a day (multiple days if issues are found), Which is better than issues making it to production -- presumably what would have happened if you didn't have the tests. > OR there's code running in production that hasn't run through e2e tests yet. This situation should never occur except if there's a production emergency causing immediate loss of revenue. Even then, the proper answer is usually to roll back to a known-good version. > I'm assuming you're running these e2e tests against a branch that's up to date with the latest master. What happens if I have 4 PRs I need to merge - they all might pass tests when run against the current master but as soon as I merge one, the other 3 haven't technically been tested with master anymore, so I'll have to re-run e2e, which pushes everything back by another day (per PR)? The way releases should work is that periodically you cut a release in a new release branch. After that, further changes to master don't affect the release. All the e2e tests need to run against the release branch before it can be deployed. It should be deployed immediately to your test realm and given time to soak. If bugs are found, minimal fixes should be made against master and then cherry-picked into the release branch, followed by a rerun of tests and redeployment to the test realm. You should aim for a zero-cherry-pick situation. Do this by enabling features and changes via configuration rather than via binary changes. Once the release branch reaches the appropriate level of stability, it should be gradually rolled out, zone by zone, to production. Zones should have some soak time before further zones are touched. The response to production anomalies should be a roll back to the previous version. This applies to big, complex services that are mission-critical. For small, less critical services, a daily/push-on-green type of approach might be fine. But for such services testing should be much easier and it shouldn't be a huge imposition to run all tests before each release.

Sodman 6y ago
My problem with the "release-branch" approach is mostly that your prod releases end up being big-bang multi-feature releases. Any bug in one feature can delay the entire release. More changes usually means more room for things to go wrong. It's also harder to debug issues when they do happen, and they will happen eventually, regardless of how thorough your test cycle is. Your developers may also have lost context if they time between writing the code and deploying it is so long (eg if this process forces you to monthly or quarterly releases). > Once the release branch reaches the appropriate level of stability, it should be gradually rolled out, zone by zone, to production. Zones should have some soak time before further zones are touched. The response to production anomalies should be a roll back to the previous version. I 100% agree with you on this. Canary releases, followed by gradual rollouts with the option to roll back are ideal. > Do this by enabling features and changes via configuration rather than via binary changes. If your CI/CD pipeline is fast enough that you can deploy on each commit to master (multiple times daily), then you can roll forward in the rare situations that rolling back isn't an option. Feature flag frameworks certainly have some useful features, but they add a layer of complexity to your deployments which feels like a band-aid to help get around slow deployments. Totally valid for eg an iOS app that takes 2 weeks for Apple to review, but if you're deploying to the web where you can push new releases whenever you want, I don't think the added complexity is worth it. I think your thorough, managed release approach is absolutely appropriate when the consequences of failure are life-threatening, or heavy revenue losses or brand damage. But the worst case scenario for most of the software most of us work on are closer to "1% of users could potentially see an error message during a 5-60s window before we detect the increased error rate and roll back the canary release"
- deleted 6y ago
  [deleted]