3 ms·

Blue-green deployment is the only way to fly: https://martinfowler.com/bliki/BlueGreenDeployment.html https://martinfowler.com/bliki/BlueGreenDeployment.html T

Blue-green deployment is the only way to fly: https://martinfowler.com/bliki/BlueGreenDeployment.html https://martinfowler.com/bliki/BlueGreenDeployment.html There are two identical prod servers/cloud configurations/datacenters: blue and green. Each new version is deployed intermittently on blue and green areas: if version N is on blue, version N-1 is on green, and vice versa. If some critical issue happens, rolling back is just switching the front router/balancer to the other area, which can be done instantly.

gfodor 7y ago
Yup, can't beat blue/green. A big part of the reason I recommend it is because unlike most other rollback schemes, the actual mechanism for rolling back is the same as for a normal release (a load balancer flip), so it's continually exercised and validated. To roll back, you literally do a deploy, but you just skip the step where you alter the bits on the dark cluster. This is highly unlikely to fail, since the only thing that has to not go wrong is the part where the software update is skipped, which basically boils down to a conditional. Any mechanism for rollbacks that isn't tested continuously is likely to fail during incident response. It's a huge anti-pattern to have 'dark' processes only used during incident response -- same thinking behind why you should also be continually testing your backups, continuously killing servers to verify recovery, etc.
- Angostura 7y ago
  They you just have to worry about rolling back the changed database schema that you have rolled out to both green and bluie variants - I presume. That bit feels tricky
  - gfodor 7y ago
    Yep, that's a fair point. In scenarios where you have made non-backwards compatible database changes, this approach cannot be used. Typically the dark cluster is accessible for pre-flight testing, so you can run tests there to determine if the service is still operating as expected despite the database being one migration step ahead. In your response runbook, you should have a contingency for the scenarios where that happens. (For many projects, the % of changes that have this characteristic are small enough that it doesn't move the needle on the risk profile, but Murphy's law can mess with you :))
keithyjohnson 7y ago
Cost can be a killer here though. If you're flipping from blue to green and vice versa you either have to have capacity in stand-by(expensive) or spin up new capacity before flipping(time-consuming).
- 1ba9115454 7y ago
  We use a blue/green strategy but our infrastructure can be created on demand with Pulumi. https://www.pulumi.com/ https://www.pulumi.com/ It takes Pulumi about 15 minutes to create our kubernetes cluster with all pods and monitoring in place. After a succesful rollout we can pulumi down one of the clusters and reduce costs (We're on azure.)

gfodor 7y ago

Yup, can't beat blue/green. A big part of the reason I recommend it is because unlike most other rollback schemes, the actual mechanism for rolling back is the same as for a normal release (a load balancer flip), so it's continually exercised and validated. To roll back, you literally do a deploy, but you just skip the step where you alter the bits on the dark cluster. This is highly unlikely to fail, since the only thing that has to not go wrong is the part where the software update is skipped, which basically boils down to a conditional. Any mechanism for rollbacks that isn't tested continuously is likely to fail during incident response. It's a huge anti-pattern to have 'dark' processes only used during incident response -- same thinking behind why you should also be continually testing your backups, continuously killing servers to verify recovery, etc.

Angostura 7y ago
They you just have to worry about rolling back the changed database schema that you have rolled out to both green and bluie variants - I presume. That bit feels tricky
- gfodor 7y ago
  Yep, that's a fair point. In scenarios where you have made non-backwards compatible database changes, this approach cannot be used. Typically the dark cluster is accessible for pre-flight testing, so you can run tests there to determine if the service is still operating as expected despite the database being one migration step ahead. In your response runbook, you should have a contingency for the scenarios where that happens. (For many projects, the % of changes that have this characteristic are small enough that it doesn't move the needle on the risk profile, but Murphy's law can mess with you :))

keithyjohnson 7y ago

Cost can be a killer here though. If you're flipping from blue to green and vice versa you either have to have capacity in stand-by(expensive) or spin up new capacity before flipping(time-consuming).

1ba9115454 7y ago
We use a blue/green strategy but our infrastructure can be created on demand with Pulumi. https://www.pulumi.com/ https://www.pulumi.com/ It takes Pulumi about 15 minutes to create our kubernetes cluster with all pods and monitoring in place. After a succesful rollout we can pulumi down one of the clusters and reduce costs (We're on azure.)