3 ms·

The simple answer is "you don't," because a couple hours downtime on your CI/CD platform shouldn't matter. You should be able to deploy by hand in an emergency.

The simple answer is "you don't," because a couple hours downtime on your CI/CD platform shouldn't matter. You should be able to deploy by hand in an emergency. The less simple answer is "by doing Disaster Recovery planning". Go outside and throw a rock and you'll hit a company doing hosted CI/CD. Set up two of them; when one fails, use the other. GitHub could mitigate this by making its services region and AZ specific, and then only one region should go down at a time, letting you switch regions. Not sure why this hasn't happened, other than GHA is more of a "consumer product" and they may assume customers are too dumb to figure out how to switch regions. (This may sound harsh, but this is the same company that wouldn't update documentation about how to generate ssh keys to make them more secure because it might confuse their users)

numbsafari 4y ago
For example, if you are using AWS, you could use Code Pipeline for your CI/CD alongside GitHub actions. Use GHA for “PR open” triggered tasks, but Code Pipeline for tag or “PR merge” triggered tasks. This way your deployment is decoupled from GH availability, but you get the benefits of GHAs first-party integration with your core software development loops.
itsmemattchung 4y ago
maintaining bi modal behavior introduces a new set of problems. I'd take "should be able to deploy by hand in an emergency" a step further: deploy by hand, frequently. make sure that your production outage isn't the first time you are exercising this runbook.
- kylecordes 4y ago
  Related: Avoid the temptation to code up all the details of your build and deploy process in a proprietary single vendor CI tool. Code your build using a build tool, script you deploy using tools suitable for that. Now you can run the same process on multiple different CI services, or locally.
  - carlsmedstad 4y ago
    I recently delved into the topic of moving much of the logic from CI pipelines into Makefiles. This made me build https://github.com/mkincl/mkincl https://github.com/mkincl/mkincl

numbsafari 4y ago

For example, if you are using AWS, you could use Code Pipeline for your CI/CD alongside GitHub actions. Use GHA for “PR open” triggered tasks, but Code Pipeline for tag or “PR merge” triggered tasks. This way your deployment is decoupled from GH availability, but you get the benefits of GHAs first-party integration with your core software development loops.

itsmemattchung 4y ago

maintaining bi modal behavior introduces a new set of problems. I'd take "should be able to deploy by hand in an emergency" a step further: deploy by hand, frequently. make sure that your production outage isn't the first time you are exercising this runbook.

kylecordes 4y ago
Related: Avoid the temptation to code up all the details of your build and deploy process in a proprietary single vendor CI tool. Code your build using a build tool, script you deploy using tools suitable for that. Now you can run the same process on multiple different CI services, or locally.
- carlsmedstad 4y ago
  I recently delved into the topic of moving much of the logic from CI pipelines into Makefiles. This made me build https://github.com/mkincl/mkincl https://github.com/mkincl/mkincl