4 ms·

Now that everyone and their dog is moving to "serverless" hosting, how does one institute redundancies in case something like this happens again? It would suck

Now that everyone and their dog is moving to "serverless" hosting, how does one institute redundancies in case something like this happens again? It would suck if half the Internet and its open source projects go down because of a bad server configuration. Does anyone know of a "load balancer" for Github projects or source management in general? It looks like a solution many could benefit from.

smeagull 4y ago
Also use gitlab?
numbsafari 4y ago
Or just some flexibility? We don’t rely on GH for our production systems. We have a clone of our Git repo hosted in-house for anything that needs a Git repo. We don’t rely on GH for our Primary CI/CD. We can make changes and push them through CI/CD without GH. We have an understanding of the business impact of GH being down. It’s an enabler that we love using, but we can operate without it for a known period of time, albeit without certain workflows and value added tools. If our tolerance for it being down we’re much greater, then it would be worth the money and overhead to host it on our own, which they support.
- Turbots 4y ago
throwaway892238 4y ago
The simple answer is "you don't," because a couple hours downtime on your CI/CD platform shouldn't matter. You should be able to deploy by hand in an emergency. The less simple answer is "by doing Disaster Recovery planning". Go outside and throw a rock and you'll hit a company doing hosted CI/CD. Set up two of them; when one fails, use the other. GitHub could mitigate this by making its services region and AZ specific, and then only one region should go down at a time, letting you switch regions. Not sure why this hasn't happened, other than GHA is more of a "consumer product" and they may assume customers are too dumb to figure out how to switch regions. (This may sound harsh, but this is the same company that wouldn't update documentation about how to generate ssh keys to make them more secure because it might confuse their users)
- numbsafari 4y ago
  For example, if you are using AWS, you could use Code Pipeline for your CI/CD alongside GitHub actions. Use GHA for “PR open” triggered tasks, but Code Pipeline for tag or “PR merge” triggered tasks. This way your deployment is decoupled from GH availability, but you get the benefits of GHAs first-party integration with your core software development loops.
- itsmemattchung 4y ago
  maintaining bi modal behavior introduces a new set of problems. I'd take "should be able to deploy by hand in an emergency" a step further: deploy by hand, frequently. make sure that your production outage isn't the first time you are exercising this runbook.
  - kylecordes 4y ago
    Related: Avoid the temptation to code up all the details of your build and deploy process in a proprietary single vendor CI tool. Code your build using a build tool, script you deploy using tools suitable for that. Now you can run the same process on multiple different CI services, or locally.
    - carlsmedstad 4y ago
      I recently delved into the topic of moving much of the logic from CI pipelines into Makefiles. This made me build https://github.com/mkincl/mkincl https://github.com/mkincl/mkincl
ChrisLTD 4y ago
Git is already decentralized, and you can push your code to any of the dozens of competitors easily. I often keep important projects on both Bitbucket and GitHub.

smeagull 4y ago

Also use gitlab?

numbsafari 4y ago

Or just some flexibility? We don’t rely on GH for our production systems. We have a clone of our Git repo hosted in-house for anything that needs a Git repo. We don’t rely on GH for our Primary CI/CD. We can make changes and push them through CI/CD without GH. We have an understanding of the business impact of GH being down. It’s an enabler that we love using, but we can operate without it for a known period of time, albeit without certain workflows and value added tools. If our tolerance for it being down we’re much greater, then it would be worth the money and overhead to host it on our own, which they support.

Turbots 4y ago

throwaway892238 4y ago

The simple answer is "you don't," because a couple hours downtime on your CI/CD platform shouldn't matter. You should be able to deploy by hand in an emergency. The less simple answer is "by doing Disaster Recovery planning". Go outside and throw a rock and you'll hit a company doing hosted CI/CD. Set up two of them; when one fails, use the other. GitHub could mitigate this by making its services region and AZ specific, and then only one region should go down at a time, letting you switch regions. Not sure why this hasn't happened, other than GHA is more of a "consumer product" and they may assume customers are too dumb to figure out how to switch regions. (This may sound harsh, but this is the same company that wouldn't update documentation about how to generate ssh keys to make them more secure because it might confuse their users)

numbsafari 4y ago
For example, if you are using AWS, you could use Code Pipeline for your CI/CD alongside GitHub actions. Use GHA for “PR open” triggered tasks, but Code Pipeline for tag or “PR merge” triggered tasks. This way your deployment is decoupled from GH availability, but you get the benefits of GHAs first-party integration with your core software development loops.
itsmemattchung 4y ago
maintaining bi modal behavior introduces a new set of problems. I'd take "should be able to deploy by hand in an emergency" a step further: deploy by hand, frequently. make sure that your production outage isn't the first time you are exercising this runbook.
- kylecordes 4y ago
  Related: Avoid the temptation to code up all the details of your build and deploy process in a proprietary single vendor CI tool. Code your build using a build tool, script you deploy using tools suitable for that. Now you can run the same process on multiple different CI services, or locally.
  - carlsmedstad 4y ago
    I recently delved into the topic of moving much of the logic from CI pipelines into Makefiles. This made me build https://github.com/mkincl/mkincl https://github.com/mkincl/mkincl

ChrisLTD 4y ago

Git is already decentralized, and you can push your code to any of the dozens of competitors easily. I often keep important projects on both Bitbucket and GitHub.