3 ms·

The nice thing about git, from my perspective, is that if your entire hosted service vanishes, you can still reconstruct what you need from your users’ working

The nice thing about git, from my perspective, is that if your entire hosted service vanishes, you can still reconstruct what you need from your users’ working directories. All of the important branches should be there. Somewhere. And any important integration branches that aren’t cached can be reconstructed. Of all the many dependencies on cloud services, git is by far the last I’d worry overly much about.

jcims 2y ago
I think my point is we are losing the option to worry about it or not. There needs to be an answer with a runbook to restore services within a given window of time should we lose that vendor.
dangoldin 2y ago
Problem is that often you also end up relying on GitHub for CI/CD so not as easy of a change. Imagine GH being down and you need to deploy a hotfix. How do you handle that? Especially, if you followed best practices and set up a system where all PRs need to go through code review.
- hamandcheese 2y ago
  Systems like these should have an escape hatch of some sort. The key part is that it needs to be auditable. Anything you do in CI should be possible outside of CI, at least by some subset of users.
  - dangoldin 2y ago
    Yea - definitely. Just not ideal and something that needs to be built out, tested, etc.
    - hamandcheese 2y ago
      Yes, it is easier said than done. At my company we use buildkite and many people wrote scripts that simply fail outside of buildkite. GitHub actions is even worse, it seems like it was designed from the ground up to create lock in. Nix helps a bit on the bootstrapping and dependency management problem, but won't save you from writing a script that is too tightly coupled to its runtime environment.
  - citizenpaul 2y ago
    I've seen numerous "escape hatches" over the years that actually just turned out to be painted on the wall. If you actually tried to use them. No one ever does though. I don't think its malice. I just think its pretty uncommon for anyone to intentionally back out of a structural tech decision so it gets forgotten about and remains un-battle tested.. That or the timeline is longer than SaaS has been around.
- nosefurhairdo 2y ago
  I've run into a scenario where one of our rarely used environments needed a hotfix and the GitHub action we used to deploy there was broken. Was easy enough to translate GitHub action steps to shell scripting for a quick, manual deployment.
- supriyo-biswas 2y ago
  This is why I personally like to use none of the CI features, and mostly use it like a shell script executor. Images? Stick to OS images only so that you can easily spin them up with `docker run` locally. Artifacts? Read and write them into S3 buckets and avoid the native artifact features. This is obviously more difficult in the Github actions ecosystem, but I have mostly used Gitlab CI so far. My CI pipelines mostly look like this: image: ubuntu:24.04 before_script: - apt-get install ... script: - ./ci/build-project.sh after_script: - ./ci/upload-build-artifacts.sh
theteapot 2y ago
Git is only a small part of Github these days.
deathanatos 2y ago
The source code, perhaps, but a good many of the orgs I've worked at also use Github for PRs, Actions (CI), and for triggering deployments. Those things take time to setup, and across a whole org, I wouldn't want to have to have an unplanned change to another vendor. In particular, I've moved a CI for a large repository between different CI systems. It was anything but trivial: you want to believe "it's just a YAML that runs commands, right? Translate the format, right?" but it's really not; differences between how CI systems map commands to machines, external integrations (e.g., in this case, into Action's artifacts system, or output system) etc. all make it more complicated.
- notpushkin 2y ago
  Totally agree (and also that’s the reason I think all CI ecosystems are a nightmare nowadays). But GitHub Actions are somewhat portable: there’s the standalone act [0] runner, and the Forgejo/Gitea Actions (e.g. on Codeberg [1]) that use act under the hood and are pretty much drop-in replacement – they even use GitHub-hosted actions transparently. It might not be a 100% compatible standard, but it’s pretty nice. It would be nice for others to follow lead! [0]: https://nektosact.com/ https://nektosact.com/ [1]: https://docs.codeberg.org/ci/actions/ https://docs.codeberg.org/ci/actions/
- jamesfinlayson 2y ago
  Ugh, agreed - I remember doing a GitHub+TeamCity to GitLab migration and I had specifically designed all of my jobs to just directly call out to a bash script. Artifacts and GitLab's infinite hooks made that much harder than it seemed like it would be.
- physicsguy 2y ago
  This is a good reason for keeping your build steps in scripts or a Makefile rather than jumping head first into the ecosystem. I think unless you’ve been burned by having to move CI provider before it’s easy to lean in. I had to change from Travis many years ago because of pricing changes.

jcims 2y ago

I think my point is we are losing the option to worry about it or not. There needs to be an answer with a runbook to restore services within a given window of time should we lose that vendor.

dangoldin 2y ago

Problem is that often you also end up relying on GitHub for CI/CD so not as easy of a change. Imagine GH being down and you need to deploy a hotfix. How do you handle that? Especially, if you followed best practices and set up a system where all PRs need to go through code review.

hamandcheese 2y ago
Systems like these should have an escape hatch of some sort. The key part is that it needs to be auditable. Anything you do in CI should be possible outside of CI, at least by some subset of users.
- dangoldin 2y ago
  Yea - definitely. Just not ideal and something that needs to be built out, tested, etc.
  - hamandcheese 2y ago
    Yes, it is easier said than done. At my company we use buildkite and many people wrote scripts that simply fail outside of buildkite. GitHub actions is even worse, it seems like it was designed from the ground up to create lock in. Nix helps a bit on the bootstrapping and dependency management problem, but won't save you from writing a script that is too tightly coupled to its runtime environment.
- citizenpaul 2y ago
  I've seen numerous "escape hatches" over the years that actually just turned out to be painted on the wall. If you actually tried to use them. No one ever does though. I don't think its malice. I just think its pretty uncommon for anyone to intentionally back out of a structural tech decision so it gets forgotten about and remains un-battle tested.. That or the timeline is longer than SaaS has been around.
nosefurhairdo 2y ago
I've run into a scenario where one of our rarely used environments needed a hotfix and the GitHub action we used to deploy there was broken. Was easy enough to translate GitHub action steps to shell scripting for a quick, manual deployment.
supriyo-biswas 2y ago
This is why I personally like to use none of the CI features, and mostly use it like a shell script executor. Images? Stick to OS images only so that you can easily spin them up with `docker run` locally. Artifacts? Read and write them into S3 buckets and avoid the native artifact features. This is obviously more difficult in the Github actions ecosystem, but I have mostly used Gitlab CI so far. My CI pipelines mostly look like this: image: ubuntu:24.04 before_script: - apt-get install ... script: - ./ci/build-project.sh after_script: - ./ci/upload-build-artifacts.sh

theteapot 2y ago

Git is only a small part of Github these days.

deathanatos 2y ago

The source code, perhaps, but a good many of the orgs I've worked at also use Github for PRs, Actions (CI), and for triggering deployments. Those things take time to setup, and across a whole org, I wouldn't want to have to have an unplanned change to another vendor. In particular, I've moved a CI for a large repository between different CI systems. It was anything but trivial: you want to believe "it's just a YAML that runs commands, right? Translate the format, right?" but it's really not; differences between how CI systems map commands to machines, external integrations (e.g., in this case, into Action's artifacts system, or output system) etc. all make it more complicated.

notpushkin 2y ago
Totally agree (and also that’s the reason I think all CI ecosystems are a nightmare nowadays). But GitHub Actions are somewhat portable: there’s the standalone act [0] runner, and the Forgejo/Gitea Actions (e.g. on Codeberg [1]) that use act under the hood and are pretty much drop-in replacement – they even use GitHub-hosted actions transparently. It might not be a 100% compatible standard, but it’s pretty nice. It would be nice for others to follow lead! [0]: https://nektosact.com/ https://nektosact.com/ [1]: https://docs.codeberg.org/ci/actions/ https://docs.codeberg.org/ci/actions/
jamesfinlayson 2y ago
Ugh, agreed - I remember doing a GitHub+TeamCity to GitLab migration and I had specifically designed all of my jobs to just directly call out to a bash script. Artifacts and GitLab's infinite hooks made that much harder than it seemed like it would be.
physicsguy 2y ago
This is a good reason for keeping your build steps in scripts or a Makefile rather than jumping head first into the ecosystem. I think unless you’ve been burned by having to move CI provider before it’s easy to lean in. I had to change from Travis many years ago because of pricing changes.