4 ms·

And just as we're about to migrate 4 kubernetes clusters with a total of ~4k pods. Terraform in github actions on selfhosted runners and argoCD is failing.

mrweasel 3y ago
Oh that sucks, there's always going to be those who will say that it's the price you pay for using Github, but locally hosted VCS and CI/CD systems have issues as well. External dependencies are always problem, but do you have the capacity and resources required to manage those dependencies internally? Most don't and will still get a better product/service by using an external service.
- marginalia_nu 3y ago
  Rate of outages on github last few years has been orders of magnitude higher than anything I've encountered on a locally hosted VCS. Local also means you can orchestrate maintenance windows to avoid outages at critical phases.
- darkwater 3y ago
  > Most don't Define "most". There is a surprisingly high number of small/mid-sized companies which have dedicated people for this kind of things.
  - mrweasel 3y ago
    Perhaps "many" would have been better. If you only count companies that view themselves as IT companies then number of companies with self hosted/managed solution grows, but if you include everything then I'd guess that more than 50% don't run these services internally. If you're every small company with one or two developers that doubles as the IT staff then the numbers add up pretty quickly.
snarkyturtle 3y ago
That's where I feel like it's actually pretty nice to not have CI tied to your source code. It's probably more expensive to use Travis/Circle but at least you don't have a single point of failure for deploys.
- sleepychu 3y ago
  Doesn't this give you 2xSPF? Or can I use my local copy of the source to kick of Travis/Circle?
  - jpgvm 3y ago
    Ideally you don't ever rely on CI specific automation tooling to actually accomplish anything and instead just use it as a dumb "not my dev machine" to execute workflows. You should always engineer things so you can fall back to something akin to: ./scripts/deploy_the_things Ideally backed by a real build system and task engine ala Bazel, Gradle, whatever else floats your boat. It also means you are free to move between different runners/CI providers and just have to work out how to re-plumb secrets/whatever. GH actions/friends really provide minimal value, the fact they have convinced everyone that encoding task graphs into franken-YAML-bash is somehow good is one of the more egregious lies sold to the development community at large.
  - snarkyturtle 3y ago
    Well, git ops might not be impacted on Github, usual Github outages tend to be through Actions/the site, not the actual git operations. Doesn't seem like you can use a local copy but you can use Gitbucket/Gitlab.

mrweasel 3y ago

Oh that sucks, there's always going to be those who will say that it's the price you pay for using Github, but locally hosted VCS and CI/CD systems have issues as well. External dependencies are always problem, but do you have the capacity and resources required to manage those dependencies internally? Most don't and will still get a better product/service by using an external service.

marginalia_nu 3y ago
Rate of outages on github last few years has been orders of magnitude higher than anything I've encountered on a locally hosted VCS. Local also means you can orchestrate maintenance windows to avoid outages at critical phases.
darkwater 3y ago
> Most don't Define "most". There is a surprisingly high number of small/mid-sized companies which have dedicated people for this kind of things.
- mrweasel 3y ago
  Perhaps "many" would have been better. If you only count companies that view themselves as IT companies then number of companies with self hosted/managed solution grows, but if you include everything then I'd guess that more than 50% don't run these services internally. If you're every small company with one or two developers that doubles as the IT staff then the numbers add up pretty quickly.

snarkyturtle 3y ago

That's where I feel like it's actually pretty nice to not have CI tied to your source code. It's probably more expensive to use Travis/Circle but at least you don't have a single point of failure for deploys.

sleepychu 3y ago
Doesn't this give you 2xSPF? Or can I use my local copy of the source to kick of Travis/Circle?
- jpgvm 3y ago
  Ideally you don't ever rely on CI specific automation tooling to actually accomplish anything and instead just use it as a dumb "not my dev machine" to execute workflows. You should always engineer things so you can fall back to something akin to: ./scripts/deploy_the_things Ideally backed by a real build system and task engine ala Bazel, Gradle, whatever else floats your boat. It also means you are free to move between different runners/CI providers and just have to work out how to re-plumb secrets/whatever. GH actions/friends really provide minimal value, the fact they have convinced everyone that encoding task graphs into franken-YAML-bash is somehow good is one of the more egregious lies sold to the development community at large.
- snarkyturtle 3y ago
  Well, git ops might not be impacted on Github, usual Github outages tend to be through Actions/the site, not the actual git operations. Doesn't seem like you can use a local copy but you can use Gitbucket/Gitlab.