6 ms·

For people using GitHub Actions how often do these types of incidents impact your CI/CD?

by beermonster 5y ago

toomanybeersies 5y ago
Since most CI systems pull code from Github, it impacts other CI providers as well.
- latch 5y ago
  Most? I'd wager that there's more self-hosted CI's running code from self-hosted repositories than from GitHub (however you measure it: e.g., lines of code, build time, etc)
  - yakubin 5y ago
    Same here. Self-hosted Jenkins pulling from self-hosted repos seems to be the standard in the companies I worked for and their partners/customers.
hughrr 5y ago
All the time. It’s not even that reliable when it’s up. Total embarrassment for us when we have to explain to management. If I started again I’d probably use something standalone, light weight and 100% self hosted ie GoCD. I’d change the operating model to keep GitHub as a collaboration place but not have any dependencies on it for pipeline. We have other issues as well like the GitHub flow mod being inadequate for maintaining stability on high traffic repos too. I’d pull from master into a production branch and run that off line entirely.
- modo_mario 5y ago
  From what I heard it was fine until Microsoft bought it and decided to bring over .net code (and people) from azure devops to replace a bunch of the pre-existing github actions stuff for some reason.
  - bbrree66 5y ago
    What actions stuff existed before the acquisition?
  - tester34 5y ago
    You're saying that .NET is not capable or bad when it comes to handling such a tools like building systems?
    - Jenk 5y ago
      I infered that it is more about Microsoft and their management of software development than I did about it being anything technical related.
    - prepend 5y ago
      I read this as more a situation where an existing architecture got stuff shoehorned in for non-technology reasons. I don’t think there’s a problem with .NET inherently. But swapping out components for new tech stacks for no user or tech reason is likely to introduce bugs and at best will negatively impact schedule. Although it would be interesting to hear any new company that chose .NET over other alternatives. I really likes how Spolsky explained why they used the MS stack [0] to build stackoverflow, among other things. [0] https://channel9.msdn.com/Blogs/Seth-Juarez/Joel-Spolsky-Talks-About-Building-Stack-Overflow-and-Values-Developers-Care-About https://channel9.msdn.com/Blogs/Seth-Juarez/Joel-Spolsky-Tal...
- Jenk 5y ago
  > GoCD Don't. Just don't. I'm assuming that the fundamental flaws* of GoCD haven't been fixed, based on the knowledge that in 2015 they (ThoughtWorks) still hadn't addressed them, and I think 5-6 years isn't long enough for them to be addressed. *Pipelines, as modeled in GoCD, were so inflexible that any kind of flow resulted in having many, many, pipelines configured. To the point that the number pipelines you operate and maintain is actually the multiple of any fork in logic you need.
  - hughrr 5y ago
    Thanks for the heads up. I will avoid that then!
  - vergessenmir 5y ago
    Hate to pile on GoCD. But I second this. In 2015 they were a contender when Jenkins was showing it's age and Gitlab was in its infancy. Development was fairly stagnant and felt 'closed' to external contributions. I was hopeful, but ultimately disappointed.
    - hughrr 5y ago
      Thanks. This sort of stuff is why I come here :)
onion2k 5y ago
Very rarely, and (so far) not in any meaningful way. Having to wait a few hours for a CI process isn't a problem 99% of the time. Deploys are affected by issues on AWS and Azure far more often than GH in my experience. I'd probably change my mind if I ever get unlucky enough to need to deploy a critical hotfix at the same time GH is unavailable though.
- dry_soup 5y ago
  If you're OK with waiting multiple hours for CI, I suppose these issues wouldn't affect you very much. Where I work we make relatively small PRs and deploy to production tens of times per day. The recent outages have been a nightmare for us.
  - atatatat 5y ago
    > deploy to production tens of times per day Any chance you're willing to share more? It sounds like there's something interesting to learn here.
    - wjossey 5y ago
      Not the OP but in a continuous release process you treat main (master) as “the source of truth” and merges to main trigger a new release to production. At the company I work at, we have about 160 full time engineers with about half our repos on continuous release. We average right now somewhere around 50-55 releases per day, with some hot repos getting up towards 20 releases per day on their own. The point of small releases is that it makes rollbacks super easy (you always know what broke), roll forwards easy (you can easily test post rollout to find out if things aren’t functioning as expected), and you can mentally move on to your next task sooner because your code gets released when you want it released- not when the next “big develop release” happens. To pull this off requires a few things. You need good tests that run against PRs. You need fast deploys (under 10 minutes is a good starting point, but get them faster if you can). And, ideally, you have ephemeral environments for each PR so that changes can be tested fully in isolation without requiring a shared staging environment. This is legit what I do for a living, so AMA.
andrew_ 5y ago
Just about a monthly occurrence now. Harks back to the days of Travis it's frequent downtime.
toper-centage 5y ago
It's been a weekly occurance for months now.

For people using GitHub Actions how often do these types of incidents impact your CI/CD?

toomanybeersies 5y ago

Since most CI systems pull code from Github, it impacts other CI providers as well.

latch 5y ago
Most? I'd wager that there's more self-hosted CI's running code from self-hosted repositories than from GitHub (however you measure it: e.g., lines of code, build time, etc)
- yakubin 5y ago
  Same here. Self-hosted Jenkins pulling from self-hosted repos seems to be the standard in the companies I worked for and their partners/customers.

hughrr 5y ago

All the time. It’s not even that reliable when it’s up. Total embarrassment for us when we have to explain to management. If I started again I’d probably use something standalone, light weight and 100% self hosted ie GoCD. I’d change the operating model to keep GitHub as a collaboration place but not have any dependencies on it for pipeline. We have other issues as well like the GitHub flow mod being inadequate for maintaining stability on high traffic repos too. I’d pull from master into a production branch and run that off line entirely.

modo_mario 5y ago
From what I heard it was fine until Microsoft bought it and decided to bring over .net code (and people) from azure devops to replace a bunch of the pre-existing github actions stuff for some reason.
- bbrree66 5y ago
  What actions stuff existed before the acquisition?
- tester34 5y ago
  You're saying that .NET is not capable or bad when it comes to handling such a tools like building systems?
  - Jenk 5y ago
    I infered that it is more about Microsoft and their management of software development than I did about it being anything technical related.
  - prepend 5y ago
    I read this as more a situation where an existing architecture got stuff shoehorned in for non-technology reasons. I don’t think there’s a problem with .NET inherently. But swapping out components for new tech stacks for no user or tech reason is likely to introduce bugs and at best will negatively impact schedule. Although it would be interesting to hear any new company that chose .NET over other alternatives. I really likes how Spolsky explained why they used the MS stack [0] to build stackoverflow, among other things. [0] https://channel9.msdn.com/Blogs/Seth-Juarez/Joel-Spolsky-Talks-About-Building-Stack-Overflow-and-Values-Developers-Care-About https://channel9.msdn.com/Blogs/Seth-Juarez/Joel-Spolsky-Tal...
Jenk 5y ago
> GoCD Don't. Just don't. I'm assuming that the fundamental flaws* of GoCD haven't been fixed, based on the knowledge that in 2015 they (ThoughtWorks) still hadn't addressed them, and I think 5-6 years isn't long enough for them to be addressed. *Pipelines, as modeled in GoCD, were so inflexible that any kind of flow resulted in having many, many, pipelines configured. To the point that the number pipelines you operate and maintain is actually the multiple of any fork in logic you need.
- hughrr 5y ago
  Thanks for the heads up. I will avoid that then!
- vergessenmir 5y ago
  Hate to pile on GoCD. But I second this. In 2015 they were a contender when Jenkins was showing it's age and Gitlab was in its infancy. Development was fairly stagnant and felt 'closed' to external contributions. I was hopeful, but ultimately disappointed.
  - hughrr 5y ago
    Thanks. This sort of stuff is why I come here :)

onion2k 5y ago

Very rarely, and (so far) not in any meaningful way. Having to wait a few hours for a CI process isn't a problem 99% of the time. Deploys are affected by issues on AWS and Azure far more often than GH in my experience. I'd probably change my mind if I ever get unlucky enough to need to deploy a critical hotfix at the same time GH is unavailable though.

dry_soup 5y ago
If you're OK with waiting multiple hours for CI, I suppose these issues wouldn't affect you very much. Where I work we make relatively small PRs and deploy to production tens of times per day. The recent outages have been a nightmare for us.
- atatatat 5y ago
  > deploy to production tens of times per day Any chance you're willing to share more? It sounds like there's something interesting to learn here.
  - wjossey 5y ago
    Not the OP but in a continuous release process you treat main (master) as “the source of truth” and merges to main trigger a new release to production. At the company I work at, we have about 160 full time engineers with about half our repos on continuous release. We average right now somewhere around 50-55 releases per day, with some hot repos getting up towards 20 releases per day on their own. The point of small releases is that it makes rollbacks super easy (you always know what broke), roll forwards easy (you can easily test post rollout to find out if things aren’t functioning as expected), and you can mentally move on to your next task sooner because your code gets released when you want it released- not when the next “big develop release” happens. To pull this off requires a few things. You need good tests that run against PRs. You need fast deploys (under 10 minutes is a good starting point, but get them faster if you can). And, ideally, you have ephemeral environments for each PR so that changes can be tested fully in isolation without requiring a shared staging environment. This is legit what I do for a living, so AMA.

andrew_ 5y ago

Just about a monthly occurrence now. Harks back to the days of Travis it's frequent downtime.

toper-centage 5y ago

It's been a weekly occurance for months now.