8 ms·

GitHub was having issues

121 points by Kiro 5y ago

brunooliv 5y ago
Just found this out now, which is why I decided to submit my current blog post as it is :D But I'll finish it soon enough eheh
Kiro 5y ago
OT but why did it capitalize the title like that? I submitted "GitHub is having issues". I don't see that for other submissions (unless people manually edit them afterwards). EDIT: Before the change to "GitHub was having issues" it had converted my original submission from "GitHub is having issues" to "GitHub Is Having Issues".
- eurekin 5y ago
  I first interpreted it as a puny way to advertise some improved github issue tracker
- rapnie 5y ago
  There are some rules for when this happens and when it does not. Don't know what they are exactly. Maybe the camelcase in GitHub may have triggered. Anyway, as you say, afterwards you can edit the title and give the desired case format.
- nikotom 5y ago
  From what I can tell from looking at older submissions, it might be doing it if the title is 4 words or less. Not sure why that would be the case though..
  - jbverschoor 5y ago
    Too much smarts in the wrong places imo..
- terramex 5y ago
  It is title case: https://en.wikipedia.org/wiki/Title_case https://en.wikipedia.org/wiki/Title_case
  - deleted 5y ago
    [deleted]
- Raed667 5y ago
  I also find myself writing titles in a "Title Case" like that. It is a habit I got from the days when I used to blog.
- petercooper 5y ago
  BTW you can see changes that occur like this on https://hackernewstitles.netlify.app/ https://hackernewstitles.netlify.app/ - it seems like the automated title casing is being overridden by mods later on, perhaps.
nik736 5y ago
What's the reason these Git services have so many issues all the time?
- cerved 5y ago
  possibly Azure?
  - danuker 5y ago
    Doesn't look like it, unless the status-page update is also affected :) https://status.azure.com/en-us/status https://status.azure.com/en-us/status
    - 8draco8 5y ago
      It seems that status pages never work properly in those kind of situations. The only reliable source of information seems to be twitter and HN
      - Manozco 5y ago
        Saying that twitter is the only source of truth for something usually means that this something is utterly broken... :D
        atatatat 5y ago
        ...or you're in the mind of your user.
      - yuriko 5y ago
        What if Twitter has an outage... Reminds me of the time when Twitter had an outage, because an FB outage made people go on Twitter to ask whether FB is down
  - MaxBarraclough 5y ago
    As far as I know GitHub doesn't use Azure at all, I think it's still hosted by Carpathia. https://github.com/holman/ama/issues/553 https://github.com/holman/ama/issues/553
rvz 5y ago
Here we go again [0] and again [1] and again [2] and again [3]. Time to reset the counter once again. Now everything there is degraded performance. Oh dear. Consider a self-hosted alternative or have one as a backup like I have said before. [0] https://news.ycombinator.com/item?id=27192869 https://news.ycombinator.com/item?id=27192869 [1] https://news.ycombinator.com/item?id=27172443 https://news.ycombinator.com/item?id=27172443 [2] https://news.ycombinator.com/item?id=26666843 https://news.ycombinator.com/item?id=26666843 [3] https://news.ycombinator.com/item?id=26667101 https://news.ycombinator.com/item?id=26667101
- dijit 5y ago
  Cue the age old argument chain about outsourcing responsibility vs having control of downtimes. I’m tired of them now, anyone using hosted solutions should know this by now and they’ve made that choice. Personally I believe people underestimate how easy it is to host your own services. But I’m sick of having the conversation over and over, if people aren’t willing to investigate then, what’s the saying: “you can lead a horse to water but you can’t make them drink”
  - benjaminjosephw 5y ago
    But the argument isn't static. GitHub has become much more of a target to large-scale attacks as it's grown in popularity and has become mission critical to so many companies and projects. Security risk isn't a one-and-done decision. It's a changing landscape and, arguably, there are far too many eggs in the GitHub basket for this not to be something people might want to think twice about now.
  - meibo 5y ago
    GitHub also arguably just has a good product that's a good fit for a lot of use-cases, regardless of if there's better options. Yeah, you might be able to self-host gitlab/gitea/etc but their free offering for organizations is so good that for my 30 commits/week open source project, I can deal with half an hour of downtime once in a while. Combined with free hosting, it's really not that much of a stretch that people are willing to put up with this, especially as there have been periods of time with less outages before, though that's been a while now.
beermonster 5y ago
For people using GitHub Actions how often do these types of incidents impact your CI/CD?
- toomanybeersies 5y ago
  Since most CI systems pull code from Github, it impacts other CI providers as well.
  - latch 5y ago
    Most? I'd wager that there's more self-hosted CI's running code from self-hosted repositories than from GitHub (however you measure it: e.g., lines of code, build time, etc)
    - yakubin 5y ago
      Same here. Self-hosted Jenkins pulling from self-hosted repos seems to be the standard in the companies I worked for and their partners/customers.
- hughrr 5y ago
  All the time. It’s not even that reliable when it’s up. Total embarrassment for us when we have to explain to management. If I started again I’d probably use something standalone, light weight and 100% self hosted ie GoCD. I’d change the operating model to keep GitHub as a collaboration place but not have any dependencies on it for pipeline. We have other issues as well like the GitHub flow mod being inadequate for maintaining stability on high traffic repos too. I’d pull from master into a production branch and run that off line entirely.
  - modo_mario 5y ago
    From what I heard it was fine until Microsoft bought it and decided to bring over .net code (and people) from azure devops to replace a bunch of the pre-existing github actions stuff for some reason.
Tainnor 5y ago
Maybe these are just a bunch of coincidences, but I have a feeling that GitHub is being poorly run recently. Apart from quite frequent uptime issues, it seems that Github Actions is pretty half-baked and they completely botched the migration of Dependabot (for example, take a look at this issue and especially, how it was handled: https://github.com/dependabot/dependabot-core/issues/3253 https://github.com/dependabot/dependabot-core/issues/3253)
- ljm 5y ago
  A shame they removed all their uptime graphs on the status page, since it's much harder to get a read on things now. However, scrolling back through the status page's history for the last year shows that there between 6 and 12 incidents every month, and it often hovers at the high end of that range. That has to be an utter misery for anyone who's on call.
  - capableweb 5y ago
    > A shame they removed all their uptime graphs on the status page, since it's much harder to get a read on things now. I've started using that (uptime/performance/latency graphs being removed from status pages) as a signal to for when companies feel embarrassed over their quality of service but don't have enough interest/resources/profits to fix it.
    - snakeboy 5y ago
      > interest/resources/profits It doesn't seem like Microsoft should have any of these issues. At most "interest", but even that should be a pretty easy sell for a service as massive as github.
      - shakna 5y ago
        As Microsoft has SLAs, it probably isn't in their interest to make it easy to tell, to begin with. Management would probably just see liability.
  - 5y ago
virgilp 5y ago
I would bet on it being related to their identity server. It seems to almost work in browser(s) where I was signed on - but in a new browser, it gives HTTP 500 after SSO.
rainboiboi 5y ago
Does it affect self-hosted runners as well?
- godsgardener 5y ago
  We're looking at switching to self-hosted runners, so I'd be very interested in hearing if it would shield us from some of Github downtime. We're also looking into Github enterprise for the IP allowlist feature which unfortunately means we'll have to migrate our whole CI/CD to self-hosted since Github-hosted runners don't work with this feature enabled.
user7878 5y ago
https://status.circleci.com/ https://status.circleci.com/ CircleCi having issue due to this
- toomanybeersies 5y ago
  Pretty much any CI provider will be "having issues", as most of the time builds are triggered by GitHub webhooks (and also pull code from GH)
abunuwas 5y ago
Can confirm. I almost go crazy a few minutes ago when I pushed some commits and couldn't see them appearing in the Pull Requests
pdimitar 5y ago
Things like this make me want to have more free time to start my own paid CI service. I can likely start from a few languages I know and take it from there. I wonder how hard it is to write your own CI/CD though? Anybody has thoughts on this?
- cortexio 5y ago
  for some people its very easy and for some its impossible.
- pm90 5y ago
  If you Google around you’ll see a crap ton of companies trying to do this. IMO the only “serious” threat to Jenkins are platform integrated services like Gitlab CI/CD, GitHub Actions, Google CloudBuild etc. Everybody else seems to continue to use Jenkins.
  - pdimitar 5y ago
    Good to know, thank you. I'm not scared of competition, I'm mostly wondering how viable and profitable such a business would be. I happen to believe people would pay for good specialized CI (say, Elixir-only, Rust-only, Golang-only etc.) but you're also quite right that the real value-add seems to be in integration.
    - dathinab 5y ago
      As long as you don't provide a huge degree of additional value you will be more likely to lose money through free plans which are needed to onboard people then you are to make money. While specialized CI sounds great you would need a good general purpose CI which also sports specialization. Ort people will need to change CI once they e.g. need to build and link against some FFI. Or have some project in another language. Which is quite a bit reason not to just it even if you currently only need specialized CI.
- rvanlaar 5y ago
  I recommend looking at buildbot[1]. It can be tailored towards your needs. I developed and maintain one for scummvm [2]. It has a steep learning curve, as with any sufficiently advanced system. [1] https://buildbot.net/ https://buildbot.net/ [2] https://john.scummvm.org https://john.scummvm.org
smcleod 5y ago
It was also impacting acting raw gists which is not noted in their report.
motiejus 5y ago
Plug: after almost a full year of hosting alone (for $50) I recently started using srht's builds, somewhat equivalent to github actions. It's been awesome so far. Look at the downloadable coverage.html: super easy to set up! https://builds.sr.ht/~motiejus/job/517389 https://builds.sr.ht/~motiejus/job/517389 And unlike any megacorp, Drew is approachable. I recommend having a look at it for anyone re-evaluating their software forge.

121 points by Kiro 5y ago

Just found this out now, which is why I decided to submit my current blog post as it is :D But I'll finish it soon enough eheh

Kiro 5y ago

OT but why did it capitalize the title like that? I submitted "GitHub is having issues". I don't see that for other submissions (unless people manually edit them afterwards). EDIT: Before the change to "GitHub was having issues" it had converted my original submission from "GitHub is having issues" to "GitHub Is Having Issues".

eurekin 5y ago
I first interpreted it as a puny way to advertise some improved github issue tracker
rapnie 5y ago
There are some rules for when this happens and when it does not. Don't know what they are exactly. Maybe the camelcase in GitHub may have triggered. Anyway, as you say, afterwards you can edit the title and give the desired case format.
nikotom 5y ago
From what I can tell from looking at older submissions, it might be doing it if the title is 4 words or less. Not sure why that would be the case though..
- jbverschoor 5y ago
  Too much smarts in the wrong places imo..
terramex 5y ago
It is title case: https://en.wikipedia.org/wiki/Title_case https://en.wikipedia.org/wiki/Title_case
- deleted 5y ago
  [deleted]
Raed667 5y ago
I also find myself writing titles in a "Title Case" like that. It is a habit I got from the days when I used to blog.
petercooper 5y ago
BTW you can see changes that occur like this on https://hackernewstitles.netlify.app/ https://hackernewstitles.netlify.app/ - it seems like the automated title casing is being overridden by mods later on, perhaps.

nik736 5y ago

What's the reason these Git services have so many issues all the time?

cerved 5y ago
possibly Azure?
- danuker 5y ago
  Doesn't look like it, unless the status-page update is also affected :) https://status.azure.com/en-us/status https://status.azure.com/en-us/status
  - 8draco8 5y ago
    It seems that status pages never work properly in those kind of situations. The only reliable source of information seems to be twitter and HN
    - Manozco 5y ago
      Saying that twitter is the only source of truth for something usually means that this something is utterly broken... :D
      - atatatat 5y ago
        ...or you're in the mind of your user.
    - yuriko 5y ago
      What if Twitter has an outage... Reminds me of the time when Twitter had an outage, because an FB outage made people go on Twitter to ask whether FB is down
- MaxBarraclough 5y ago
  As far as I know GitHub doesn't use Azure at all, I think it's still hosted by Carpathia. https://github.com/holman/ama/issues/553 https://github.com/holman/ama/issues/553

rvz 5y ago

Here we go again [0] and again [1] and again [2] and again [3]. Time to reset the counter once again. Now everything there is degraded performance. Oh dear. Consider a self-hosted alternative or have one as a backup like I have said before. [0] https://news.ycombinator.com/item?id=27192869 https://news.ycombinator.com/item?id=27192869 [1] https://news.ycombinator.com/item?id=27172443 https://news.ycombinator.com/item?id=27172443 [2] https://news.ycombinator.com/item?id=26666843 https://news.ycombinator.com/item?id=26666843 [3] https://news.ycombinator.com/item?id=26667101 https://news.ycombinator.com/item?id=26667101

dijit 5y ago
Cue the age old argument chain about outsourcing responsibility vs having control of downtimes. I’m tired of them now, anyone using hosted solutions should know this by now and they’ve made that choice. Personally I believe people underestimate how easy it is to host your own services. But I’m sick of having the conversation over and over, if people aren’t willing to investigate then, what’s the saying: “you can lead a horse to water but you can’t make them drink”
- benjaminjosephw 5y ago
  But the argument isn't static. GitHub has become much more of a target to large-scale attacks as it's grown in popularity and has become mission critical to so many companies and projects. Security risk isn't a one-and-done decision. It's a changing landscape and, arguably, there are far too many eggs in the GitHub basket for this not to be something people might want to think twice about now.
- meibo 5y ago
  GitHub also arguably just has a good product that's a good fit for a lot of use-cases, regardless of if there's better options. Yeah, you might be able to self-host gitlab/gitea/etc but their free offering for organizations is so good that for my 30 commits/week open source project, I can deal with half an hour of downtime once in a while. Combined with free hosting, it's really not that much of a stretch that people are willing to put up with this, especially as there have been periods of time with less outages before, though that's been a while now.

beermonster 5y ago

For people using GitHub Actions how often do these types of incidents impact your CI/CD?

toomanybeersies 5y ago
Since most CI systems pull code from Github, it impacts other CI providers as well.
- latch 5y ago
  Most? I'd wager that there's more self-hosted CI's running code from self-hosted repositories than from GitHub (however you measure it: e.g., lines of code, build time, etc)
  - yakubin 5y ago
    Same here. Self-hosted Jenkins pulling from self-hosted repos seems to be the standard in the companies I worked for and their partners/customers.
hughrr 5y ago
All the time. It’s not even that reliable when it’s up. Total embarrassment for us when we have to explain to management. If I started again I’d probably use something standalone, light weight and 100% self hosted ie GoCD. I’d change the operating model to keep GitHub as a collaboration place but not have any dependencies on it for pipeline. We have other issues as well like the GitHub flow mod being inadequate for maintaining stability on high traffic repos too. I’d pull from master into a production branch and run that off line entirely.
- modo_mario 5y ago
  From what I heard it was fine until Microsoft bought it and decided to bring over .net code (and people) from azure devops to replace a bunch of the pre-existing github actions stuff for some reason.

Tainnor 5y ago

Maybe these are just a bunch of coincidences, but I have a feeling that GitHub is being poorly run recently. Apart from quite frequent uptime issues, it seems that Github Actions is pretty half-baked and they completely botched the migration of Dependabot (for example, take a look at this issue and especially, how it was handled: https://github.com/dependabot/dependabot-core/issues/3253 https://github.com/dependabot/dependabot-core/issues/3253)

ljm 5y ago
A shame they removed all their uptime graphs on the status page, since it's much harder to get a read on things now. However, scrolling back through the status page's history for the last year shows that there between 6 and 12 incidents every month, and it often hovers at the high end of that range. That has to be an utter misery for anyone who's on call.
- capableweb 5y ago
  > A shame they removed all their uptime graphs on the status page, since it's much harder to get a read on things now. I've started using that (uptime/performance/latency graphs being removed from status pages) as a signal to for when companies feel embarrassed over their quality of service but don't have enough interest/resources/profits to fix it.
  - snakeboy 5y ago
    > interest/resources/profits It doesn't seem like Microsoft should have any of these issues. At most "interest", but even that should be a pretty easy sell for a service as massive as github.
    - shakna 5y ago
      As Microsoft has SLAs, it probably isn't in their interest to make it easy to tell, to begin with. Management would probably just see liability.
- 5y ago

virgilp 5y ago

I would bet on it being related to their identity server. It seems to almost work in browser(s) where I was signed on - but in a new browser, it gives HTTP 500 after SSO.

rainboiboi 5y ago

Does it affect self-hosted runners as well?

godsgardener 5y ago
We're looking at switching to self-hosted runners, so I'd be very interested in hearing if it would shield us from some of Github downtime. We're also looking into Github enterprise for the IP allowlist feature which unfortunately means we'll have to migrate our whole CI/CD to self-hosted since Github-hosted runners don't work with this feature enabled.

user7878 5y ago

https://status.circleci.com/ https://status.circleci.com/ CircleCi having issue due to this

toomanybeersies 5y ago
Pretty much any CI provider will be "having issues", as most of the time builds are triggered by GitHub webhooks (and also pull code from GH)

abunuwas 5y ago

Can confirm. I almost go crazy a few minutes ago when I pushed some commits and couldn't see them appearing in the Pull Requests

pdimitar 5y ago

Things like this make me want to have more free time to start my own paid CI service. I can likely start from a few languages I know and take it from there. I wonder how hard it is to write your own CI/CD though? Anybody has thoughts on this?

cortexio 5y ago
for some people its very easy and for some its impossible.
pm90 5y ago
If you Google around you’ll see a crap ton of companies trying to do this. IMO the only “serious” threat to Jenkins are platform integrated services like Gitlab CI/CD, GitHub Actions, Google CloudBuild etc. Everybody else seems to continue to use Jenkins.
- pdimitar 5y ago
  Good to know, thank you. I'm not scared of competition, I'm mostly wondering how viable and profitable such a business would be. I happen to believe people would pay for good specialized CI (say, Elixir-only, Rust-only, Golang-only etc.) but you're also quite right that the real value-add seems to be in integration.
  - dathinab 5y ago
    As long as you don't provide a huge degree of additional value you will be more likely to lose money through free plans which are needed to onboard people then you are to make money. While specialized CI sounds great you would need a good general purpose CI which also sports specialization. Ort people will need to change CI once they e.g. need to build and link against some FFI. Or have some project in another language. Which is quite a bit reason not to just it even if you currently only need specialized CI.
rvanlaar 5y ago
I recommend looking at buildbot[1]. It can be tailored towards your needs. I developed and maintain one for scummvm [2]. It has a steep learning curve, as with any sufficiently advanced system. [1] https://buildbot.net/ https://buildbot.net/ [2] https://john.scummvm.org https://john.scummvm.org

smcleod 5y ago

It was also impacting acting raw gists which is not noted in their report.

motiejus 5y ago

Plug: after almost a full year of hosting alone (for $50) I recently started using srht's builds, somewhat equivalent to github actions. It's been awesome so far. Look at the downloadable coverage.html: super easy to set up! https://builds.sr.ht/~motiejus/job/517389 https://builds.sr.ht/~motiejus/job/517389 And unlike any megacorp, Drew is approachable. I recommend having a look at it for anyone re-evaluating their software forge.