9 ms·

GitHub Is Down Again

149 points by derkoe 4y ago

deleted 4y ago
[deleted]
daenz 4y ago
Good thing I spent time refactoring my Github workflow files to put all of the meaningful logic into separate shell scripts, so that I can still deploy without relying on Github Actions to be available.
- yonixw 4y ago
  After a few outages and bugs from multi AAA cloud providers, I do the same. Rely most on bash scripts that are portable and only use the cloud platform for offloading builds and secrets management. Every built in feature always comes to bite you back in the ass when least expected.
mritzmann 4y ago
#HugOps to GitHub Engineers! Such things are usually stressful, especially before the weekend. In addition, in some European countries today is a bridge day (day between a holiday and the weekend, many people take off on such a day).
- tssva 4y ago
  In the US this coming Monday is Memorial Day. Memorial Day weekend is a very popular travel time and many people will be taking today off to extend those trips an additional day. Hopefully no GitHub engineers will be forced to sacrifice such plans.
  - joshvm 4y ago
    I would imagine this is mostly the domain of on-call engineers/SREs (wasn't this historically a problem that was fixed by a reboot?) and you have to assume that you're going to be called at 4AM on a Saturday morning. So even booking vacation when you're on shift is risky, and unfortunately that means someone has to work public holidays. Good companies will give you a day off in lieu. Having done on-call work, the most annoying bugs are the ones that you feel could be automated (especially when the pager goes off when you're asleep). Not to say that I hope these guys get called out, but it's the nature of the job.
    - tssva 4y ago
      I have managed many on-call people during my career. I hated whenever one of my guys actually had to answer a call. It meant we had failed somewhere and needed to get better. I certainly didn't accept it as just the nature of the job.
      - nix23 4y ago
        >It meant we had failed somewhere and needed to get better. That is the work-ethic i would love to see from all manager. Excellent attitude to life/work.
hamasho 4y ago
No problem. It's almost Friday's 5 PM in Japan right now. It's time to stop working and go out for some beer.
- traveler01 4y ago
  Damn, I'm starting now
- holografix 4y ago
  Very jealous when is Japan opening its borders to tourists?!
  - innocenat 4y ago
    It's already open now*. * Subject to a very strict definition of tourism.
  - 2143 4y ago
    > Very jealous It isn't always Friday 5 PM in Japan.
- jalino23 4y ago
  what is it like programming in japan? do language look like japanese ?
  - uranusjr 4y ago
    Honestly this is so cringe I can’t tell if you’re being sarcastic
    - Majestic121 4y ago
      You'd be surprised what people around the world do : https://en.wikipedia.org/wiki/Non-English-based_programming_languages https://en.wikipedia.org/wiki/Non-English-based_programming_... I don't think any of those are used for anything serious though (at least I hope not)
deleted 4y ago
[deleted]
fyhn 4y ago
Does anyone have long-term stats for GitHub status incidents over the last few years or so?
- ngalaiko 4y ago
  do you think this is incorrect https://www.githubstatus.com/history https://www.githubstatus.com/history?
  - fyhn 4y ago
    Not at all. I'm looking for something that shows incident frequency over time, without scrolling through a bunch of pages.
- oefrha 4y ago
  They used to have graphs showing incident frequency right on the main status page, and a detailed uptime page. Guess those got retired after things got real embarrassing. https://web.archive.org/web/20200101000934/https://www.githubstatus.com/ https://web.archive.org/web/20200101000934/https://www.githu...
  - fyhn 4y ago
    That's it! I had a feeling I'd seen it before.
derkoe 4y ago
Seems to be up again - but there are still some performance issues. The main page was down for approximately 15 minutes
quaintdev 4y ago
And that is why you host your own mirror of all your repositories. I have Gitea mirror hosted on Raspberry PI connected to 7200RPM HDD at my father's house. I also host other cool apps like paperless, jellyfish, etc,. It's awesome. Self hosting is cheap people, you just need to invest some time in it.
- deleted 4y ago
  [deleted]
- soco 4y ago
  Where time might be the more expensive item.
- deleted 4y ago
  [deleted]
- lamontcg 4y ago
  I don't think I've ever seen a GitHub outage that has actually impacted my ability to deliver software that week. I rarely need to do something time critical with a window of only a few hours and can't shuffle what I'm doing around.
kkfx 4y ago
Honestly anyone can have issue, that's not a point. The point is the amplitude of that issue, or why do you FLOSS devs want GH instead of develop more modern Email clients to effectively develop over mailing-lists and why not usenet? We have some very modern from 50+ years ago, perhaps it's time to adjust their aesthetic and rediscover their power?
- plonk 4y ago
  I don't think mailing lists can do CI/CD.
  - inferiorhuman 4y ago
    Why not? SMTP is nothing if not a robust queuing system and surely procmail or your MDA of choice should be able to rig something up.
    - plonk 4y ago
      Current mailing list software can't. Of course you could control a CI/CD system with a Nokia 3310 if you wanted, but why spend the effort when CI/CD systems already have all the right features? Unless OP is actually advocating for self-hosting.
      - klibertp 4y ago
        > Of course you could control a CI/CD system with a Nokia 3310 I have a feeling that such a system actually exists somewhere in this vast world, and probably even operates to this day...
    - kkfx 4y ago
      I've learnt enough to hate modern DevOps, where the Ops is not a contraction of "operation" but a literal "ops!" where a casual dev mistakes have created a devastating cascading effect that no one know what to do. Oh, BTW I've switched time ago from procmail to MailDrop, it's more flexible and more maintained :D
  - 4y ago
alaricus 4y ago
This is why centralized services are a bad idea.
biermic 4y ago
Seems Github is imitating Bitbucket :)
- c0balt 4y ago
  A nodern company needs to follow all Industry trends. /s
MaKey 4y ago
Resolved after 1 hour and 21 minutes, that's quick.
executesorder66 4y ago
When I go to the link the page keeps refreshing, and I get duplicate messages at the top of the page that says "Hang on there partner, You're subscribing too fast (give it a quick second and try again)." There is also a native browser notification that says "Oops, something went wrong while subscribing you! Let's reload the page and try again." Is anyone else getting that? I haven't "subscribed" to any thing. I'm using the latest version of Firefox with the uBlock Origin addon.
kome 4y ago
I heard sourceforge is doing great
- mdaniel 4y ago
  > If a tree^W sourceforge fell in the forest and no one was using it, did it really fall?
gtirloni 4y ago
There is zero new knowledge is these threads. I understand the HN crowd relies heavily on GitHub but it feels like Twitter. Anyway, I'm not adding anything too so...
Roybot 4y ago
Self host gitlab
anoojb 4y ago
How do you prevent this from impacting your systems? What kind of features can GitHub invest in to make it easy for you to insulate from downtime?
nrvn 4y ago
As usual the advice is: Duplicate your efforts and do not rely on a single git hosting provider! Gitlab.org is there. Use it. There are multiple ways to sync workflows and git repositories
vanous 4y ago
Don't be afraid to use another code forge, there are many alternatives. People will come over if your project is interesting.

149 points by derkoe 4y ago

deleted 4y ago

[deleted]

Good thing I spent time refactoring my Github workflow files to put all of the meaningful logic into separate shell scripts, so that I can still deploy without relying on Github Actions to be available.

yonixw 4y ago
After a few outages and bugs from multi AAA cloud providers, I do the same. Rely most on bash scripts that are portable and only use the cloud platform for offloading builds and secrets management. Every built in feature always comes to bite you back in the ass when least expected.

mritzmann 4y ago

#HugOps to GitHub Engineers! Such things are usually stressful, especially before the weekend. In addition, in some European countries today is a bridge day (day between a holiday and the weekend, many people take off on such a day).

tssva 4y ago
In the US this coming Monday is Memorial Day. Memorial Day weekend is a very popular travel time and many people will be taking today off to extend those trips an additional day. Hopefully no GitHub engineers will be forced to sacrifice such plans.
- joshvm 4y ago
  I would imagine this is mostly the domain of on-call engineers/SREs (wasn't this historically a problem that was fixed by a reboot?) and you have to assume that you're going to be called at 4AM on a Saturday morning. So even booking vacation when you're on shift is risky, and unfortunately that means someone has to work public holidays. Good companies will give you a day off in lieu. Having done on-call work, the most annoying bugs are the ones that you feel could be automated (especially when the pager goes off when you're asleep). Not to say that I hope these guys get called out, but it's the nature of the job.
  - tssva 4y ago
    I have managed many on-call people during my career. I hated whenever one of my guys actually had to answer a call. It meant we had failed somewhere and needed to get better. I certainly didn't accept it as just the nature of the job.
    - nix23 4y ago
      >It meant we had failed somewhere and needed to get better. That is the work-ethic i would love to see from all manager. Excellent attitude to life/work.

hamasho 4y ago

No problem. It's almost Friday's 5 PM in Japan right now. It's time to stop working and go out for some beer.

traveler01 4y ago
Damn, I'm starting now
holografix 4y ago
Very jealous when is Japan opening its borders to tourists?!
- innocenat 4y ago
  It's already open now*. * Subject to a very strict definition of tourism.
- 2143 4y ago
  > Very jealous It isn't always Friday 5 PM in Japan.
jalino23 4y ago
what is it like programming in japan? do language look like japanese ?
- uranusjr 4y ago
  Honestly this is so cringe I can’t tell if you’re being sarcastic
  - Majestic121 4y ago
    You'd be surprised what people around the world do : https://en.wikipedia.org/wiki/Non-English-based_programming_languages https://en.wikipedia.org/wiki/Non-English-based_programming_... I don't think any of those are used for anything serious though (at least I hope not)

deleted 4y ago

[deleted]

fyhn 4y ago

Does anyone have long-term stats for GitHub status incidents over the last few years or so?

ngalaiko 4y ago
do you think this is incorrect https://www.githubstatus.com/history https://www.githubstatus.com/history?
- fyhn 4y ago
  Not at all. I'm looking for something that shows incident frequency over time, without scrolling through a bunch of pages.
oefrha 4y ago
They used to have graphs showing incident frequency right on the main status page, and a detailed uptime page. Guess those got retired after things got real embarrassing. https://web.archive.org/web/20200101000934/https://www.githubstatus.com/ https://web.archive.org/web/20200101000934/https://www.githu...
- fyhn 4y ago
  That's it! I had a feeling I'd seen it before.

derkoe 4y ago

Seems to be up again - but there are still some performance issues. The main page was down for approximately 15 minutes

quaintdev 4y ago

And that is why you host your own mirror of all your repositories. I have Gitea mirror hosted on Raspberry PI connected to 7200RPM HDD at my father's house. I also host other cool apps like paperless, jellyfish, etc,. It's awesome. Self hosting is cheap people, you just need to invest some time in it.

deleted 4y ago
[deleted]
soco 4y ago
Where time might be the more expensive item.
deleted 4y ago
[deleted]
lamontcg 4y ago
I don't think I've ever seen a GitHub outage that has actually impacted my ability to deliver software that week. I rarely need to do something time critical with a window of only a few hours and can't shuffle what I'm doing around.

kkfx 4y ago

Honestly anyone can have issue, that's not a point. The point is the amplitude of that issue, or why do you FLOSS devs want GH instead of develop more modern Email clients to effectively develop over mailing-lists and why not usenet? We have some very modern from 50+ years ago, perhaps it's time to adjust their aesthetic and rediscover their power?

plonk 4y ago
I don't think mailing lists can do CI/CD.
- inferiorhuman 4y ago
  Why not? SMTP is nothing if not a robust queuing system and surely procmail or your MDA of choice should be able to rig something up.
  - plonk 4y ago
    Current mailing list software can't. Of course you could control a CI/CD system with a Nokia 3310 if you wanted, but why spend the effort when CI/CD systems already have all the right features? Unless OP is actually advocating for self-hosting.
    - klibertp 4y ago
      > Of course you could control a CI/CD system with a Nokia 3310 I have a feeling that such a system actually exists somewhere in this vast world, and probably even operates to this day...
  - kkfx 4y ago
    I've learnt enough to hate modern DevOps, where the Ops is not a contraction of "operation" but a literal "ops!" where a casual dev mistakes have created a devastating cascading effect that no one know what to do. Oh, BTW I've switched time ago from procmail to MailDrop, it's more flexible and more maintained :D
- 4y ago

alaricus 4y ago

This is why centralized services are a bad idea.

biermic 4y ago

Seems Github is imitating Bitbucket :)

c0balt 4y ago
A nodern company needs to follow all Industry trends. /s

MaKey 4y ago

Resolved after 1 hour and 21 minutes, that's quick.

executesorder66 4y ago

When I go to the link the page keeps refreshing, and I get duplicate messages at the top of the page that says "Hang on there partner, You're subscribing too fast (give it a quick second and try again)." There is also a native browser notification that says "Oops, something went wrong while subscribing you! Let's reload the page and try again." Is anyone else getting that? I haven't "subscribed" to any thing. I'm using the latest version of Firefox with the uBlock Origin addon.

kome 4y ago

I heard sourceforge is doing great

mdaniel 4y ago
> If a tree^W sourceforge fell in the forest and no one was using it, did it really fall?

gtirloni 4y ago

There is zero new knowledge is these threads. I understand the HN crowd relies heavily on GitHub but it feels like Twitter. Anyway, I'm not adding anything too so...

Roybot 4y ago

Self host gitlab

anoojb 4y ago

How do you prevent this from impacting your systems? What kind of features can GitHub invest in to make it easy for you to insulate from downtime?

nrvn 4y ago

As usual the advice is: Duplicate your efforts and do not rely on a single git hosting provider! Gitlab.org is there. Use it. There are multiple ways to sync workflows and git repositories

vanous 4y ago

Don't be afraid to use another code forge, there are many alternatives. People will come over if your project is interesting.