6 ms·

GitHub Status – Incident on 2019-03-12

57 points by fniephaus 8y ago

howiroll 8y ago
tyPiCaL rUbY
- deleted 8y ago
  [deleted]
rinchik 8y ago
Yep. Can't view newly open PRs, also random GH related failures in CI
deleted 8y ago
[deleted]
lhorak 8y ago
Seem working alright... hmmm
MuffinFlavored 8y ago
Does anybody else feel GH has a much higher than normal incident rate?
- ashelmire 8y ago
  Nah, seems pretty normal. Compare to Netflix, Hulu, Heroku, etc, which have some outages nearly daily. This was just degraded performance for two hours, and the only more serious one I can recall was pretty recent.
  - MuffinFlavored 8y ago
    dumb question but, why can't the teams of these super large companies who are probably getting paid top dollar get their deployment cycle right down to the point where they aren't causing interruptions?
- yjftsjthsd-h 8y ago
  Unless they under-report, we shouldn't need to feel anything; we can just check https://www.githubstatus.com/history https://www.githubstatus.com/history and objectively look at frequency. Of course, some services do under-report; dunno if GH does or not.
  - zenexer 8y ago
    The overwhelming majority seem to under-report these days. I can’t tell you how many conversations I’ve had that go like this: — Me: We’re having issues on our site, and we’ve narrowed it down to your service. Can you confirm that you’ve had a partial outage for the last three hours? Them: Yes, about 25% of API calls are failing. Me: Why is your status page green? Them: Our team has decided it doesn’t warrant a status update. — When this happens, we tend to switch services providers fairly quickly. If you’re not going to be honest with us, we’re stuck wasting our time troubleshooting. We don’t care if you have occasional outages—it happens; that’s why we have fallback providers. But you need to tell us when you’re down. tl;dr: Your status page is not an advertisement for your “amazing” uptime. It’s to make my life easier and save me time/money. Use it for its intended purpose.
- colinbartlett 8y ago
  Depends on what you mean by normal but compared to many of the status pages I am tracking, GitHub has a pretty low incident rate (though an uptick very recently): https://statusgator.com/services/github https://statusgator.com/services/github
Legogris 8y ago
This could be just a coincidence, but just now I also noticed HN was down as well as several global services (non-tech related stuff) we integrate with that are all timing out. Some deeper infrastructure issues at hand?
zilian 8y ago
Deploy to github pages delayed for more than an hour here
mcrider 8y ago
Odd side effect -- I've received three duplicate 'Github Explore' emails about every hour this morning.
honopu 8y ago
I wasn't getting 2fa notifications either, haven't tried lately but switched to google authenticator as a precaution against this happening in the future.
Vervious 8y ago
Travis CI git clone over http is borked as a result. https://www.traviscistatus.com/ https://www.traviscistatus.com/
justinjlynn 8y ago
Update 1: Info. Update 2: Same info Again. Update 3: The same info again phrased differently. What's the point of giving regular "updates" when you don't add any information? Is it just to tick a managerial box somewhere?
- Operyl 8y ago
  Some clients will feel ignored, even if you don't regurgitate the same stuff over and over again while there's no new useful information to disseminate (or not had the time to properly write it up). At my job, I wrote a small slack bot that'll ping the person who is dealing with the status page/customer impact, it'll ask "I'm posting an automated update in 15 minutes, would you like that or do you have a full size one?" That way, in the heat of a fire, nobody forgets about the client's status page updates. It works rather well.
- jressey 8y ago
  If you're on a flight, do you not appreciate when the pilot updates you, even if he says "We are still investigating the maintenance issue?"
  - gerardnll 8y ago
    It's funny because if you think it carefully it's like you think they look at the problem for a while and stop until you ask again for updates. haha Like you need to keep asking for them to work on it. Sounds like an oldschool boss expecting you to be on your chair even if that doesn't mean you're working.
  - samirm 8y ago
    No, because there's literally nothing I could do to change the situation...
- abathur 8y ago
  This is probably SLA compliance stuff (stipulations for how quickly they have to provide notice of issues, how often they're obliged to provide updates, and how quickly they'll fix an issue). These stipulations, obviously, don't say much about the quality of the notice or updates. So, yes--ticking a box (but a box that will matter for securing and maintaining contracts with larger organizations).
msghacq 8y ago
I’ve been hearing from friends at GitHub and MS that things are on fire internally. The GitHub employees are in semi-revolt because they think Nat is integrating too fast with MS. Many people have quit or are being pushed out. It may actually be for the best because the post-founder/pre-buyout team was pretty mediocre. Nat is also trying to crack down on some of the more overt activism in the company. They do “eq” screenings for political fit during the interview process and he’s trying to get them to stop. They also have a brand new head of product and engineering. Product was taken from Jason Warner about two months ago and placed directly under Nat. Sounds really bad and I don’t expect much turn around.
- ambulancechaser 8y ago
  > They do “eq” screenings for political fit during the interview process and he’s trying to get them to stop. This sounds crazy to me. My last company would freak out if the dev team got together and discussed the person's politics and if we thought they should be hired or not based on that. I hope they do stop this. The more i think about this the crazier it sounds. Imagine a Houston oil consultancy firm discussing a candidate who applied and asking "when we talked about lowering taxes, did the inflection in his voice sound real or feigned?". This is what an "old boys club" looks like.
  - debt 8y ago
    "the person's politics" I interpreted it as a how a person deals with internal politics not what their political views are. the latter would most definitely be illegal.
    - ambulancechaser 8y ago
      from the OP > It stands for “emotional quotient” and they ask you a bunch of questions about diversity, feminism... They watch your responses for tone and vocabulary and can veto your hire if they deem you insufficiently enthusiastic. It sounds very much like your opinions on current political topics.
    - 8y ago
elesbao 8y ago
that's a shitty incident report and a sad thread if there's this kind of activism internally while shit is on fire.

57 points by fniephaus 8y ago

howiroll 8y ago

tyPiCaL rUbY

deleted 8y ago
[deleted]

rinchik 8y ago

Yep. Can't view newly open PRs, also random GH related failures in CI

deleted 8y ago

[deleted]

lhorak 8y ago

Seem working alright... hmmm

MuffinFlavored 8y ago

Does anybody else feel GH has a much higher than normal incident rate?

ashelmire 8y ago
Nah, seems pretty normal. Compare to Netflix, Hulu, Heroku, etc, which have some outages nearly daily. This was just degraded performance for two hours, and the only more serious one I can recall was pretty recent.
- MuffinFlavored 8y ago
  dumb question but, why can't the teams of these super large companies who are probably getting paid top dollar get their deployment cycle right down to the point where they aren't causing interruptions?
yjftsjthsd-h 8y ago
Unless they under-report, we shouldn't need to feel anything; we can just check https://www.githubstatus.com/history https://www.githubstatus.com/history and objectively look at frequency. Of course, some services do under-report; dunno if GH does or not.
- zenexer 8y ago
  The overwhelming majority seem to under-report these days. I can’t tell you how many conversations I’ve had that go like this: — Me: We’re having issues on our site, and we’ve narrowed it down to your service. Can you confirm that you’ve had a partial outage for the last three hours? Them: Yes, about 25% of API calls are failing. Me: Why is your status page green? Them: Our team has decided it doesn’t warrant a status update. — When this happens, we tend to switch services providers fairly quickly. If you’re not going to be honest with us, we’re stuck wasting our time troubleshooting. We don’t care if you have occasional outages—it happens; that’s why we have fallback providers. But you need to tell us when you’re down. tl;dr: Your status page is not an advertisement for your “amazing” uptime. It’s to make my life easier and save me time/money. Use it for its intended purpose.
colinbartlett 8y ago
Depends on what you mean by normal but compared to many of the status pages I am tracking, GitHub has a pretty low incident rate (though an uptick very recently): https://statusgator.com/services/github https://statusgator.com/services/github

Legogris 8y ago

This could be just a coincidence, but just now I also noticed HN was down as well as several global services (non-tech related stuff) we integrate with that are all timing out. Some deeper infrastructure issues at hand?

zilian 8y ago

Deploy to github pages delayed for more than an hour here

mcrider 8y ago

Odd side effect -- I've received three duplicate 'Github Explore' emails about every hour this morning.

honopu 8y ago

I wasn't getting 2fa notifications either, haven't tried lately but switched to google authenticator as a precaution against this happening in the future.

Vervious 8y ago

Travis CI git clone over http is borked as a result. https://www.traviscistatus.com/ https://www.traviscistatus.com/

justinjlynn 8y ago

Update 1: Info. Update 2: Same info Again. Update 3: The same info again phrased differently. What's the point of giving regular "updates" when you don't add any information? Is it just to tick a managerial box somewhere?

Operyl 8y ago
Some clients will feel ignored, even if you don't regurgitate the same stuff over and over again while there's no new useful information to disseminate (or not had the time to properly write it up). At my job, I wrote a small slack bot that'll ping the person who is dealing with the status page/customer impact, it'll ask "I'm posting an automated update in 15 minutes, would you like that or do you have a full size one?" That way, in the heat of a fire, nobody forgets about the client's status page updates. It works rather well.
jressey 8y ago
If you're on a flight, do you not appreciate when the pilot updates you, even if he says "We are still investigating the maintenance issue?"
- gerardnll 8y ago
  It's funny because if you think it carefully it's like you think they look at the problem for a while and stop until you ask again for updates. haha Like you need to keep asking for them to work on it. Sounds like an oldschool boss expecting you to be on your chair even if that doesn't mean you're working.
- samirm 8y ago
  No, because there's literally nothing I could do to change the situation...
abathur 8y ago
This is probably SLA compliance stuff (stipulations for how quickly they have to provide notice of issues, how often they're obliged to provide updates, and how quickly they'll fix an issue). These stipulations, obviously, don't say much about the quality of the notice or updates. So, yes--ticking a box (but a box that will matter for securing and maintaining contracts with larger organizations).

msghacq 8y ago

I’ve been hearing from friends at GitHub and MS that things are on fire internally. The GitHub employees are in semi-revolt because they think Nat is integrating too fast with MS. Many people have quit or are being pushed out. It may actually be for the best because the post-founder/pre-buyout team was pretty mediocre. Nat is also trying to crack down on some of the more overt activism in the company. They do “eq” screenings for political fit during the interview process and he’s trying to get them to stop. They also have a brand new head of product and engineering. Product was taken from Jason Warner about two months ago and placed directly under Nat. Sounds really bad and I don’t expect much turn around.

ambulancechaser 8y ago
> They do “eq” screenings for political fit during the interview process and he’s trying to get them to stop. This sounds crazy to me. My last company would freak out if the dev team got together and discussed the person's politics and if we thought they should be hired or not based on that. I hope they do stop this. The more i think about this the crazier it sounds. Imagine a Houston oil consultancy firm discussing a candidate who applied and asking "when we talked about lowering taxes, did the inflection in his voice sound real or feigned?". This is what an "old boys club" looks like.
- debt 8y ago
  "the person's politics" I interpreted it as a how a person deals with internal politics not what their political views are. the latter would most definitely be illegal.
  - ambulancechaser 8y ago
    from the OP > It stands for “emotional quotient” and they ask you a bunch of questions about diversity, feminism... They watch your responses for tone and vocabulary and can veto your hire if they deem you insufficiently enthusiastic. It sounds very much like your opinions on current political topics.
  - 8y ago

elesbao 8y ago

that's a shitty incident report and a sad thread if there's this kind of activism internally while shit is on fire.