6 ms·
GitHub was down
- FrostKiwi 3y agoGlad to see the status page being honest and reflective of reality for a change.
- sethammons 3y agoIt was maybe 5 minutes late, but that is pretty decent seeing as most status pages are never updated ... ever.
- peanut-walrus 3y agoThey've had a lot of practice
- TeMPOraL 3y agoMost status pages seem to be hosted on the same infra they're monitoring, so they tend to go down with the ship, while visitors are greeted by a conveniently cached "all green" version of the page. Not surprising when one realizes that status pages, by necessity, become marketing tools.
- atonse 3y agoI had the same feeling. When my PRs started failing, a few mins later I went to the status page and got immediate feedback that they were aware of it. That was useful.
- acedTrex 3y agoA historic event, something not seen since last week!
- dylan604 3y agoearlier this year, a team i'm no longer a part of switched from using AWS CodeCommit to using GitHub. at the time, i questioned why switch to something that goes down a lot. the response was it never goes down. the next week, outage. there was no valid reason on why switching to github other than "some potential VC questionnaire asked if the company was using github". i'm much happier no longer being there when these kinds of red flags are replaced with klaxons
- capableweb 3y ago> the response was it never goes down. Seems those folks maybe have never used GitHub in any larger capacity. They've been having issues constantly for a long time now. September - 7 incidents August - 17 incidents July - 11 incidents June - 13 incidents And so on. Last time they had less than 10 "incidents" was December 2022.
- Nimitz14 3y agoI find github to be a much nicer experience than codecommit. Not surprised that people would want to switch.
- progval 3y ago> some potential VC questionnaire asked if the company was using github Were the repositories even public?
- dylan604 3y agono.
- deleted 3y ago[deleted]
- FrostKiwi 3y agoOhh, back up for me here in Japan. GitHub's incident team is on crack, that turn around time is quite something. Definitely some impressive stuff.
- sfritz 3y agoI haven't seen "on crack" used in a positive light, but apparently it is sometimes? Usually I'd say "they must have a crack team on incident response", meaning a highly functional, smooth-running team.
- burnished 3y agoIts pretty unusual, I think it has to do with their local time coupled with a presumed enthusiastic and swift response on the part of github. I'm more used to 'on crack' meaning like 'delirious, over confident'.
- nacs 3y agoYeah "on crack" usually means they're not right in the head. In this context though, OP is referring to the drug where the Github engineers are being quick/energetic about fixing the issue because they're "boosted" by the drug's effects.
- wintogreen74 3y agoYes, they obviously meant the more common, positive associations related to methamphetamines like, "great job on that quick PR, Kim. You must be on meth!" :)
- progrus 3y agoOr “gee, Hunter Biden is a swell son, he sure got to Vegas really fast! He must be on crack!”
- FrostKiwi 3y agoHuh, I swear I heard in a positive context before. But English in not my native language, so my feel on the nuance of this expression is probably a bit off.
- DiracSpace 3y agoSo that’s why profiles aren’t loading
- username135 3y ago[flagged]
- simple10 3y agoGithub.com is back up for me in San Francisco. At least it let me login and create a new repo.
- horsemessiah 3y agoWtf is going on at Github? They're having outages almost every other day. August had 11 incidents it seems. September already has 3. https://www.githubstatus.com/history https://www.githubstatus.com/history
- TylerE 3y agoThey run on Azure, don’t they?
- rickette 3y agoOnly partially I believe, there quite famous for running their own hardware (metal cloud). I think some new features like Actions, Codespaces and of course Copilot run on Azure.
- sgt 3y agoThey are doing rapid deployment.
- nine_zeros 3y agoLots of attrition of good engineers. Management is not held accountable for failures, only engineers are scapegoated. As a result, remaining engineers are disgruntled and don't care. Source: https://www.teamblind.com/company/GitHub/reviews https://www.teamblind.com/company/GitHub/reviews
- no_wizard 3y agoBlind is an echo chamber, and attracts a certain type of person, for sure. Vast majority of engineers I know aren't even on it and often haven't heard of it. I think Blind is a data point, but its one of many, and certainly not definitive in my opinion
- jonny_eh 3y agoIt's like a bitter GlassDoor.
- whalesalad 3y agoFunny to finally see this. I have been having actions issues/failures since 3am EST.
- OnlyMortal 3y agoMust be that push I made!
- revskill 3y ago[flagged]
- rvz 3y agoPerhaps you should actually consider self-hosting? [0] As I said and predicted years ago? [1] But as expected [2] every month, there is an incident and something at GitHub breaks and goes down. Maybe GitHub is gradually falling apart. It is that unreliable. [0] https://news.ycombinator.com/item?id=36526334 https://news.ycombinator.com/item?id=36526334 [1] https://news.ycombinator.com/item?id=22867803 https://news.ycombinator.com/item?id=22867803 [2] https://news.ycombinator.com/item?id=37320925 https://news.ycombinator.com/item?id=37320925
- tredre3 3y agoGithub is down a few hours per month but it has never lost data. Work doesn't grind to a halt because your git host is down. So really, Github still provides a lot more value than self hosting. But sure, keep evangelizing and talking down to people who prefer it to self-hosting, I'm sure it will accomplish your goal.
- rvz 3y ago> Work doesn't grind to a halt because your git host is down. When you cannot push your critical change to master on GitHub or run your GitHub Action, then it does grind work to a halt, as it has done with the many commenters here. > So really, Github still provides a lot more value than self hosting. Centralizing everything to GitHub is much more riskier than self-hosting given the number of times GitHub has fallen over or had an 'incident'. > But sure, keep evangelizing and talking down to people who prefer it to self-hosting, I'm sure it will accomplish your goal. My point is self-proving every month and week that something in GitHub goes down or has an incident. I'll be expecting you to complain again once it falls over in a month's time at the latest.
- maxbond 3y agoI've been an incident responder where the incident was, "We promised a customer that changes would go out today, but GitHub is down, and thus our CI/CD is also down." It definitely does matter when GitHub goes down.
- sameg14 3y agoSeems like it's been a common occurrence ever since Microsoft took it over
- naikrovek 3y agogot any data on that, or are you just using this as an opportunity to take a stab at a company you don't like? either is fine, just be open about it.
- BearhatBeer 3y agoNothing wrong with hating Microsoft, it's a multiple felon.
- tempodox 3y agoThis incident has been resolved. Posted 1 hour ago. Sep 05, 2023 - 17:01 UTC
- vkoskiv 3y agoDidn't notice it this time! At work we use bitbucket, and for the last few weeks I've been hacking on a hobby project on a 37 year old Mac. It can't run Git, but it does have Amend[1]! [1] https://amendhub.com/ https://amendhub.com/
- Arbortheus 3y agoThere was an outage yesterday too when the GitHub action “checkout@v3” broke when they released “checkout@v4” Yes, they broke the ability for GitHub CI to checkout repos… https://github.com/actions/checkout/issues/1448 https://github.com/actions/checkout/issues/1448
- zen928 3y agothis is offset by taking your job slightly seriously when implementing gh actions for your org by understanding how they're resolved and pinning them to commit SHAs, like how github themselves recommend under their basic security guide https://docs.github.com/en/actions/security-guides/security-hardening-for-github-actions#using-third-party-actions https://docs.github.com/en/actions/security-guides/security-... after hearing similar stories from frontend developers who haven't even heard of "npm ci" or use range pinned dependencies on production systems, im almost never going to be sympathetic on blaming the package maintainers over developer ignorance following basic security guidelines
- Arbortheus 3y agoSure, but then again the link you sent says: `Pin actions to a tag only if you trust the creator` If I don't trust GitHub, why am I storing my source code on it and running CI via GitHub actions? There are also some security gains from ranged pinning. Suppose I pin my library to `1.1.x` instead of `1.1.0`, when a security patch comes along for some CVE, my service will automatically run the `1.1.1` patch release the next time we deploy it. I would say the likelihood of a developer getting lazy and not bumping a dependency causing a security incident is higher than a supply chain attack if you're already using reputable libraries. Let's be frank, how many developers care enough to go into the 10 repos they've ended up owning to go and change their `actions/checkout@v3.1.6` to `actions/checkout@v3.1.7`.