12 ms·
GitHub incident 2022-03-23
- toastal 5y agoAnd to think Git can easily be decentralized. I wonder if the community could fork GitHub to fix it. Oh, it's not open source. Devs must be too busy working on more 'social' features like "For You (Beta)" to milk the attention economy.
- jonnybarnes 5y ago2nd day in a row isn’t it?
- momothereal 5y agoYes: https://news.ycombinator.com/item?id=30767635 https://news.ycombinator.com/item?id=30767635
- stepri 5y agoAnd 6 days ago: https://news.ycombinator.com/item?id=30711269 https://news.ycombinator.com/item?id=30711269
- fishnchips 5y agoYesterday they had two.
- rvz 5y agoIt is. 24 hours later [0] and I only expected it to happen once every month. Looks like it is getting worse. Oh dear. Not a good idea to go 'all in' on GitHub. [0] https://news.ycombinator.com/item?id=30767821 https://news.ycombinator.com/item?id=30767821
- etimberg 5y agoThe quality of GH seems to be slipping
- Trasmatta 5y agoI've actually been pretty impressed with the quality of the product and new features over the past couple of years, but it seems to be having a lot of stability issues recently.
- etimberg 5y agoI've liked the new features too, especially after so many years of not many features. Maybe they've moved too fast now
- xtracto 5y agoFunny that it happened since they were acquired by Microsoft... reminds me of Hotmail, Skype, LinkedIn, Rare, among several others.
- deleted 5y ago[deleted]
- deleted 5y ago[deleted]
- amelius 5y agoI hope it doesn't affect security ...
- paskozdilar 5y ago
- deleted 5y ago[deleted]
- iBotPeaches 5y agoIt seems like we haven't had a non-robot status update on the status page in days since this what seems like daily occurrence. I figure at this point we'd get something of why this is happening. I also don't appreciate our builds freezing, unable to be cancelled and then eating up hundreds of minutes.
- lucasyvas 5y agoBilling should always be built on a "ping" IMO and not start/stop hooks. The latter is shockingly bad for customers during times of unreliability. The former sounds stupid and requires more infrastructure from the one offering the service, but I think it's more fair. I haven't used GA in a way where it actually costed me anything, but having minutes just tick away while you can't do anything is really stupid if that's the case. Edit: Another sane solution would probably be to record outage periods and have Billing automatically reconcile for every customer when invoicing. This would require them to admit the outage durations however, so it may be flawed from a human perspective.
- drusepth 5y agoThe "ping" solution is an interesting one that I haven't seen proposed before. At what rate would you do these pings? I don't know how upgrading/downgrading works at GitHub but if they do any sort of refund/credit when you downgrade, it seems like there's some interesting implications for abusing the system (e.g. upgrading/downgrading between pings for "free" service if the time between them is too long) versus performance (e.g. how do you update all users per ping in a timely manner if the time between them is too short?). Would love to read up more on this approach; seems interesting!
- easton 5y agoDo they give you the minutes back if there's an incident during the period where a job is running?
- no_wizard 5y agoYou will have to contact them for them to credit you, that's what we did
- cube2222 5y agoLooks like they really want to get a PR deployed, but there's still not enough duct tape on it.
- Xarodon 5y agoThis has been a pretty rough week for GitHub
- stuff4ben 5y agoGithub Enterprise hasn't been faring too well at my work either this week. When you work on both open and closed source products and GH and GHE are both down, it leads to a very unproductive week.
- jrowley 5y agoDoes GitHub enterprise result in dedicated instance or any better availability?
- jon-wood 5y agoIt Depends. GitHub Enterprise is confusingly both a "call us for pricing" tier of GitHub the website, and also an on-premise version of GitHub that you can run as an appliance in your own data centre. The first of those is ultimately just GitHub and so has the same outages, the second is running on your own hardware so (shouldn't be) tied to the website's availability.
- bewuethr 5y agoThere are multiple products: self-hosted (Enterprise Server) and hosted by GitHub (Enterprise Cloud). I don't know about uptime guarantees, but you can buy Premium or Premium Plus support with 30-minute SLA or a dedicated account manager.
- nimbius 5y agohttps://www.githubstatus.com/history https://www.githubstatus.com/history 21 incident outages in just 3 months. At this rate the benefits of running your own gitea or gitlab are starting to become competitive.
- edgyquant 5y agoI’m not sure at what organization that is true. My company lives out of GitHub and Jira and I’ve hardly noticed the three month surge. GitHub would have to do a lot worse to get many companies to want to host their own services. This is the argument people have said about the cloud from day one. People want to know it isn’t their problem, that makes cloud computing (and things like GitHub) worth their weight in gold. I have real problems to solve I don’t want to deal with a git repo manager on top of that.
- jeltz 5y agoMaybe you are in a different time zone because our organization certainly noticed and was disrupted by this.
- gjulianm 5y agoAt my organization it's always been true. Setting up GitLab is fairly easy, in my company we do it and it's cheap (on-prem hosting is basically zero, and we had the IPs/domains already) and it hasn't given us too many headaches. I think last time I had to do something was maybe a few months ago when I restarted it so that it picked up the updated SSL certificate.
- rvz 5y agoAgain? Last time that happened was 24 hours ago? [0] It is really getting unreliably bad. Like I said before, having a self-hosted backup seems to make more sense. [0] https://news.ycombinator.com/item?id=30767821 https://news.ycombinator.com/item?id=30767821
- blueplanet200 5y agoI hope they figure out what’s going on every morning. Heard from inside they don’t know why the db dies everyday but restarting it fixes it.
- cube00 5y agoBreak out the early morning restart cron job.
- Kostic 5y agoEarly morning in which timezone?
- afterburner 5y agoGaryOldman.gif
- glenneroo 5y agoWhen the least amount of users are online?
- gaoshan 5y agoHere you go, Github: 0 4 * * * /etc/init.d/postgresql restart I'll take an architect position as compensation, but only if there is equity.
- exikyut 5y agoWhat's "the db"? It sounds like something of small to medium scale if you can just restart it like that. In any case, why not just relocate some vendor engineers on site for a bit? Or, better, why does the vendor not have a small presence in the corner? Sounds like whatever "the db" is it's probably some (objectively) small but very scary thing that's currently on fire and people are trying to figure out how to put it out without crashing the plane and also making too many waves internally, which is probably even harder. So asking about making vendor noises is (as useful as it may be) probably going down the wrong path - in much the same way this is probably not related to the outages (it may well be, but from the outside it's all coincidence anyway).
- einpoklum 5y agoThe page at the link is not much more informative than the link itself :-(
- okareaman 5y agoWhat's the difference between GitHub and GrubHub? GrubHub delivers
- intunderflow 5y agoWith how often these happen we might as well sticky this thread for the next one
- max23_ 5y agoLooks like the same services that were affected in yesterday incident.
- koolba 5y agoI really wish they would add the word “outage” to these titles. “Incident” alone makes me think something got hacked or leaked.
- arez 5y agoThat's SRE lingo --> https://sre.google/sre-book/managing-incidents/ https://sre.google/sre-book/managing-incidents/
- zufallsheld 5y agoIt's also itil lingo, which predates sre.
- mtnops 5y agoIt's NIMS - FEMA lingo, which predates ITIL. Which was developed in USFS wildland firefighting, which predates FEMA. It's incident management all the way down.
- zacharynewton 5y ago"The Simpsons already did it"
- higeorge13 5y agoThe usual services (actions) again down around the same time. This is embarrassing.
- grumple 5y agoAgain?! Jeez. I wish I had customers this tolerant.
- frjalex 5y agoLooking at the "GitHub" prefix in the title, I was half-expecting this to point to a report explaining the outage a week ago... But rest assured, it is a new outage!
- teekert 5y agoOh I thought it was about the one from yesterday :)
- aaaaaaaaata 5y agoAre their CI/CD toys that shiny that people still willingly choose them even with all the issues? I find myself regularly asking this — about every major SaaS used for critical ops stuff like this.
- teekert 5y agoWork choose GitHub (we are a MicroSoft shop), I have to say, I like GitHub a lot. The disruptions have been annoying sometimes, that's true. But due to the nature of Git I could always just keep working.
- deleted 5y ago[deleted]
- annexrichmond 5y agoI thought it was going to be a Postmortem. I couldn't have been more wrong!
- mirekrusin 5y agoWhat's the best crowdsourced status monitor?
- eckza 5y agohttps://outage.bingo/ https://outage.bingo/
- mirekrusin 5y ago+1 :)
- mirekrusin 5y agoStatus page says only degraded performance. It's a nice way of putting it. I'm trying to run github action for couple of hours now. They don't work at all. But apparently this means they run, but in infinite time, hence == degraded performance, nice.
- raffraffraff 5y agoIt's just a way to avoid SLA breaches. "Of course it wasn't down! It was just infinitely slow!"
- bob1029 5y agoWe are scheduling a call with an enterprise sales person next week. If I can get all the Github features I had as of ~2020, but on an instance that wont get hit by the public cloud/update bus, I would be exceptionally happy. The only complaints we have are regarding availability. If we can fix that one problem, this is a perfect product in our view.
- andruby 5y agoHow do you evalute running your own gitlab instance?
- eatonphil 5y agoGithub Actions are back for me now.
- mfashby 5y agoI'm inclined to look at tools like fossil again, for it's distributed issue tracking and wiki capability https://fossil-scm.org/home/doc/trunk/www/index.wiki https://fossil-scm.org/home/doc/trunk/www/index.wiki
- edgyquant 5y agoI had forgotten about that, thanks!
- JonChesterfield 5y agoFossil is faultless for a team size of one. I've been using it for nearly a decade, doing totally non-optimal things like using versions released years apart on different OSs with the same database. I also ctrl-c it when I spot a typo in a commit message and check in binaries. Never missed a beat. As headcount goes up I think the inability to locally rewrite history into easily reviewable patches would be sorely missed. So it's git for team stuff and fossil for my own.