9 ms·
GitHub incident: 2022/03/24
- raybb 5y agoHow does GitLab compare in terms of it's CI/CD reliability?
- colinmhayes 5y agoMy company recently switched to github actions from Gitlab CI because gitlab had so much downtime.
- pzduniak 5y agoWere you using self-hosted runners or the SaaS per-minute ones? My experience with Gitlab.com is probably irrelevant at this point, but years ago I didn't have any issues whatsoever with the combination of a Gitlab.com org with self-hosted runners... on AWS, I think?
- globular-toast 5y agoOur on-prem instance hasn't had any downtime I'm aware of in the past year at least.
- chockchocschoir 5y agoSame uptime as my development environment of GitHub then! (And similarly, my statement is as relevant as yours when it comes to the services CI uptime)
- phphphphp 5y agoI’m a big fan of GitHub Actions so take this with that in mind. GitLab (self-hosted) is very reliable however GitLab CI/CD is very different in design to GitHub Actions: it’s much more a bunch of shell scripts stuck together with YAML. Personally, I’d never go back (unless they redesign it).
- danogentili 5y agoPersonally, I hate Github Actions from the bottom of my heart. Ever since Travis CI sold its soul and I've been forced to use Github Actions, and I've come to hate every single moment I have to work with some broken or buggy action, with the inability to restart only failed jobs (they added this vital feature only a few weeks ago, 2+ years after launching actions), with slow job spin-up times, which are only slightly reduced when using the self-hosted, RAM-hogging action runners which are only capable of running one job per instance, forcing me to waste RAM setting up multiple instances of the runners, each bundling its own copy of node and the .NET runtime. I've been using gitlab CI @ work, and I absolutely love its reliability and ease of use and configuration.
- deathanatos 5y ago… yeah, we just moved some stuff to Actions, and it's been rather disappointing as onboarding goes. While the YAML represents jobs as a DAG, it seems like Actions can't actually evaluate them as such, and it inserts dependencies where none exist, i.e., it inserts stalls in the pipeline. Action builds aren't cached, so they're slow. The default token can't be extended to allow cross-repository access, forcing you to write your own app that can. The jobs, not the workflow, is the CI status/result, so you can't branch protect on a workflow. (You have to just have a dummy job … but you can't have an empty job, no, it has to run `true`…)
- rezonant 5y agoI've been very happy with CircleCI
- stnmtn 5y agoCircleCI has had multiple outages for us this past week, causing a lot of frustration.
- rezonant 5y agoInteresting. None of our builds have failed due to CircleCI during that time period, or ever, that I can remember. We do maybe 5-10 builds a day during PST working hours. Not a huge volume but still would think it would have hit our radar. CircleCI's status page, which they are usually pretty good at updating, indicates two outages over the last two months, though one of them was a long one (9 hours?) but apparently only affected remote docker and machine runners. Neither of those were in the last week though.
- fernandotakai 5y agoi honestly prefer gitlab to github wrt CI/CD -- but i'm biased, i have ~4y of gitlab ci/cd usage, while i've only been using github for 6mo. gitlab feels easier to use, easier to write and i prefer the interface.
- jasonm89 5y agoGitLab seems to have been having their own set of issues this month. https://status.gitlab.com/pages/history/5b36dc6502d06804c08349f7 https://status.gitlab.com/pages/history/5b36dc6502d06804c083...
- Beltiras 5y agoI was a heavy GitLab CI/CD user till a couple of weeks ago when I switched jobs. Loved it for the 3 1/2 years I was a user. Sometimes a bit tricky to set up and understand (especially after we switched the executors to docker-in-docker, was a pain to figure out), but once it's running it goes off without a hitch and runs very reliably. My new position entails Github, I can give a comparison in a couple of months when I'm acclimated. EDIT: I see there are people down-thread that are discussing the SAAS version, ours was a on-site deployment so we could pretty much just spin up more iron if the executors were straining.
- skwirl 5y agoGitLab has its share of downtime. I don't use GitHub regularly so I can't really compare the two, but you will occasionally have a few hours where GitLab is down. Maybe 2 or 3 times per year?
- djschnei 5y agoAnother day, another Github incident...
- Cthulhu_ 5y agoName one service with as many users, traffic, and features that doesn't have any issues.
- HL33tibCe7 5y agoConversely, can you name a service that has this many users that has this many issues?
- floodle 5y agoGitHub before Microsoft?
- meibo 5y agoGitHub before Microsoft notoriously shied away from innovating/iterating on their core business for years. We had half as many features before then. IMO feature wise, GitHub really has improved tenfold. I remember having to fight third party CIs and project trackers, now we're running them all on GitHub and it's a well-integrated pleasure. If it does work, which it usually does and hopefully goes back to doing. I also really enjoy their live vscode IDE, which lets you e.g. make small changes to PRs in a web vscode by pressing . on the PR page. This is different from their hosted VM offering and free, confusingly.
- deleted 5y ago[deleted]
- grumple 5y agoMany of us run applications with more users, traffic, data, and/or features without having such a high frequency of outages.
- deleted 5y ago
- shrikant 5y agoI came up with what I thought was an appropriate riff on XKCD 303, but I fear we might be running out of excuses now :( https://twitter.com/shr1k/status/1504464470462017542 https://twitter.com/shr1k/status/1504464470462017542
- rvz 5y agoNot one, but three outages in a row [0], if you went 'all in' on GitHub (Including GitHub Actions). Even under 'scheduled maintenance' something else unexpectedly goes down. From [0]: >> Until the next time GitHub goes down again (hopefully that won't be in another month's time). >> That says it all really. Lets reset the counter and try this again. Going to reset the counter once again. Hopefully GitHub won't have another outage in a week's (or even a months) time. It turns out that my whole comment chain in [0] was right in the 'long term' of not 'centralizing everything' [1] on GitHub since 2020. [0] https://news.ycombinator.com/item?id=30784663 https://news.ycombinator.com/item?id=30784663 [1] https://news.ycombinator.com/item?id=22867803 https://news.ycombinator.com/item?id=22867803
- JCMais 5y agohttps://github.blog/2022-03-23-an-update-on-recent-service-disruptions/ https://github.blog/2022-03-23-an-update-on-recent-service-d...
- darkwater 5y agoThanks for the link, they probably need to add another entry to the list. #hugops
- iBotPeaches 5y agoWell this time looks different than the last. * We get a non-machine message that migrations to GitHub are temporarily disabled. * We get an exact message that the delivery of webhooks and action kickoff is delayed. So while there are no 500s - I just have to wait a few more minutes for builds to kick off. So at least I can still review code in the meantime.
- nimbius 5y agohttps://www.githubstatus.com/ https://www.githubstatus.com/ at the time of posting, webhooks and actions are still struggling :( there have now been 23 incident outages in 90 days. if anyone from Microsoft is reading this, you need to start considering this degradation as the companies highest priority. At this point, Gitea, Gitlab and others (self hosted or otherwise) are beginning to gain traction as a more reliable and performant alternative. Youre facing the very real possibility of spending nearly eight billion dollars to capture the developer mindshare of the internet, only to lose it in just four years.
- awill 5y agoIsn't that basically what happened to Skype? Microsoft paid $8.5B for Skype when it was on top of the world, and now I don't know a single person who uses it.
- iratewizard 5y agoThey got patent trolled and flinched. They removed the things that made Skype best in class and turned it into just another low quality corporate messenger.
- pzduniak 5y agoWas it really due to that lawsuit? IIRC the main issue with P2P was that you could discover anyone's IP address by knowing their Skype username. I recall that people used this to DoS people in multiplayer video games - they looked up accounts with roughly the same name as the in-game one and proceeded to drop their connection. It was especially common in League of Legends.
- throwaway89645 5y agoYep - It was bad enough that it was common practice for people to proxy their skypes through a third party. (It was pretty nice that Skype let you do that really easily though)
- 5y ago
- _fat_santa 5y agoAfter this latest incident I brought it up to my company that we should start considering alternatives if this situation doesn't improve. From an outsiders perspective, it looks like all the new features they recently introduced seem to be crippling their databases. Microsoft needs to get a grip on this situation as the parent company. Their golden goose acquisition is about to become persona non grata if these outages continue.
- anaccountexists 5y agoMy company uses the same DB sharding tech. It took us about a year of daily / weekly outages until we finally were able to fix our performance issues. 256 database splits, lots of cross-shard queries removed, etc before we finally reached a happy-ish state for a year. Now it’s scheduled to fall over in about 6 months and everyone is freaking out again. It’s not new features that are hurting us, it’s the existing core product line. All the new stuff is built on horizontally sharded DBs from the get-go.
- reasonabl_human 5y agoWow. What kind of tech debt are you reckoning with / what better design decisions would you implement if redesigning today?
- gtirloni 5y agoHow are you doing cost analysis between moving off of GitHub and working around the issues?
- nijave 5y agoUsed to work at BigBank$ where they had an internal 8 node Bit bucket cluster. Down multiple times a week for 6+ months before they could finally get it under control (Jira + CI was apparently putting a lot of load on and apparently BitBucket doesn't scale past 8 nodes) Needless to say, self hosting can be more reliable but you'll probably end up dedicating a lot of resources to building that out and supporting it
- 5y ago
- bob1029 5y agoAfter the incident yesterday I sent an email to GH sales to talk about moving to on-prem enterprise so we don't have to go down with the rest of the boat. Still waiting on that callback/reply. Starting to wonder if Microsoft even wants our money anymore.
- rpadovani 5y agoLast year, we were investigating if we wanted to go with GitHub or GitLab on-prem. EU company, they don't sell directly and forwarded us to some EU vendor (strange, but whatever). We were never able to get a proper quote. While I, personally, love GitLab, it wasn't hard to convince other people in the team: we installed the free version, played around, bought a license. I still have no idea how much the final quote for GitHub could have been.
- gtirloni 5y agoNot even a full day since your email and you're making wild assumptions?
- deleted 5y ago[deleted]
- bob1029 5y agoHow long should I expect to wait for someone else to take more of my money, or at least answer 3 simple pre-sales questions? If I dont hear back from GitHub enterprise sales before the end of the week, I am going to take this as a strong hint that we are too small for microsoft to care about, suggesting that we would be safer & happier on a different vendor's product.
- gtirloni 5y agoA few days to a week is normal for enterprise sales.
- WinterMount223 5y ago
- deleted 5y ago[deleted]
- tastywheat 5y ago
- semiquaver 5y agoGitHub didn’t impose any requirement to use ‘main’ as the default branch. We merely changed the default for new repos unless the user or org has configured otherwise. Nothing about that (optional) change would have required any company to change any of what you listed.
- tastywheat 5y ago
- Cthulhu_ 5y agoGit (hub) doesn't care what you name your main branch, that was all you / your org; see https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/managing-branches-in-your-repository/changing-the-default-branch https://docs.github.com/en/repositories/configuring-branches... on how to change the default branch in your Github-hosted repository. The new default for new repos is 'main' these days but it's arbitrary (also consider how many repos have 'development' as their default branch)
- tuwtuwtuwtuw 5y ago
- tastywheat 5y ago
- HL33tibCe7 5y agoGitHub SREs are going to end up with thousand-yard stares by the time this is all over…
- birracerveza 5y agoThis is starting to get ridiculous.
- deleted 5y ago[deleted]
- anoplus 5y agoSeems it happens at the same time of the day in each case.
- AndrewOMartin 5y agoWhat we need is some kind of distributed version control system, then this would never happen. Ideally it'd be a free and open source distributed version control system designed to handle everything from small to very large projects with speed and efficiency, but I think that's just unrealistic.
- danogentili 5y agoMy irony meter is currently down for maintenance, but that's precisely what git is, a free and open source distributed version control system designed to handle everything from small to very large projects with speed and efficiency :)
- floodle 5y agoI think that was the point OP was making, through a layer of thick sarcasm.
- hwbehrens 5y agoMy meter was working perfectly until I read the parent comment, at which point it measured off the scale and promptly broke.
- groby_b 5y agoCan't be git they're talking about, they mentioned speed and efficiency for very large projects.
- dharmab 5y agoIronically, Microsoft has been a major contributor to improvements in git for handling large repos after Windows was migrated to git. https://github.com/microsoft/git https://github.com/microsoft/git
- deleted 5y ago[deleted]
- peeters 5y ago
- candiddevmike 5y agoAnyone know of an open source/self hosted alternative to GitHub's new project boards? Gitea has the old ones (Kanban style) but I really, really like the table layout.
- yazboo 5y agoFrom their earlier posts it sounds like they're encountering some kind of MySQL performance issue, which in my (horrible) experience can be extremely difficult for your jack of all trades software engineer or SRE to troubleshoot. I would hope a company Github's size would have MySQL expertise on staff, but if not I will say a prayer for the poor souls who are feverishly reading the Percona blog and trying to decide whether to tune the doublewrite buffer or redo log, or both, or neither.
- llamataboot 5y agoI agree that getting deep into the weeds on some of that stuff can be taxing on a smaller development team with a few senior generalists (of which I tend to be one) but I'm quite sure that companies at github scale have deep levels of performance expertise - still not always easy of course, because lots of these types of things only come up at some certain scale
- outworlder 5y agoDo not immediately assume that they do. Only those who have management that recognizes the need to have those experts even have a shot at getting them. If they do have the required staff it still might not be readily available due to org chart boundaries. Unclear if GitHub has the staff or if they are able to draw from the larger Microsoft pool. If they keep having issues I expect Microsoft to push them to move everything to MSSQL
- akshayshah 5y agoGitHub probably _had_ deep levels of performance expertise, but getting acquired by a megacorporation comes with a big shift in culture. I’d bet that many tenured GitHubbers left, and that there are relatively few people remaining who understand the core systems deeply. Medium-term, the more closely aligned with the rest of Microsoft’s technology they can become, the better - not many MSFT folks understand the ins and outs of sharded Percona, but many of them do understand SQL Server and .NET.
- 5y ago
- timcavel 5y ago
- tastywheat 5y ago
- niffydroid 5y agoI moved away from bitbucket to github to get away from these sort of issues.
- ram_rar 5y agoI wonder, how much of this could be attributed to attrition and gap in knowledge base created when key engineers of the related team have left.
- spencera 5y agoEveryone do your part! I took the rest of the week off to give GitHub a chance to recover. Flatten the peaks, let GitHub migrate
- deleted 5y ago[deleted]
- mh- 5y ago2 days to flatten the curve