10 ms·
GitHub experiencing issues with actions, pull requests, packages
- pestkranker 6y ago~10 downtime per month. What is happening at GitHub?
- tylfin 6y agoThe last time this happened, it was after they shipped the phone app. I wonder if they have a big feature underway or are just migrating more infrastructure to Azure? EDIT: Either way, some postmortems would be appreciated before more customers have to look for a backup solution...
- capableweb 6y agoIf you're working on a serious project, hosting it mainly on GitHub via Git and don't already have a backup solution in place, I'm afraid you're late. But better late than never! Make sure you can always deploy when less reliable services are down, and GitHub has always been one of those. Git makes it incredibly easy as well, as long as you have your CI/CD externalized already.
- tylfin 6y agoYeah, this is very good advice. I think if revenue or product quality is tied to a VCS, having an active-active or active-passive setup is the way to go. Fortunately, I'm on an on-prem product so that investment hasn't seemed worth it yet. This doesn't mean we don't escrow our code, but rather than try to rebuild from source, I just take a short coffee break and wait for the impacted service to come back up :)
- dessant 6y agoI'm still awaiting the promised post-mortem [1] of a retracted blog post [2][3] which has annnounced the deprecation of the GitHub Developer Program. > There must be quite a story behind this - will you be putting up a post-mortem ? (Post mortems of business "outages" are usually more instructional) > Yes. We will. Please stay tuned. [1] https://news.ycombinator.com/item?id=21718171 https://news.ycombinator.com/item?id=21718171 [2] https://news.ycombinator.com/item?id=21718083 https://news.ycombinator.com/item?id=21718083 [3] https://web.archive.org/web/20191205225751/https://developer.github.com/changes/2019-12-05-gdp-deprecation/ https://web.archive.org/web/20191205225751/https://developer...
- TruthWillHurt 6y agoMicrosoft.
- rvz 6y agoAll I know is that it doesn't seem like a wise choice to be locked into GitHub features or even use their tools with these frequent downtime episodes. If a large open-source organisation was to rely on say, GitHub Actions for example, well you'll probably see more and more of "GitHub down" posts and they'll be unable to push that critical patch or run that cloud CI on GitHub, and some maybe considering solutions like this [0]. Every time this happens, you'll be completely locked in and ending up contacting / complaining to the CEO of GitHub for support via Twitter. No thanks and no deal I'm afraid. [0] https://news.ycombinator.com/item?id=23849565 https://news.ycombinator.com/item?id=23849565
- bob1029 6y agoGood deal for us, actually. I used to get upset about this, but the alternative really does suck more. You can either deal with the occasional non-productivity from a SAAS offering (which for GH has never lasted more than ~half a work day), or you can spin up all your own stack on-prem and generate 10 additional full-time problems in the quest to solve this one periodic issue. The trick is to never put yourself in a position where your tools absolutely must work immediately or you lose a customer. Why make a promise on delivering a piece of software until its already in hand? Also, if github goes down and I really wanted to get an issue comment in, I can just open a text editor and keep a note around in my local repo until everything is back up. I can even do some crazy things, like share my local branches with other developers over side channels until things get back to normal in the centralized system. Wasn't this kinda the entire point of talking developers into moving to the git model? Would be fun to rewind the clock and use these takes as an argument for sticking with TFS, et. al.
- yarcob 6y ago> I can even do some crazy things, like share my local branches with other developers over side channels Haha, right, totally crazy, you could use git just like it was designed to be used! Who would do that!! (Joking aside, despite Githubs frequent outages, I don't recall the git service itself ever being affected)
- 6y ago
- hulitu 6y ago> ~10 downtime per month. What is happening at GitHub? They belong to Microsoft. Reliability was never a feature of Microsoft products.
- Droobfest 6y agoMaybe they're porting over the Azure DevOps code which also has 'degraded performance' issues about every other day.
- Florin_Andrei 6y ago> What is happening at GitHub? They were acquired by Microsoft a while ago, and now the chicken are coming home to roost. It's pretty much on par with my Azure experience.
- robbyt 6y agoMe too. The only people who think Azure is viable, haven't actually used it.
- seniorThrowaway 6y agoI recently left a project using AWS for one using Azure. I thought the AWS API's were inconsistent and janky but they look great compared to Azure. Azure is also extremely slow to perform actions in my experience and the documentation is very heavily tilted towards being sales funnels. I do like the keyvault service and the idea of resource groups. The whole tenant / subscription / roles / user mess of permissions not so much, but I expected that from Microsoft.
- Florin_Andrei 6y agoThe AWS API is just fragmented. Too many teams that probably didn't communicate very closely. That being said, you can sort of follow the logic - or the multiple logics. But that's really the only negative. I'm trying to find a single term to describe all of Azure, and I'm having difficulty with it. Sophomoric? Like a place where the leaders are a bunch of B-class players, who lead all sorts of C-, D- and all the way down to Z-class characters. And hopefully you'll never need support with Azure, especially urgent support, because it's atrocious.
- hiyou102 6y agoWhat specific issues are you having? What do you mean by "B-class players, who lead all sorts of C-, D- and all the way down to Z-class characters"? I assume in this ranking you are an A+ player yourself, which is very impressive.
- bredren 6y agoThis is a big problem with GitHub Actions. It took a community drumbeat and persistence from an enterprise customer to get the status message to even show a problem last month. [1] It does suck and I do think there must be some political infighting going on that the service is having so many disruptions. There’s no excuse for something this important to not only have so much unplanned downtime, but no resources to connect with the community by offering post mortems or other reasonable interactions. That said, I’m still all in on GA. It’s amazing and the coupling with repos is great. It continues to be subtly refined. So I just hope whoever is holding this product back gets out of the way. [1] https://github.community/t/random-unknown-blob-error-when-pushing-to-docker-github-package-repository/16725/36?u=banagale https://github.community/t/random-unknown-blob-error-when-pu...
- dustinmoris 6y agoThe Azure DevOps team has taken over. Things will only go downhill from here.
- eurasiantiger 6y agoThe orcs are raiding the woods around Isengard. When Sauron promised your people stability, did you think he was going to do it for free? Welcome to AzureHub.
- jbm 6y agoYou said it. I'm consulting for a company that uses Azure DevOps and I cannot believe how much harder it is to use than Github Enterprise for getting things done. The documentation is also strictly worse, and localization is just not there at all. -edit- I assume someone who works on Azure DevOps might be looking, so a few small specific things so you don't think I'm just a hater. - It is hard to use markdown consistently, and it is particularly painful when doing any project management work on DevOps (which I assume is one of its strong points) - The lack of a Japanese menu really sucks for something that is supposedly aimed at enterprises. Having to explain both native and English language vocabulary terms is double plus ungood (again, in an enterprise product put out by a company that has usually done excellent business in Japan) - It's baffling that I can't make changes to the template when you give me a template option. I assume it is a permissions issue, but really?
- danielheath 6y agoI personally know multiple senior ex-githubbers who quit over the ICE contracts; it seems plausible to me that they’ve lost key expertise and can’t safely integrate with their older systems.
- hiyou102 6y agoIn the last availability report they mentioned MySQL issues: https://github.blog/2021-02-02-github-availability-report-january-2021/ https://github.blog/2021-02-02-github-availability-report-ja... Based off these reports most of their recent issues have been them hitting scale limits for their MySQL configurations and not having sufficient monitoring.
- GordonS 6y agoI noticed that code search seems to be broken too - hasn't been working for 1-2 hours. Search still works for issues, repos etc, but not code.
- inetknght 6y agoGithub's code search has never been useful though
- mistersys 6y agoGitHub search is not very smart, but I prefer that when searching code. When searching code, I'm usually trying to find exact tokens i.e. a variable name or an error message string vs. searching documents. I find it useful all the time.
- inetknght 6y agoI've found that `grep` with PCRE enabled is far more useful to find exact tokens like variable names or error message strings. I've never had Github's search find what I'm looking for.
- remram 6y agoThat requires you to clone. It's a minor hassle if you search over a single repo, but when searching across an organization or the whole site, using grep is not an option.
- inetknght 6y ago> That requires you to clone. It's a minor hassle if you search over a single repo, but when searching across an organization or the whole site, using grep is not an option. Are you kidding? The finer granularity that searching over an organization or whole site makes grep the far better choice especially since its output can be fed as input to more filtering steps.
- deleted 6y ago[deleted]
- brahyam 6y agoI believe this incident has been happening for more than 30 mins. I've had problems with gradle pipelines failing without reason for at least the last 6 hours
- exikyut 6y ago"The world might be ending." "Update: The world is slightly closer to ending." "Update: If more thing breaks the world will implode." "Update: This issue has been resolved." Me: [Wonders what happened] - Every outage status tracker ever
- darknavi 6y agoOne day you'll learn about the one Russian submariner who saved all of our web services from destruction.
- cyberlurker 6y agoReference: https://en.m.wikipedia.org/wiki/Stanislav_Petrov https://en.m.wikipedia.org/wiki/Stanislav_Petrov
- gknoy 6y agoI really enjoy seeing the postmortems that some teams publish after these kinds of things.
- drewcoo 6y agoBut there was customer communication. It's checked off the list, see?
- gkop 6y agoA few years ago GH actually had a fairly transparent status page that displayed error rates across components over narrow timeslices. I guess they canned it not too long after their $100M a16z raise.
- WrtCdEvrydy 6y agoBecause they are glorified marketing pages.
- itsjloh 6y agoGitHub generally do monthly uptime reports that go into more detail about any outages they experienced, see the latest one here https://github.blog/2021-02-02-github-availability-report-january-2021/ https://github.blog/2021-02-02-github-availability-report-ja...
- freakynit 6y agoIs anyone facing issue with github pages too? Updated content not getting reflected?
- forlorn 6y agoOh, weekly Github is down.
- encom 6y agoRelated: Seems like every other day "$PRODUCT is down" is #1 on HN. Why? This can't possibly satisfy anyones intellectual curiosity.
- cronix 6y agoBecause many people here actually use the affected services?
- hulitu 6y agoSo the solution is easy: don't use it anymore.
- encom 6y agoI understand that. That doesn't make it interesting.
- brutal_chaos_ 6y agoReceiving crumbs of info as to WHY a service is down is interesting to some. These are also great for tracking postmortems (unless they take awhile, then it's a separate post). Speculation is entertaining to many as well. Or perhaps this sparks an idea for someone (omg, GH is down ALL THE TIME, time to build a novel competitor!). And given the crap state status pages are in these days (stop showing green when your site is down!), these are great for knowing when a service is operating again. EDIT: words r hard
- judge2020 6y agoProbably something about centralization and relying on a company for keeping your business running. But, of course GitHub Enterprise Server deployments aren't down, so anyone that really needs uptime can pay for the privilege (or can pay for GH One which has a "30-minute SLA").
- yarcob 6y agoEven though the status page shows all green checkmarks, we are still experiencing issues. The web interface is showing stale data and we can't perform merges in git. Looks like some of their systems got out of sync and they aren't done resynchronising yet.
- yarcob 6y agoUpdate: After nothing happened, I rewrote the last commit and force pushed to trigger an update. That seemed to do the trick. I have no inside knowledge, but from the outside it looks like whatever outage they had broke the propagation of commit data from the git repo to to their MySQL database. Maybe they use webhooks for their internal systems as well? That would explain why they first saw issues with webhooks, and later issues with pull requests that might depend on them. It also looks like after they fixed the issue, they didn't replay the failed notifications. That's why I saw stale data even after they apparently fixed the issue. Then my push to the repo seemed to trigger an update, and now it's back in sync. I'm curious to hear what really happened, but I doubt this incident is important enough to anyone to warrant a detailed blog post.
- faitswulff 6y agoAfter seeing this happen over and over again, I wonder if status pages should even be run by their own parent companies.
- robbyt 6y agoSounds like Azure... Solid 500s from API, but status is green.
- eurasiantiger 6y agoI assume you’re on the lowest tiers; HTTP 500 randomly for hours seems to be the Azure way of upselling.
- viraptor 6y agoWhat's the path from random 500 for hours from a services and decision to give them more money? That seems backwards.
- FunnyLookinHat 6y agoThis same pattern happened last week - Webhooks reporting issues, then Pull Requests, etc. This one seems to be affecting the rest of their systems a bit more (or maybe they're just being a bit more detailed in exactly what is down this time). https://www.githubstatus.com/incidents/5zb8gfbl7qkt https://www.githubstatus.com/incidents/5zb8gfbl7qkt
- expliced 6y agoCommits I pushed were not showing up on my pull-request in the UI, was wondering what was going on. This explains it.
- superkuh 6y agoPlay stupid games with centralized entities, win stupid centralized prizes. I can understand using github/lab/etc if you're forced to by work in order to earn money in order to live. But willingly choosing them for personal projects is just stupid.
- fartcannon 6y agoWhat options exist for decentralized source discovery indexes/pages? Like the Pirate Bay used to be (might still be, I haven't looked in a while). Just searchable links to torrent based git repos. Googling suggests there was once (and still might be) a 'GitTorrent' which is a fantastic name for the service.
- qbasic_forever 6y agoGit was designed for decentralized use. It's not as slick and polished as github, but it can get the job done and scales to infinity (the linux kernel has thousands of contributors and runs entirely dencentralized with git). For read-only access you can host a git repo on any static file store, like S3, netlify, digital ocean, etc. Just rsync/rclone/upload a bare repo from your machine and you're done: https://git-scm.com/book/en/v2/Git-on-the-Server-Getting-Git-on-a-Server https://git-scm.com/book/en/v2/Git-on-the-Server-Getting-Git... For write access it's more difficult without running your own server (which is super easy with gitea, gogs, etc. and just a couple clicks to setup on popular hosts like digital ocean). You could take an entirely decentralized approach and run things like the linux kernel--all patches (aka pull requests) get sent to an email list where they're reviewed, discussed, and integrated by the maintainer of the read-only repo.
- fartcannon 6y agoYes, I know. I guess I skipped a chunk of my thinking. I'd like everyone to move off github, but still be searchable like github. Thanks though!
- edoceo 6y agoIn other news, folks who are self-hosting GitLab are in great shape. And once the GitHub issues are sorted we can make sure to push to those public end-points of convenience. Self hosting critical services (email, chat, git, etc) is not a terrible idea. Of course, CBA/risk factor for your team.
- alex_g 6y agoOn a related note, I have been unable to transfer a repo for two weeks now. The sender is able to initiate it but no notification ever appears on my end. If anyone knows how to get the attention of someone at GitHub please let me know. I wish GitHub had a paid support option that was reasonably priced for a single inquiry.
- deleted 6y ago[deleted]
- alfiedotwtf 6y agoCurious: are Microsoft slowly replacing the innards of GitHub with a Microsoft stack?
- qbasic_forever 6y agoThe main web backend is still ruby it seems: https://github.blog/2020-08-25-upgrading-github-to-ruby-2-7/ https://github.blog/2020-08-25-upgrading-github-to-ruby-2-7/ I can't remember where it was exactly but I remember recently stumbling on a random github-related OSS project that had chatter in the issues from githubbers talking about .NET and C#. It would not surprise me to see significant pressure from the now 2-year post-acquisition engineering org to get in line with the rest of MS's online services that are all .NET stack.
- eurasiantiger 6y agoWhy pick up a new gun you’re not used to, when you have a perfectly working footgun you know inside and out—even if it only allows you to shoot yourself in the foot?
- meddlepal 6y agoMicrosoft does not have a consistent internal stack nor is anyone "encouraged" to use .NET. I don't know where this idea that Microsoft's online services all use .NET, for example, my team which delivers a large service as part of a top-level Azure offering is mostly all Go.
- qbasic_forever 6y agoEhh, I worked there on online services before Azure even existed and it would have been career suicide and an enormous legal fight to use anything outside MS's stack--IIS, MSSQL, COM & C++, then .NET. Hotmail got away with it for some years because they were an acquisition.
- meddlepal 6y ago
- LinusS1 6y agoAzure just needed a quick reboot
- beermonster 6y agoGitHub have a historic status page [1]. Do GitLab have anything similar for comparison? [1] https://www.githubstatus.com/history https://www.githubstatus.com/history
- john_cogs 6y agoThis link will take you to GitLab's status history: https://status.gitlab.com/pages/history/5b36dc6502d06804c08349f7 https://status.gitlab.com/pages/history/5b36dc6502d06804c083... You can also find the link at the bottom of the GitLab status page: https://status.gitlab.com/ https://status.gitlab.com/