8 ms·
GitHub actions is down
- dtech 4y agoHave had this for a few hours. Pretty annoying that the status page shows all green.
- zeepzeep 4y agoThe Action to change the status page doesn't work, what do you expect?
- rozenmd 4y agoThey're using a 3rd party status page, it's a fancy CMS at best.
- throwaway892238 4y agoYou mean the liability announcement page.
- kylecordes 4y agoWhen the movement toward status pages started, it was great. Early adopters were eager to communicate their status, to the extent they technically could. Nowadays it feels like most status pages show green even when things are decidedly not green. Conspiracy theorists abound, but in most cases it's probably just ordinary human tendency to not broadcast one's foul-ups eagerly and broadly at the first opportunity.
- jopsen 4y agoOr just as likely: that only some customers are affected. So it's not a full outage.
- FireBeyond 4y agoSo set it Yellow. "Some customers are/may be affected." Customers who are affected? Appreciate the acknowledgment. Customers who aren't? Appreciate that if they are next time, even if it's only a small subset, they'll be made aware.
- dandellion 4y agoIMHO GitHub should be spending less resources on adding achievement badges and more resources on preventing frequent service disruptions...
- o_m 4y agoIn a way it is good that they are focusing on the wrong things. It makes it easier for competitors to catch up and compete.
- Raed667 4y agoMore likely the team that worked on the badges (UI and data engineers) have no impact or ownership over disruptions (SREs and infra)
- dandellion 4y agoExactly, they could be spending more resources on hiring more SREs and infra and less UI and data engineers.
- matsemann 4y agoStuffing twice the amount of engineers into a team is not exactly a good way to reduce complexity and reduce downtime, though.
- hksoftware 4y ago
- lutrinus 4y agoThe title seems to be a bit misleading as this seems to be - at least right now - not a widespread issue. We currently don't have any issues and the status page also doesn't list anything.
- dncornholio 4y agoStatus page updated not long after this post.. makes you wonder who is misleading who..
- shamiln 4y agoIt looks like it's an artefact cache issue rather than a more widespread issue?
- Turbots 4y ago
- shard972 4y ago
- rvz 4y agoAgain? Last time that happened was 12 days ago. [0] Worse part is, it hasn't reflected on the status page. Typical of status pages. Looks like relying on and going all in on Github and GitHub Actions really doesn't make any sense and I now expect this to go down once or even twice a month. Oh dear. Perhaps self-hosting like how ReactOS, ARM, RedoxOS, GNOME, etc have done seems like a good idea. [0] https://news.ycombinator.com/item?id=31579932 https://news.ycombinator.com/item?id=31579932
- dicknuckle 4y agoFor now you can just host GitHub Actions locally. https://github.com/nektos/act https://github.com/nektos/act
- reitanuki 4y agoUnfortunately it's not fully-fledged enough to work properly with all CI configs. I had trouble with some conditions but also I note that it doesn't seem to implement reusable workflows yet. It's also a bit fiddly to set up so you don't run into API rate limits. I think that people who are interested in having resilience should write their GHA CI config for act and then be happily surprised when it works on GitHub too. Or write them as something more portable (Earthfiles are interesting, but a bit clunky when I tried) that can also be executed locally — having the GitHub-specific stuff just be a very light wrapper to 'call into' the portable scripts.
- mdaniel 4y agoI started to bring awareness to the Earthfiles goofy license, but it seems they've switched to MPL! https://github.com/earthly/earthly/releases/tag/v0.6.15 https://github.com/earthly/earthly/releases/tag/v0.6.15 The (unfortunately named) Dagger is also an entry into that space: https://github.com/dagger/dagger#readme https://github.com/dagger/dagger#readme (Apache 2)
- Dracophoenix 4y agoNow that everyone and their dog is moving to "serverless" hosting, how does one institute redundancies in case something like this happens again? It would suck if half the Internet and its open source projects go down because of a bad server configuration. Does anyone know of a "load balancer" for Github projects or source management in general? It looks like a solution many could benefit from.
- smeagull 4y agoAlso use gitlab?
- numbsafari 4y agoOr just some flexibility? We don’t rely on GH for our production systems. We have a clone of our Git repo hosted in-house for anything that needs a Git repo. We don’t rely on GH for our Primary CI/CD. We can make changes and push them through CI/CD without GH. We have an understanding of the business impact of GH being down. It’s an enabler that we love using, but we can operate without it for a known period of time, albeit without certain workflows and value added tools. If our tolerance for it being down we’re much greater, then it would be worth the money and overhead to host it on our own, which they support.
- Turbots 4y ago
- throwaway892238 4y agoThe simple answer is "you don't," because a couple hours downtime on your CI/CD platform shouldn't matter. You should be able to deploy by hand in an emergency. The less simple answer is "by doing Disaster Recovery planning". Go outside and throw a rock and you'll hit a company doing hosted CI/CD. Set up two of them; when one fails, use the other. GitHub could mitigate this by making its services region and AZ specific, and then only one region should go down at a time, letting you switch regions. Not sure why this hasn't happened, other than GHA is more of a "consumer product" and they may assume customers are too dumb to figure out how to switch regions. (This may sound harsh, but this is the same company that wouldn't update documentation about how to generate ssh keys to make them more secure because it might confuse their users)
- wkillerud 4y agoThis is hitting workflows that use caching [1][2]. - [1] https://github.com/actions/setup-node/issues/516 https://github.com/actions/setup-node/issues/516 - [2] https://github.com/actions/cache/issues/820 https://github.com/actions/cache/issues/820 As of now (11:28 UTC) the status page has been updated. https://www.githubstatus.com https://www.githubstatus.com
- smcleod 4y agoActions goes down so much more frequently than they disclose on their status page, every few days there's timeouts, caching problem etc...
- nine_k 4y agoLooks like an area ripe for disruption! (Only half-joking.)
- maweki 4y agoIt seems like there's enough disruption in this area already.
- glenjamin 4y agoI think a hosted mulit-tenant CI platform appears fairly simple on the surface, but there's actually a lot of complexity under the hood when your product is arbitrary code execution as-a-service - and you want to offer a smoother user experience than something as general-purpose as a container runtime. Especially when you throw in a large and generous free tier.
- nixgeek 4y agoNone of the recent examples of unbundled multi-tenant CI/CD appear to have wildly successful exit stories, or even continuing to run with great growth and profitability stories. I’m thinking of companies like Travis with that comment (which was, IMO, delightful to use for its era). It’s probably pretty hard to sell this to the average enterprise when you’re competing against product suites which do 100x more? Definitely feels like a situation where those suites are “Jack of all trades, and master of none” though, and collectively mediocre vs. individually excellent.
- plonk 4y agoI'd like GitHub to stop disrupting itself please.
- FireBeyond 4y ago
- Shadonototra 4y agothey should be rewarded with "down every weeks" achievement
- anonymoushn 4y agoIt seems like we could just sticky this post at this point.
- imdsm 4y ago```diff - GitHub Actions Is Down (github.community) + GitHub Actions Is Down Again (github.community) ```
- dustinmoris 4y agoWhy is GitHub Actions so frequently down since the rewrite to C# and moving the Azure DevOps engineers to GitHub?
- BarryMilo 4y agoI feel like you're answering your own question.
- oneepic 4y agoI'd be interested in reading a source for both of these points, if you have one. By "moving the Azure DevOps engineers to GitHub" do you mean they switched internal tooling on their own team? or that engineers from the ADO team switched orgs and now work for GitHub?
- chrisux 4y agoI can't find any info about anything in Github, much less actions specifically, being rewritten in c# either.
- corrral 4y agoBad integration work, and/or bad architecture on one or both of Github and Azure's sides.
- gxt 4y agoAt what point would these outages impacting productivity be considered hostile and anti competitive?
- dijit 4y agoWhen they impede a competitor (and only a competitor). As it stands CI/CD is a bit of a commodity, so you can go with any of the solutions and they're largely similar.
- gxt 4y agoAs an individual sure, but as an organization? No way a team of 100 can just pivot to another provider. CICD providers are not interoperable, once one is picked, it's a complete lock in. If I was evil, I would offer a cheap service to employees of any potential contributors, one MBA types don't understand, then once they've made themselves dependent on it, make it unreliable. Boom. There goes your productivity and your credibility as competition.
- donatj 4y agoGitHubs status page went from industry leading with uptime graphs and instant updates to a static “everything’s ok” page (as the whole service 500s) page overnight with the Microsoft acquisition. They are burning good will like it’s a fuel.
- cmiles74 4y agoI find this super frustrating. This weak sauce on the burrito of disappointment that is downtime adds insult to injury.
- throwaway892238 4y agoThat's what happens when you join a trillion-dollar company. Somebody lets it slip that the service might not be reliable and suddenly you lose a billion dollars on the stock market.
- leaflets2 4y agoSource? Any graphs to look at, for example. That covers years before and after the acquisition? I've been thinking that GitHub has pretty good uptime. (I wouldn't personally want to rely on a specific 3rd party like GitHub or Circle CI or anyone, for deploying my software, though. Wow, 560 people have followed the githubstatus.com link (I see in the Discourse forum) that's many people affected I think!)
- hatware 4y agoIt happens once a quarter, sometimes more!
- dilyevsky 4y agoGithub has outages nearly ever week now, sometimes multiple times a week not quarter
- cmrdporcupine 4y agoLots of complaining here. I'd be ticked if I was a paying customer. But I just started using Actions yesterday. On my free open source project. And I've been using it all morning (no problems). And the whole time I was thinking: it's bloody amazing that I'm getting this for free. (I used to work on a bit of a similar system at Google [internal remove build execution system for non-google3 projects] and ... there's a lot involved.)
- leaflets2 4y ago> And the whole time I was thinking: it's bloody amazing that I'm getting this for free. Me too
- rurban 4y agoAzure team != GitHub team