8 ms·
Incident with GitHub Actions, API Requests, Git Operations, Issues and more
- dgorges 5y agoWe are also seeing issues with Github Actions. Here we go again...
- mcjiggerlog 5y agoYup definitely more than just webhooks - I can't perform basic git operations like push.
- misnome 5y agoI really wish there was some pressure to have these status pages be somewhat "realistic" and responsive rather than hoping that temporary problems go away so they don't have SLA questions. I checked after none of my pushed commits showed in PR's, and it was "Webhooks" only. That's not useful.
- annowiki 5y agoIt seems more than just actions/webhooks. I can't push any commits at all.
- berkes 5y agoSlowly everything is turning yellow at the status page.
- etimberg 5y agoHaven't been able to clear notifications since 11:15. Looks like almost everything is down now
- blacksmith_tb 5y agoWhich seems like an understatement on their part - shouldn't everything be 'red' if all services are down? If not, what kind of outage does it take to get to red?
- stnmtn 5y agoIt seems like most of this is services going "read-only", I would imagine Red for PRs would be that you can't access or read them whatsoever This is different from commits, which probably should be Red.
- KrishnaShripad 5y agoCan't push my commits too!
- penagwin 5y agoSame here! > remote: fatal error in commit_refs
- jackson1442 5y agoSame here- I was doing a bit more complicated rebase than usual and thought it was my fault for a good few minutes there.
- NiekvdMaas 5y agoThe title of the incident has been updated to: "Incident with GitHub Actions, API Requests, Issues, GitHub Packages, GitHub Pages, Pull Requests, and Webhooks". Basically all core functionality is down at the moment.
- rvz 5y agoExactly. Basically all GitHub services are currently degraded or down today.
- howolduis 5y agoThis is not correct. I was able to open the website just now.
- williamscales 5y agoThat doesn't mean it's not degraded.
- jobstijl 5y agoThis happens quit a lot with github lately it seems.
- ehakan 5y agoGit LFS is down too, failing with lock errors
- sethd 5y agoPerhaps one could say that falls under "Git Operations", but it feels like it deserves a mention of its own. :)
- boros2me 5y agoYeah, not just Webhooks... Git pushes are failing too.
- 0des 5y agoPlease disregard the following offtopic rant; I'm not sure if I'm just getting older and jaded or if Github (and the web at large, if I might be frank) is having these issues more often as of late. Is it that what we are doing is becoming more complex, or are we, as a web, moving faster and breaking more things? I wonder sometimes if the advancement and affordability of our hardware and the systems we can run on them has decreased our reliance on optimization and correctness as a requirement of deploying on affordable hardware. Basic functions used to have to be combined with great rigor to run on the same machine for swaths of users. Now we are so far from that we have 1000 layers of abstraction, and it isn't hard to imagine how lots of tiny inefficiencies stacked on top of each other may be adding up to bigger issues. In my org we have a term for this, where you're encumbered by faults deep in the toolchain - getting 'middlewared'.
- thih9 5y agoEven more offtopic: your comment makes me think of other industries and realise how harmful the heavily processed food must be.
- grumple 5y agoWhile the web is quite complex now, I think the expectation of 100% uptime for anything is unreasonable. My local taco shop ran out of rice a few weeks ago. A few days ago they had some unspecified kitchen issue and had to close. They don't open on sundays. Occasional downtime is normal. We can try to prevent it, as devs, working with machines, but it is inevitable. Nothing continues working forever. All industries have downtime, it needs to be accepted and planned for. Even in the old days, with the simplest possible thing that needs a db connection, you could outscale that db, or bandwidth, or whatever, and crash. It was easier back then in a lot of ways - 1000 people visit your new site? Crash. Before auto scaling infra you had to either deal with it or pay for too much hardware. I'm sure there are many other examples.
- misnome 5y ago> All industries have downtime, it needs to be accepted and planned for. And those that can't - e.g. those with lives at risk over failure - have very slow, conservative and burdensome processes for changes and approval, that would never be accepted without the necessary risk.
- deleted 5y ago[deleted]
- rvz 5y agoTwo months since the last time a serious issue like this happened. [0] Not really good if you don't have a backup method to avoid this. Unlike those who went all in. [0] https://news.ycombinator.com/item?id=27366805 https://news.ycombinator.com/item?id=27366805
- grumple 5y agoI could see things like this messing up launch date/times occasionally, but what workplace falls apart if you can't push code to github for a few hours? Surely you can do other work while waiting for them to fix things.
- Operyl 5y agoOne finding itself in a particular rush to put out a fix, I suppose. I’d say “you should have back up ways to deploy” but I can certainly see why a lot of workflows get centralized on a provider like GitHub.
- yuriko 5y agoNow Git operations also down...
- Exuma 5y agoWho else googled and saw git gc/git fsck SO answer and tried it.
- dkersten 5y ago> Update - Git Operations is experiencing degraded performance. We are continuing to investigate. I hate that companies are saying degraded performance for "its not working at all" (even if only for some people). That's not degraded performance, that's broken. At least the status grid says Partial Outage, but stop with the degraded performance bullshit.
- derefr 5y agoDegraded performance is what it says. Things that took 20s are currently taking 60s. Things that took 60s are now taking 180s. Things are going slower. Maybe by a constant factor. Maybe exponentially. Where this sometimes turns into a "partial outage", is that there might be response timeouts somewhere in the backend of the stack, meaning that things that took 60s are now taking [100s and then a 504 error response]. ("Because," the backend dev thinks, "why would clients want a response after 100s? They've probably given up and gone home already at that point.") Or, equally likely, there's a response deadline somewhere in your client—where you figured "their API will never take more than 100s to respond", so even though the API is still chugging away at the 100s mark, your client cuts it off, drops the connection, and displays an error. Either way, the literal problem at hand is still, simply, "things are going slower." "Degraded performance" can help you predict whether you'll be one of the customers encountering failures, in a way that "partial outage" cannot. Were your particular querying workloads to the API fast before? You'll probably be okay. Were they slow before? Then you might experience the further slowdown as an outage. If you want to design for fault-tolerance of this case, it's important to know how APIs you talk to deal with long-running workloads. Do they have timeouts? Are they customizable? Are they different per plan level? These are things the API's documentation should communicate. Once you know these things—and you have metrics on how long requests to a given upstream API are taking to complete—you'll know exactly whether "degraded performance" will affect you or not, before actually getting customer reports.
- pera 5y agoWell I am getting 500s instantly, not after some internal timeout
- Flex247A 5y agoI was getting server 500 error while forking godotengine/godot when my internet cut off. I thought it was an issue from my side.
- aashcan 5y agomeh, here I thought my git-foo was wrong. Ran garbage clearance for nothing also. :-(
- twistedpair 5y ago`watch -n1 "git push origin"` Time for lunch. The push will get through eventually.
- paulrostorp 5y ago+1 I can't believe spamming it actually worked !
- packersville 5y ago+1 Worked for me. Beautiful
- remram 5y agoAt least make it stop once it succeeds...
- whofw 5y agoGlad it's early enough to go back to bed!
- chrishynes 5y agoSeems like GitHub is in read only mode right now -- no write actions work (creating issues, adding comments, pushing, branching, PR's)
- tryingtogetback 5y agoCurious, has anyone tried (in corp env) to promote LAN git node (on someones laptop) to the active remote, update remotes in teams' individual git configs and continue with the work as if GitHub was never even there?
- singlow 5y agoIf only we had a decentralized version control system.
- stnmtn 5y agoThere's no reason you can't host your own!
- luhn 5y agoI know you're being flippant, but if this was Subversion I couldn't commit, checkout other branches, or really do anything. With git I can do everything except push and pull from Github.
- bspammer 5y agoThe devs at my company have been emailing patches to each other for the last couple of hours. It's been fun actually.
- TacticalCoder 5y agoWait what!? Can't you just reconfigure your .git configs to temporarily use another repo locally? Seriously: for example copying then transforming any one of your local .git repo to a "bare" repository takes like two commands. Anyone with a Linux machine can do it. Then all the devs can temporarily use that repo to push/pull/whatever. I think if you're to the point where you're emailing patches, it's worth it.
- 5y ago
- _birdnerd_ 5y agoI was invited to a party Saturday hosted by the GH CEO, and while this is likely just a huge coincidence, I wonder if some key engineer(s) came down with covid. The friend who invited me said there would be rapid testing at the party.
- odiroot 5y agoThanks Github, whole team could clock out early.
- hashhar 5y agoIt's recovering - everything's working for me now. Much hugs to the GitHub SREs.
- misnome 5y agoThe status SAYS it's all resolved, but PRs are still missing commits pushed during the outage (but before the status page was updated). I guess I'll have to manually push fake commits to get it to update, or wait for whatever internal cache to drop off grumble grumble
- jackson1442 5y agoLooks like everything's green now.
- codetrotter 5y agoI also noticed that starring repos was not working. At that time, a couple of hours ago, only GitHub Actions was listed as having problems on githubstatus. So I was unsure whether being able to star things was malfunctioning for more people or just me. And I was also thinking about how much cruft has been added to the GitHub web UI lately, ever since the acquisition of GitHub by Microsoft. And I yearn for GitHub as it used to be. But the greatest value in GitHub for me is browsing and starring stuff that others have made, so it's not like I can just set up my own instance either. But I wonder about the future of GitHub. I think it will be more and more geared towards big enterprise users, and less and less a place that will be what it used to feel like for individual developers.
- thebouv 5y agoI was on the phone with them getting a demo of Enterprise Managed Users at the time and the demo was failing too, ha. So we looked at the status page together and saw all yellow. Doesn't matter, I wanted EMUs without the demo anyway. Still was funny to happen when I was talking to an engineer at the time.
- niffydroid 5y agoAnd with this, they are still better than bitbucket
- sega_sai 5y agoIt seems to happen again after the first incident was fixed few hours ago (with github actions)
- nightowl_games 5y agoAnyone got any good tips on how to run github action workflows locally or self-hosted? Incidents like this make me want to self-host our CI/CD. I'm confused as to the difference between https://github.com/nektos/act https://github.com/nektos/act and https://github.com/actions/runner https://github.com/actions/runner Is it not possible to use 'runner' to run github actions locally? The docs for this seem like they are extremely sparse - probably because this competes with the paid option?
- duped 5y agoThe runner is a pain in the butt to setup locally. Switch to gitlab or don't rely on community actions, run everything in single bash script.
- batica81 5y agoAnybody having similar issues with Github Actions again? Jobs are failing to connect to external services, or take extremely long time to do so for me.