5 ms·
Incident with Webhooks
- munksbeer 1y agoGetting failed pushes, failed PR creation, failed CI pipelines. Who else?
- digitalsushi 1y agoprobably everyone since its been on their status page since before this was asked
- hakube 1y agoCan't merge PRs atm
- madethemcry 1y agoYeah there are some issues. PR is stuck at "Checking for the ability to merge automatically..." By accident I landed on https://us.githubstatus.com/ https://us.githubstatus.com/ and everything was green. At first, I thought, yeah sure, just report green, then I realized "GitHub Enterprise Cloud" in the title. There is also a EU mirror: https://eu.githubstatus.com https://eu.githubstatus.com Edit: The report just updated with the following interesting bit. > We identified a faulty network component and have removed it from the infrastructure. Recovery has started and we expect full recovery shortly.
- nightpool 1y agoThat's interesting—my understanding is that Github Enterprise Cloud is part of the same infrastructure as Github.com, so this status page seems maybe incorrect? Probably some missing step in the runbook to update both of these pages at the same time.
- christophilus 1y agoThe thing that's been annoying me for a few weeks is an always-on notification, but my notification page shows no unread notifications.
- diath 1y agoI had that happen when somebody tagged me in a private repository that was later deleted (?). You can fix it through the API by generating an API token in your settings with notifications permission on and using this (warning: it will mark all your notifications as read up to the last_read_at day): curl -L \ -X PUT \ -H "Accept: application/vnd.github+json" \ -H "Authorization: Bearer <YOUR-TOKEN>" \ -H "X-GitHub-Api-Version: 2022-11-28" \ https://api.github.com/notifications \ -d '{"last_read_at":"2025-10-09T00:00:00Z","read":true}'
- delfinom 1y agoYes private repos that are deleted leave notifications behind. GitHub has been experiencing mass waves of crypto scam bots opening repos and mass tagging tens of thousands of users on new issues. Using the issue content body to generate massive scam marketing like content bodies.
- masklinn 1y agoYep, there was a huge spate of spam with hundreds of people pinged, and when they got reported / deleted the notifications didn't go...
- MyOutfitIsVague 1y agoI got hit with one of these, then commented on GitHub meetsn issue about it, and ironically got a hundred notifications for everybody's comments, many complaining about phantom notifications.
- RyJones 1y agoI have been getting added to spam repos and orgs several times a day for weeks. it's annoying
- masklinn 1y agoAn other trick is to go into the "done" tab, and move at least 25 issues back to unread. Then you can click the checkbox at the top and then "select all", and it'll mark the phantom notifications as read.
- lbrito 1y agoI experienced an outage (website and any push, pull commands) that lasted for about 1-2 hours on Oct 7th but didn't see anything on their status page. There was definitively a spike on https://downdetector.ca/status/github/ https://downdetector.ca/status/github/, so I know it wasn't just my ISP.
- mparnisari 1y agoI experienced that too! In Canada. It was a head-scratcher, none of my teammates had issues, and i could access just fine on my phone but i couldn't on my home's wifi.
- lbrito 1y agoSame. Fraser Valley?
- koolba 1y agoIt’s kind of funny that the top two posts right now are: 1. Why Self-Host? 2. GitHub Issues
- chistev 1y agoWhat's the joke?
- dvmazur 1y agoGitHub Pages are affected too
- logicallee 1y agoGithub hosts git repos, so it's an example of when self-hosting git repos on one's own servers could remain operational despite a github outage.
- jsheard 1y agoAnd yesterday we had "GitHub pausing feature development to prioritize moving infra to Azure", immediately followed by them breaking their infra.
- cluckindan 1y agoWell, of course the corporation wants to dogfood their platform. Expecting more and more downtime and random issues in the future.
- progbits 1y agoI thought they were already on Azure? At least I'm pretty sure the runners are, our account rep keeps trying to get us to use their GPU runners but they don't have a good GPU model selection and it seems to match what azure offers.
- thelastgallon 1y agoIts better for github to self-host.
- anon7000 1y agoHow long before moderately sized companies start hosting their own git servers again. Surely it wouldn’t be that difficult unless your repos are absolutely massive. GitHub outages are so common these days
- bntyhntr 1y agoAnd then you need to add another server to the infra / netops / tools team's maintenance burden and then they take it down for an upgrade and it doesn't come back up etc etc. I don't think outages/downtime are necessarily a good reason to switch to self-hosting. I worked at a company that self-hosted the repo and code review tool and it was great, but it still had the same issues.
- mjr00 1y agoYeah, as someone old enough to have worked at mid-sized companies before cloud-everything became the norm, self-hosting is overly romanticized. People think you'll get a full infrastructure team dedicated to making sure your self-hosted Git/Artifactory/Jira/Grafana/whatever runs super smoothly and never goes down. In reality it ends up being installed by a dev or IT as sort of a side project, and once the config is hacked together (and of course it's a special pet server and not any kind of repeatable infrastructure setup with Ansible or Docker, so yes you're stuck on Ubuntu 12.04 for a decade) they let it "just run" forever because they don't want to touch it (because making changes is the #1 reason for outages) so you're constantly 2+ years behind the latest version of everything. It's true that outages are probably less frequent, as a consequence of never making any changes, however when something does break e.g. security forces someone to actually upgrade the 5-years-since-end-of-support Ubuntu version and it breaks, it may take several days or weeks to fix because nobody actually knows anything about the configuration because it was last touched 10 years ago by 1 guy who has long left the company.
- hypeatei 1y agoI don't think running your own git server on its own is what's preventing this. It's all the other things you're missing like: CI/CD pipelines, code review tools, user management, etc...
- Kavelach 1y agoI wish the most popular software forge didn't include a bunch of other software solutions like issue tracking or forums. Having everything in one service definitely increases interoperability between those solutions, but it definitely decreases stability. In addition, each of the other systems is not the best in their class (I really detest GH Actions for example). Why do so many solutions grow so big? Is it done to increase enterprise adoption?
- cortesoft 1y agoIf the alternative is each user has to patch together all of the different solutions into one, you are just increasing the number of parts that can go wrong, too. And when they do, it won't be immediately clear who the issue is with. I do agree there are issues with a single provider for too many components, but I am not sure you get any decreased stability with that verse having a different provider for everything.
- MyOutfitIsVague 1y agoI agree to a degree, but issue tracking being able to directly work with branches and PRs is natural enough, and then discussions can share a lot of code with the issue tracker. Getting the same level of interoperability with a separate tool takes significantly more work on both sides, so the monolithic approaches tend to thrive because it can get out the door faster and better. Forgejo is doing the same thing with its actions. Honestly, I'd prefer if something like Woodpecker became the blessed choice instead, and really good integration with diverse tools was the approach.
- poly2it 1y agoOf everything potentially causing scope creep in GitHub, issue tracking and forums might be the least out of scope. That said, I agree that the execution of many features in GitHub has been lacking for some time now. Bugs everywhere and abysmal performance. We're moving to Forgejo at $startup.
- 0-bad-sectors 1y agoOh it's that time of the week again.
- nomilk 1y agoI stopped using Actions for side projects a few months ago, things are simpler now (I run tests locally). I felt like Actions were a time sink that trick you into feeling productive - like you're pursuing 'best practice' - while stealing time that could otherwise be spent talking to users or working on your application.
- raybb 1y agoMakes sense for side projects. I think there's real value for open source projects so people can get feedback quickly and maintainers can know that the tests are passing quickly.
- nixpulvis 1y agoUm... maybe some actions setups are overly complex, but CI/CD is valuable if done well. For example, running tests before merge ensures you don't forget to. Running lints/formatters ensures you don't need to refactor later and waste time there. For my website, it pushes main automatically, which means I can just do something else while it's all doing it's thing. Perhaps you should invest in simplifying your build process instead?
- nomilk 1y agoIt's valuable, but at what cost? The day I forget to run tests before merging I'll set up CI/CD (hasn't happened before, unlikely, but not impossible). My build process is gp && gp heroku main. Minor commits straight to main. Major features get a branch. This is manual, simple and loveable. And involves zero all-nighters commit-spamming the .github directory :)
- nixpulvis 1y agoI mean, I agree it would be nice to be able to test actions locally (maybe there's a tool for this). But I keep my actions very simple, so it rarely takes me a lot of time to get them right. See https://github.com/nixpulvis/grapl/blob/master/.github/workflows/rust.yaml https://github.com/nixpulvis/grapl/blob/master/.github/workf... If you want more complex functionality, that's why I suggested improving your build system, so the actions themselves are pretty simple. Where things get more frustrating for me is when you try using more advanced parts of actions like releases and artifacts which aren't as simple as running a script and checking it's output/exit code.
- nimbius 1y agoFor anyone who wants context, here is the entire history of github "Issues" https://www.githubstatus.com/history https://www.githubstatus.com/history seems like Microsoft can't keep this thing from crashing at least three times a month. At this rate it would probably be cheaper just to buy out Gitlab. Wondering when M$ will cut their losses and bail.
- kelvinjps10 1y agoSo they buy company, ruin it and then start again forever?
- olao99 1y agoI'm not looking forward to the Azure migration and potential for more issues in the coming year
- sl-1 1y agoCould you elaborate? Or link some source? I could quickly find this pointing towards them running their own servers https://github.com/orgs/community/discussions/27606 https://github.com/orgs/community/discussions/27606 If the current state of GH availability is without Azure-induced additional unreliability, I truly fear what it will be on Azure Edit: Found the discussion about this https://news.ycombinator.com/item?id=45517173 https://news.ycombinator.com/item?id=45517173
- montroser 1y agoTo run your github actions locally, we've had decent success with this tool: https://github.com/nektos/act https://github.com/nektos/act Just be warned if you try it out that if you don't specify which workflow to run, it will just run them all!