7 ms·

https://isgithubcooked.com https://isgithubcooked.com Normally I defend GH in the comments of these incidents but it’s been an impressively bad month by their

by gen220 5mo ago

https://isgithubcooked.com https://isgithubcooked.com Normally I defend GH in the comments of these incidents but it’s been an impressively bad month by their standards, even when you filter for critical components filter out sev-2’s and 3’s.

btown 5mo ago
Is the “streak” days of continuous uptime, or of days with at least one downtime incident? I think it’s the latter :]
- joshuaissac 5mo ago
  It looks like it is the number of consecutive days with no incident. If you look at 31 Dec 2025, that corresponds to an 8-day period with no incidents.
  - isityettime 5mo ago
    I guess that also means this year GitHub has not yet made it a single week without an outage of some kind.
- gen220 5mo ago
  It's a streak for continuous uptime, and yeah it is fairly depressing to imagine overseeing that :/
EduardoBautista 5mo ago
May has been filled with critical issues. It seems it's getting worse over time.
- hbn 5mo ago
  Commits are up 14x year-over-year https://x.com/kdaigle/status/2040164759836778878 https://x.com/kdaigle/status/2040164759836778878
  - tom1337 5mo ago
    Yea but thats not really an excuse, is it? They offer a service, (some) people pay for that service and should therefore expect it to work. If GitHub cannot keep up with the growth then they could disable new account registrations or start reducing free tiers so people either use the free tier more mindfully or need to pay for usage-base products like Actions which would GitHub allow to scale.
    - hbn 5mo ago
      I mean it's an easy problem to solve when it's just speculating solutions. But there's a very possible reality where in 5 years guys are making YouTube video essays about the fall of Github caused by their "obviously stupid decision" to throttle access to people who were trying to use their service in record numbers, leaving opportunity for someone else to come in and take their lunch. I don't envy their position of having to scale that fast on something that has to be instant and real-time. As far as I know, you can't do CDN/edge caching shenanigans with a remote git repository like Google can with a YouTube video. It's gotta always be reading/writing to the latest, single source of truth.
      - tom1337 5mo ago
        Sure, backseat commenting is easier and I wouldn't wanna be in charge at github right now, but on the other side there also a reality where we'd see video essays about githubs downfall because their reliability crashed so hard that businesses could not trust them and moved to competitors / self hosted instances which then meant less paid users to subsidize the ever growing demand of the free users.
      - ifwinterco 5mo ago
        Yes it's potentially a write-heavy workload which also needs to be consistent aka the worst case scenario. The easy solutions like caching and read replicas don't work and you're forced to go the route of sharding or similar techniques that have much more painful tradeoffs. I'm not sure if that's why everything keeps breaking but at that scale write-heavy workloads are never going to be easy
pluc 5mo ago
Name one thing Microsoft didn't run into the ground post-acquisition
- elzbardico 5mo ago
  GH was acquired by microsoft some eight years ago. It has been working quite well until recently. People may have had complaints about functionality, features, commercial issues, but the thing used to at least have a decent uptime until recently.
  - bsimpson 5mo ago
    It also used to be run as an independent company with access to MS's resources. Now it's a unit in their AI hype machine.
  - modriano 5mo ago
    MSFT was pretty arms length for the first 5-6 years. I was honestly kind of impressed and it made my opinion of MSFT better. But then AI made it too attractive of a target and MSFT couldn't help but make it a place the former CEO wanted to leave (and it has been running headless for about a year now). It's quite disappointing objectively, but I expected worse from MSFT.
  - chris_money202 5mo ago
    Has nothing to do with Microsoft acquisition... AI usage has increased demand and load. More PRs, more Action runners, more of everything firing. GitHub just wasn't ready for the scale and are now having issues catching up with it as it continues to increase exponentially.
    - voncheese 5mo ago
      Yeah, that and Microsoft has been slow to move the infrastructure to something that scales better to handle that load. The more surpassing part is that Microsoft hasn't figured out a way to manage/contain the AI-sourced traffic better so it doesn't create all this noisy neighbor problems for non-AI usage/users.
rvz 5mo ago
They are already cooked as this has been happening ever since the Microsoft acquisition and it was run to the ground before 2023. At this point you would get better uptime by just self-hosting your own GitLab, Forgejo or Codeberg instance instead of dealing with Github's unreliablity. There is no defending them with their clear neglet and carelessness of the platform.
- vinnymac 5mo ago
  I moved most of my projects off GitHub to Forgejo and will be using Tangled too for public repositories. I don’t think people realize that if you self host Forgejo, you get 99% of the functionality of GitHub with zero of the limitations. Especially if you have the hardware to spare for CI runners. And if self hosting isn’t your thing you can always just use Codeberg and Tangled directly. I’m working on an open source Forgejo browser called Joui. It’s coming along nicely, and is so much snappier than GitHub in every single way.
- pocksuppet 5mo ago
  If all you need is a repository, you don't even need any of these. You need SSH access to a server, and optionally, one of several web front-ends. Git comes with a CGI script that handles public anonymous checkouts via HTTP(S), although since nginx doesn't support CGI, integrating those is a little bit tricky as you need a FastCGI wrapper.
taintlord223 5mo ago
The UI of that page is so nice, should build a github competitor. The user profile / contributions and PR UX is pretty much the entire "hub" product since git is a fully separate offline app.
- embedding-shape 5mo ago
  > The UI of that page is so nice Is it? Seems a text description of "Make a website outlining 'How cooked GitHub' is with a modern style" to basically any LLM would produce exactly that UI and design, literally nothing of that design a human had any influence on, besides the ones selecting what training data the used LLMs was trained with. I think most of us who've tried using LLMs for web-design can recognize that style and design at this point, regardless of model actually used.
  - olmo23 5mo ago
    What really grinds my gears is how easy it is to get better designs out of LLMs. But if you don't ask, you get the default.
    - hansmayer 5mo ago
      Here is a provocative thought - maybe these are the so-called "better designs" from LLMs? It's not like writing English sentences is some huge secret you are sitting on that no one else knows.
      - embedding-shape 5mo ago
        > It's not like writing English sentences is some huge secret you are sitting on that no one else knows. I'd actually say what really makes an excellent engineer stick out among many great engineers, is their ability to communicate clearly and knowing what needs to be communicated vs not, basically being way better at language and communication in general, and they also understand the important of it.
mirekrusin 5mo ago
It's not physically possible to run post-mortems for issues at those rates. They should install OpenClaw for that as well.
- lenerdenator 5mo ago
  AI: The cause of, and solution to, all of your tech debt.
- baalimago 5mo ago
  Perhaps best to simply declare indefinite-mortem
  - miningape 5mo ago
    They should be running pre-mortems every morning at this point
    - drob518 5mo ago
      The operational equivalent of pre-crime.
- embedding-shape 5mo ago
  > It's not physically possible to run post-mortems for issues at those rates. Not at all, you merely move the goal post of at what layer the "root cause" actually could come from! At that speed, it's always something short and sweet, while when you actually want to long-term address things, you have to have time to even investigate organizational issues or whatever the actual problems stem from. But you have half a day? "Post-mortem: Push X wasn't properly analyzed before deployment, in future more testing" and call it a day.
  - drob518 5mo ago
    “A failure occurred. This was caused by something going wrong. Changes to operating guidelines have been instituted to ensure that things will not go wrong in the future unless we happen to do the same thing again.”
root-parent 5mo ago
Like those aviators who draw a picture on flightradar24, if you filter by All Services - Critical, somebody almost about to draw a swastika just in May... Are the AI agents revolting?
rsyring 5mo ago
Of all the sites/graphs I've seen of GH outages, this one is the most striking IMO: https://damrnelson.github.io/github-historical-uptime/ https://damrnelson.github.io/github-historical-uptime/ Unfortunately, it doesn't look like it's being updated with new data. But it wouldn't look any better for GH if it was.
- stogot 5mo ago
  I wonder what the cause of this was? Microsoft Politics? Bureaucracy? Forced move to azure?
  - felooboolooomba 5mo ago
    My guess would be a obnoxious and lethal mixture of all of the above.
    - lazide 5mo ago
      Also AI mandates.
- gen220 5mo ago
  FWIW, I'm not convinced that chart is necessarily an accurate representation of pre-acquisition reality. It would really surprise me if GitHub did not have a single sev-0 pre-acquisition, but it wouldn't surprise me if they were not formally captured and reported in a format that would make its way into their current status page's database.
  - rsyring 5mo ago
    Apparently, you aren't alone. :) https://github.com/DaMrNelson/github-historical-uptime/issues/2 https://github.com/DaMrNelson/github-historical-uptime/issue...
  - crote 5mo ago
    Sure, but it isn't completely wrong either. GH going down used to be quite rare. If it failed to load I'd spend a bunch of time trying to figure out what was wrong with my internet connection, just to read on HN that it was down for everyone. This week GH failed to load and I automatically assumed it was a GH issue - just for it to be followed up a few minutes later by a marketing coworker complaining about internet connectivity. Turns out the office internet connection was dropping about 50% of all packets. It is bad enough that business-side managers are noticing that GH issues are slowing work down. That would've been unimaginable a few years ago.
connorboyle 5mo ago
Wow, it seems that 100% of sev-3 ("critical") incidents in the last year (=365 days) have occurred between April 22, 2026 and now. Is it possible that there has been a change in the way the data are collected/recorded that even partially accounts for this sudden onset?
- gen220 5mo ago
  One tangent, I believe sev-0 is actually "critical" (at least as how I'm used to reading it), and the higher you go the less critical something is. IMO as a github-watcher, I think they changed their definition of what constitutes a sev-0 between sev-1 for the better. In particular, they had a few "sev-1"'s around the turn of the year that would be classified as sev-0's if they happened today. Pre-4/22 GitHub sev-1 was a normal SaaS company's sev-0, imo. So I think their new system is more reflective of reality. My guess is that a few of their big customers bullied them to have more accurate SEV categorization.
  - connorboyle 5mo ago
    Ah, thank you for the correction on sev-0. To be clear, your observation that "they changed their definition of what constitutes a sev-0" is based just on your external observation of incidents and their designations, correct? I.e. they haven't officially released a statement saying they have changed their standards
- lazide 5mo ago
  Waves around it had to break eventually eh?

by gen220 5mo ago

btown 5mo ago

Is the “streak” days of continuous uptime, or of days with at least one downtime incident? I think it’s the latter :]

joshuaissac 5mo ago
It looks like it is the number of consecutive days with no incident. If you look at 31 Dec 2025, that corresponds to an 8-day period with no incidents.
- isityettime 5mo ago
  I guess that also means this year GitHub has not yet made it a single week without an outage of some kind.
gen220 5mo ago
It's a streak for continuous uptime, and yeah it is fairly depressing to imagine overseeing that :/

EduardoBautista 5mo ago

May has been filled with critical issues. It seems it's getting worse over time.

hbn 5mo ago
Commits are up 14x year-over-year https://x.com/kdaigle/status/2040164759836778878 https://x.com/kdaigle/status/2040164759836778878
- tom1337 5mo ago
  Yea but thats not really an excuse, is it? They offer a service, (some) people pay for that service and should therefore expect it to work. If GitHub cannot keep up with the growth then they could disable new account registrations or start reducing free tiers so people either use the free tier more mindfully or need to pay for usage-base products like Actions which would GitHub allow to scale.
  - hbn 5mo ago
    I mean it's an easy problem to solve when it's just speculating solutions. But there's a very possible reality where in 5 years guys are making YouTube video essays about the fall of Github caused by their "obviously stupid decision" to throttle access to people who were trying to use their service in record numbers, leaving opportunity for someone else to come in and take their lunch. I don't envy their position of having to scale that fast on something that has to be instant and real-time. As far as I know, you can't do CDN/edge caching shenanigans with a remote git repository like Google can with a YouTube video. It's gotta always be reading/writing to the latest, single source of truth.
    - tom1337 5mo ago
      Sure, backseat commenting is easier and I wouldn't wanna be in charge at github right now, but on the other side there also a reality where we'd see video essays about githubs downfall because their reliability crashed so hard that businesses could not trust them and moved to competitors / self hosted instances which then meant less paid users to subsidize the ever growing demand of the free users.
    - ifwinterco 5mo ago
      Yes it's potentially a write-heavy workload which also needs to be consistent aka the worst case scenario. The easy solutions like caching and read replicas don't work and you're forced to go the route of sharding or similar techniques that have much more painful tradeoffs. I'm not sure if that's why everything keeps breaking but at that scale write-heavy workloads are never going to be easy

pluc 5mo ago

Name one thing Microsoft didn't run into the ground post-acquisition

elzbardico 5mo ago
GH was acquired by microsoft some eight years ago. It has been working quite well until recently. People may have had complaints about functionality, features, commercial issues, but the thing used to at least have a decent uptime until recently.
- bsimpson 5mo ago
  It also used to be run as an independent company with access to MS's resources. Now it's a unit in their AI hype machine.
- modriano 5mo ago
  MSFT was pretty arms length for the first 5-6 years. I was honestly kind of impressed and it made my opinion of MSFT better. But then AI made it too attractive of a target and MSFT couldn't help but make it a place the former CEO wanted to leave (and it has been running headless for about a year now). It's quite disappointing objectively, but I expected worse from MSFT.
- chris_money202 5mo ago
  Has nothing to do with Microsoft acquisition... AI usage has increased demand and load. More PRs, more Action runners, more of everything firing. GitHub just wasn't ready for the scale and are now having issues catching up with it as it continues to increase exponentially.
  - voncheese 5mo ago
    Yeah, that and Microsoft has been slow to move the infrastructure to something that scales better to handle that load. The more surpassing part is that Microsoft hasn't figured out a way to manage/contain the AI-sourced traffic better so it doesn't create all this noisy neighbor problems for non-AI usage/users.

rvz 5mo ago

They are already cooked as this has been happening ever since the Microsoft acquisition and it was run to the ground before 2023. At this point you would get better uptime by just self-hosting your own GitLab, Forgejo or Codeberg instance instead of dealing with Github's unreliablity. There is no defending them with their clear neglet and carelessness of the platform.

vinnymac 5mo ago
I moved most of my projects off GitHub to Forgejo and will be using Tangled too for public repositories. I don’t think people realize that if you self host Forgejo, you get 99% of the functionality of GitHub with zero of the limitations. Especially if you have the hardware to spare for CI runners. And if self hosting isn’t your thing you can always just use Codeberg and Tangled directly. I’m working on an open source Forgejo browser called Joui. It’s coming along nicely, and is so much snappier than GitHub in every single way.
pocksuppet 5mo ago
If all you need is a repository, you don't even need any of these. You need SSH access to a server, and optionally, one of several web front-ends. Git comes with a CGI script that handles public anonymous checkouts via HTTP(S), although since nginx doesn't support CGI, integrating those is a little bit tricky as you need a FastCGI wrapper.

taintlord223 5mo ago

The UI of that page is so nice, should build a github competitor. The user profile / contributions and PR UX is pretty much the entire "hub" product since git is a fully separate offline app.

embedding-shape 5mo ago
> The UI of that page is so nice Is it? Seems a text description of "Make a website outlining 'How cooked GitHub' is with a modern style" to basically any LLM would produce exactly that UI and design, literally nothing of that design a human had any influence on, besides the ones selecting what training data the used LLMs was trained with. I think most of us who've tried using LLMs for web-design can recognize that style and design at this point, regardless of model actually used.
- olmo23 5mo ago
  What really grinds my gears is how easy it is to get better designs out of LLMs. But if you don't ask, you get the default.
  - hansmayer 5mo ago
    Here is a provocative thought - maybe these are the so-called "better designs" from LLMs? It's not like writing English sentences is some huge secret you are sitting on that no one else knows.
    - embedding-shape 5mo ago
      > It's not like writing English sentences is some huge secret you are sitting on that no one else knows. I'd actually say what really makes an excellent engineer stick out among many great engineers, is their ability to communicate clearly and knowing what needs to be communicated vs not, basically being way better at language and communication in general, and they also understand the important of it.

mirekrusin 5mo ago

It's not physically possible to run post-mortems for issues at those rates. They should install OpenClaw for that as well.

lenerdenator 5mo ago
AI: The cause of, and solution to, all of your tech debt.
baalimago 5mo ago
Perhaps best to simply declare indefinite-mortem
- miningape 5mo ago
  They should be running pre-mortems every morning at this point
  - drob518 5mo ago
    The operational equivalent of pre-crime.
embedding-shape 5mo ago
> It's not physically possible to run post-mortems for issues at those rates. Not at all, you merely move the goal post of at what layer the "root cause" actually could come from! At that speed, it's always something short and sweet, while when you actually want to long-term address things, you have to have time to even investigate organizational issues or whatever the actual problems stem from. But you have half a day? "Post-mortem: Push X wasn't properly analyzed before deployment, in future more testing" and call it a day.
- drob518 5mo ago
  “A failure occurred. This was caused by something going wrong. Changes to operating guidelines have been instituted to ensure that things will not go wrong in the future unless we happen to do the same thing again.”

root-parent 5mo ago

Like those aviators who draw a picture on flightradar24, if you filter by All Services - Critical, somebody almost about to draw a swastika just in May... Are the AI agents revolting?

rsyring 5mo ago

Of all the sites/graphs I've seen of GH outages, this one is the most striking IMO: https://damrnelson.github.io/github-historical-uptime/ https://damrnelson.github.io/github-historical-uptime/ Unfortunately, it doesn't look like it's being updated with new data. But it wouldn't look any better for GH if it was.

stogot 5mo ago
I wonder what the cause of this was? Microsoft Politics? Bureaucracy? Forced move to azure?
- felooboolooomba 5mo ago
  My guess would be a obnoxious and lethal mixture of all of the above.
  - lazide 5mo ago
    Also AI mandates.
gen220 5mo ago
FWIW, I'm not convinced that chart is necessarily an accurate representation of pre-acquisition reality. It would really surprise me if GitHub did not have a single sev-0 pre-acquisition, but it wouldn't surprise me if they were not formally captured and reported in a format that would make its way into their current status page's database.
- rsyring 5mo ago
  Apparently, you aren't alone. :) https://github.com/DaMrNelson/github-historical-uptime/issues/2 https://github.com/DaMrNelson/github-historical-uptime/issue...
- crote 5mo ago
  Sure, but it isn't completely wrong either. GH going down used to be quite rare. If it failed to load I'd spend a bunch of time trying to figure out what was wrong with my internet connection, just to read on HN that it was down for everyone. This week GH failed to load and I automatically assumed it was a GH issue - just for it to be followed up a few minutes later by a marketing coworker complaining about internet connectivity. Turns out the office internet connection was dropping about 50% of all packets. It is bad enough that business-side managers are noticing that GH issues are slowing work down. That would've been unimaginable a few years ago.

connorboyle 5mo ago

Wow, it seems that 100% of sev-3 ("critical") incidents in the last year (=365 days) have occurred between April 22, 2026 and now. Is it possible that there has been a change in the way the data are collected/recorded that even partially accounts for this sudden onset?

gen220 5mo ago
One tangent, I believe sev-0 is actually "critical" (at least as how I'm used to reading it), and the higher you go the less critical something is. IMO as a github-watcher, I think they changed their definition of what constitutes a sev-0 between sev-1 for the better. In particular, they had a few "sev-1"'s around the turn of the year that would be classified as sev-0's if they happened today. Pre-4/22 GitHub sev-1 was a normal SaaS company's sev-0, imo. So I think their new system is more reflective of reality. My guess is that a few of their big customers bullied them to have more accurate SEV categorization.
- connorboyle 5mo ago
  Ah, thank you for the correction on sev-0. To be clear, your observation that "they changed their definition of what constitutes a sev-0" is based just on your external observation of incidents and their designations, correct? I.e. they haven't officially released a statement saying they have changed their standards
lazide 5mo ago
Waves around it had to break eventually eh?