Hacker News Search

blipvert 2mo ago
Like the way that they shipped ‘95 without a browser?
- aleph_minus_one 2mo ago
  > Like the way that they shipped ‘95 without a browser? Around the time when Windows 95 was released, Microsoft did predict that the internet will become popular also among private users. That is why - they included Winsock (Windows' TCP/UDP/IP stack) as native part of Windows 95 (in opposite to Windows 3.x where you had to install a third-party implementation of the Winsock API such as Trumpet Winsock), though it was typically not installed by default, - support to use MSN (the dial-up online service and internet servicen provider [1]) was pre-installed with Windows 95. What was not clear at that time was that the world wide web would become the killer application for the internet for consumer users (at that time, the more likely killer application of the internet for consumers was seen in proprietary content providers such as AOL, Compuserve or MSN), so Microsoft provided no web browser with the original version of Windows 95. But even at that time, it was conceivable, though not clear that the world wide web could become important. So, already at the release time of Windows 95, Microsoft provided Internet Explorer 1.0 as part of the Plus! add-on pack for Windows 95. With Windows 95 OEM Service Release 1, the Internet Explorer 2.0 got part of Windows 95, and Windows 98 introduced the (optional) Active Desktop [2], in was centered around the Internet Explorer. So, in my opinion Microsoft did everything "correct" here: they provided Internet Explorer from beginning on as an optional separate component (Plus! add-on pack), and once it was obvious that the world wide web would become very important, they reacted fast and included the Internet Explorer in the next service release of Windows 95 (Windows 95 OEM Service Release 1). -- [1] https://en.wikipedia.org/wiki/MSN_Dial-Up_Internet_Access https://en.wikipedia.org/wiki/MSN_Dial-Up_Internet_Access [2] https://en.wikipedia.org/wiki/Active_Desktop https://en.wikipedia.org/wiki/Active_Desktop

bhouston 2mo ago

> Github is struggling because AI-boosted coding increased the number of commits 14x in the past year, and the pace is still accelerating. Sure, but how is OpenAI, and Claude handling the increased usage? They are handling it a lot better than Github. Just because you have fast growth does not mean you can not handle it. Instead you have to realize that you are entering into a high growth phase and design around that. It has been clear that we are in a high growth phase for around a year now and Github still hasn't adapted to that properly. At some point you have to admit that Github ready for the moment.

traviswingo 2mo ago
GitHub is on Azure. Azure is notorious for being terrible. There was a post here recently from an ex-Azure engineer explaining all of this in a multi-part series. Edit: Found the thread: https://news.ycombinator.com/item?id=47616242 https://news.ycombinator.com/item?id=47616242
- Tiberium 2mo ago
  Isn't a lot of OpenAI infra also on Azure? They seem to be faring better than GitHub.
  - bhouston 2mo ago
    I bet OpenAI has failovers to other data centers. I think they have a variety of data center provides all over the place.
    - prewett 2mo ago
      There's already a noun for this: "resources"
- moralestapia 2mo ago
  Q. how can someone publish an article like that, with their name, etc... Don't they sign NDAs? Do they not care? Are they not enforceable? Very curious to know.
  - ornornor 2mo ago
    Maybe they didn’t take the severance part that makes the NDA valid? IIRC for an NDA to be enforceable there has to be a monetary consideration to it. Or maybe it’s for no competes?
    - jaredsohn 2mo ago
      NDAs often get signed when you join a company.
  - dmitrygr 2mo ago
    Some people reach a level of DGAF where it will not matter Some fuck-ups have such long-lasting effects that they outlast the NDAs that would cover them
- Peanuts99 2mo ago
  I didn't think they'd completed the move yet, it's only partially on Azure - which to be fair could be another problem in itself.
tekno45 2mo ago
OpenAI and Claude don't accept commits and host PRs? How are they the same?
- logicchains 2mo ago
  They're undergoing the same kind of load. Almost every AI commit that's putting pressure on GitHub's servers was written on OpenAI and Anthropic's servers.
  - simonw 2mo ago
    Scaling ChatGPT and scaling GitHub are very different problems. For LLMs, prompt handling is effectively stateless. They do try to route follow-up prompts to the same cluster to benefit from prompt caching, but those can be effectively sharded. They also need to write results to storage but that's on a per-customer basis, so also easy to shard. GitHub is a shared space, where commits and issues and PRs need to become instantly available to all readers across all geographies. They have a very different shape of scaling challenge to the LLM vendors.
    - bhouston 2mo ago
      > GitHub is a shared space, where commits and issues and PRs need to become instantly available to all readers across all geographies. They have a very different shape of scaling challenge to the LLM vendors. Github data is accessible by all (if open source), but they should be partition-able by individual repository (and their related forks.). Thus while there is more shared state across users, it isn't fully shared state. And they have been working on this semi-shared state design for over 10 years now.
  - etc-hosts 2mo ago
    Every response from GitHub must be deterministic. OpenAI and Anthropic do not have this problem.
    - 27183 2mo ago
SamBam 2mo ago
OpenAI and Anthropic are spending hundreds of billions of dollars that they may never recoup forcing the building massive new data centers in the face of widespread protest. You think GitHub ought to match that?
- bhouston 2mo ago
  > OpenAI and Anthropic are spending hundreds of billions of dollars that they may never recoup forcing the building massive new data centers in the face of widespread protest. So has Github's owner Microsoft. It's build out is on par with OpenAI and Anthropic. > You think GitHub ought to match that? Microsoft is. The issue is that the software stack is not stable. That is clearly the issue here. It isn't a hardware build out issue. It is related to the architecture of Github.
theappsecguy 2mo ago
Well they heavily limit free tier usage and for paid users they are charging a tiered pricing model that ramps up very aggressively. Github is a lot more generous in this sense and it has a lot more surface area in terms of the functionality it serves
remus 2mo ago
> Sure, but how is OpenAI, and Claude handling the increased usage? By spending tens of billions in investor cash. It is not clear what a sustainable business model looks like for these companies at the moment. > They are handling it a lot better than Github. Are they? Anthropic has fairly frequent issues. There's high growth and then there's demand increasing 10x in a year. For a mature business like github that's a massive, and I can't think of many big businesses who would be able to scale smoothly to accommodate that kind of demand.
- mixmastamyk 2mo ago
  Yes, OAI is losing tens of billions, Anthro is losing billions, and MS is cashflow positive and rewarded for that by Wall St, according to recent earnings reports.
baal80spam 2mo ago
> They are handling it a lot better than Github. No, they are not. https://status.claude.com/ https://status.claude.com/ https://www.githubstatus.com/ https://www.githubstatus.com/
1attice 2mo ago
This would require foresight, budget, and attention. MS currently has none of these.
maccard 2mo ago
For a very long time this year Claude was down multiple times a week. Can’t speak for OpenAI
stronglikedan 2mo ago
Open AI and Claude were not nearly as mature as Github when the boom started. That's a lot of legacy infra to scale up.
fl0id 2mo ago
anthropic frequently rate limits or is just unavailable... so I wouldn't exactly say they are handling it better.
never_inline 2mo ago
GitHub is not able to charge in proportion to commits pushed, and can't scale in proportion either without it being too expensive.
sigseg1v 2mo ago
Claude has 1 to 2 nines of uptime if you check their uptime history on their site. And I'll note that their definition of up is... generous. Plenty of times I encounter it saying no issues when there are abundant service issues.

arjie 2mo ago

The obvious answer is to end free unlimited private repos. Source is no longer a specific thing. GitHub is just a free S3 or Azure provider.

SoftTalker 2mo ago
I think this would cut out a lot of the mindless commit activity. Charge $1/month per repo. If your project isn't worth that, run your own git or use another free forge.
- odo1242 2mo ago
  I think GitHub leans on this free repos thing a lot for getting new users though. Like I could see people just moving to GitLab
  - hakunin 2mo ago
    …and encountering the same issues most likely.
    - odo1242 2mo ago
      Fair, but the point is GitHub’s model is: Free Users use GitHub => when a company needs source control => all their coders use GitHub => company buys expensive enterprise contract. If there’s less of an advantage to using free GitHub then that funnel works less effectively.
  - neko_ranger 2mo ago
    Free private repos on github is relatively new, only introduced after a rise in popularity of gitlab (which has free private repos). Seems like github is too popular for itself.
    - klausa 2mo ago
      I know you said "relatively", but it's been seven and a half years: https://github.blog/changelog/2019-01-08-pricing-changes/ https://github.blog/changelog/2019-01-08-pricing-changes/
- hei-lima 2mo ago
  GH isn't good enough to charge that much for its core services. 90% of its user base would just find better or cheaper alternatives, and the only ones remaining would be legacy enterprise codebases that are just impossible or too expensive to migrate.
- kypro 2mo ago
  80/20 rule would suggest that you don't need to apply $1 to all users, just those who create an excessive number of AI slop repos or make an excessive number of commits. Could do similar for GH actions usage.
- joshspankit 2mo ago
  What about $1/year per private repo?
- zmj 2mo ago
  The incentives that creates are really bad: it encourages users to organize their code across the smallest number of repos, and I'd bet anything that larger repos are disproportionately more expensive for GitHub than smaller ones. It's very possible that a per-repo charge would make things worse.
- CrimsonCape 2mo ago
  You are way too optimistic. What will realistically happen is they charge $2 per month and the quality doesn't change. The $2 per month gross will go where it always goes: launches, exec salaries, shareholders.
imjonse 2mo ago
I would have thought it's public repos that got the most AI slop recently as a form of gathering karma and boost CVs. It would be nice to know if private repo commits increased instead.
swingboy 2mo ago
A lot of private repos are probably just taking up cheap storage space, not seeing activity that would cause additional load.
never_inline 2mo ago
You can also limit number of free commits to, say 100 per month.
- gspr 2mo ago
  So if I want to push an existing project with 1000 commits to GitHub for the first time, I gotta do it over the course of 10 months? Encouraging big commits seems antithetical to the idea of git in general.
  - deleted 1mo ago
    [deleted]
fg137 2mo ago
Public or private doesn't matter. Vibe coders aren't worried about others using their AI slop

0xbadcafebee 2mo ago

If that's the superficial cause, then the root cause is a lack of limits. Any production system in the world will go down from too much load if you don't limit the load. You have to do performance testing to determine what the system's peak utilization is, and put limits in place to prevent exceeding that limit. There are multiple kinds of limits you need. First you need the raw i/o limits on things like networking, load balancer requests per second, etc so you don't slow to a crawl on high load; you can throttle max i/o so demanding users don't steal performance for others, or even throttle network traffic to the client IP and internal services. Then you need additional limits at app layer, like number of requests you can process that make DB calls (as most DBs have a connection and request limit), or limit API requests to X/minute. Then you need more business logic limits, like max number of concurrent logged-in users, max number of repos a user can have, max number of requests for free users, priority for paid requests, etc. When you hit a limit, you have a couple choices. You can shed load (dropping calls immediately, which results in 503 errors for users, or even network disconnects), but it makes a very poor user experience, and can cause larger issues/conflicts in a distributed system. You can instead keep a max number of active sessions, and adjust that number over time, to ensure that (on average) your users have a stable experience; this requires implementing a "waiting room" for new sessions when old ones expire or log out; still not great for users, but it prevents everyone from getting random errors, which means the majority of users have a good experience. There are way more kinds of limits to implement. Each business has to analyze its entire system to identify performance bottlenecks and implement limits to ensure the majority of users continue to have good service. Without those limits, you get a shit experience for everyone, your brand is tarnished, and customers run for the hills. Finally, the performance testing helps you plan how much infrastructure you need to serve a given number of users. If you plan properly, you know X infra allows for Y number of users. Once you exceed Y users, you need to start throttling or stopping new user signup (and requests per minute) until you can increase capacity. Continuing to allow signups and additional usage, without the capacity to support it, results in what we see today. But competent engineers know all this. If they had competent engineers, and it gets this bad anyway, the problem is management. Microsoft likely already made back what they acquired GitHub for. They probably figured they could keep riding the gravy train until the train went off the rails, and either eventually sell GitHub or write it off. Such is business.

darkr 2mo ago
> until the train went off the rails Ironically the core application is written in rails, which may be part of the issue
thex10 2mo ago
I really enjoyed this read, as I am starting to grapple with some of this now. Do you have any resources for learning more about this stuff and these tradeoffs?
lilerjee 2mo ago
Too complicated and the price is very huge. Fixing small errors and keeping going on maybe is a good business strategy. The last result is probably what you said.

sharts 2mo ago

I would take what C-level execs say with a grain of salt. Remember these folks hire people with plenty of system design experience. They likely also have the money for hardware resources. If load was an issue these things would have been solved already. Or they are purposely causing millions of dollars of productivity loss daily by not solving this. Something more serious is at play. And I want to hear from the actual underlings.

pan69 2mo ago
> I would take what C-level execs say with a grain of salt. 100% agree. > Remember these folks hire people with plenty of system design experience. Not my overall experience 30 years in the industry. > They likely also have the money for hardware resources. You'll always be asked to do more with less, at least, that's my own experience.
- joshspankit 2mo ago
  Also my experience
  - jasper1111 2mo ago
    Also my experience
    - rawland 2mo ago
      +1
      - haddr 2mo ago
        Also my experience
srmatto 2mo ago
Infra is a cost center for a business unless your a PaaS and even then maybe still. Cost centers are usually ferreted out for savings, not investment.
- roughly 2mo ago
  Infra is a cost center right up until its absence begins to undercut your value proposition to customers, and then it becomes an investment. GitHub seems well on the wrong side of that equation right now.
  - geodel 2mo ago
    In general true. But most of paying customers, including enterprise have made a bet that they cut down expense and resources on operational experience, system management knowledge to minimum because cloud . So even if these cloud/SAAS solutions perform poorly it is not like customers have wherewithal to create alternate solutions.
    - conartist6 2mo ago
      > it is not like customers have wherewithal to create alternate solutions. it's not impossible. they don't, right up until they do
browningstreet 2mo ago
> They likely also have the money for hardware resources. Amazing you can say this with a straight face in these DC building times. Have you tried getting Azure east instances anytime lately?
- mckee_plus_plus 2mo ago
  I've actually never tried to get an Azure instance. Azure can't earn my trust with SSO, let alone hosting my compute.
nobodysaidso 2mo ago
>They likely also have the money for hardware resources. Hey man, in case you hadn't heard, the hardware industry is having a lot of trouble keeping up. "Expand our capacity" isn't as easy as it was 2 years ago.
Illniyar 2mo ago
More load does not just mean needing to increase capacity, which would be simple for Microsoft to do. Additional load reveals limits and breaklines that were never known before - your temporary integer IDs are suddenly too small, you reach the maximum file descriptors in a Linux distro, your hash function suddenly has repeating conflicts, your DB index is suddenly oversaturated. Having massive exponential growth of your traffic would probably require constant refactoring to meet the ever changing system limits.
- lokar 2mo ago
  A good rule of thumb I once heard (from Jeff Dean, I think) was that (in a complex system) every 10X you go up in scale you need to re-consider basic design elements, trade-offs, etc, and probably change some of them. It's hard to design something (large and complex) for much more than 10X scale with only HW capacity increases.
  - transcriptase 2mo ago
    There are iirc some good write ups from years ago by employees at Blizzard about these things and why “spinning up more servers” wouldn’t suddenly let millions of people login to games simultaneously on launch days.
fooey 2mo ago
I think owning GitHub is a prestige play, not necessarily thought of as a profit center. Now however, it's competing for resources directly with the corporations core strategic plans. GitHub having issues is embarrassing, but the shareholders don't really care.
- lokar 2mo ago
  I've always wondered about that. I've suspected the same as you, it's mostly a brand / "ecosystem" thing.
27183 2mo ago
My pet theory is they backed themselves into a corner with Actions. That was a step change in computational scaling effort. It also fell immediately on the heels of the Microsoft acquisition. Something about these two factors seems to have conspired to wreak absolute havoc on their uptime in the ~7yr hence. The AI slopocalypse hasn't helped matters, but the foundations were laid far earlier. Would love to hear from an insider, either for or against this conspiracy theory.

vb-8448 2mo ago

I'm curios what is the bottleneck then? Because github(ms) has enough resources to hire competent people and provision needed hw to solve scaling the issues.

c0rruptbytes 2mo ago
need competent people to hire competent people first - it seems like their only directive is to raise $MSFT not make good tech (talking about leadership, not my lovely msft engineers reading this)
moralestapia 2mo ago
Yeah, aren't they super geniuses with infra that scales with a button, etc.?
- aleph_minus_one 2mo ago
  > infra that scales with a button That is how Microsoft markets Azure. :-)
lallysingh 2mo ago
Probably contention on databases.
tsss 2mo ago
Looking at the design of Github Actions, I don't see any competency.
dspillett 2mo ago
Ignoring the possibility of plain ol' management incompetence, when things are changing as fast as they are the problem becomes one of lead time. Hiring competent people does not happen at the click of a finger, nor in the current climate does getting new hardware resources. Hardware probably isn't going to do a lot with demand rising 10x or more at peak times, if they had scaled and tested systems with a view to supporting an order of magnitude more requests than expected then all that contingency resource is already gone. Adding 10% more hardware is going to be very expensive and will likely not have that much effect anyway without design changes to make better use of that hardware, scaling by orders of magnitude often reveals bottlenecks that were not apparent at the previous scales, or were apparent but dismissed (I imagine somewhere in a comment or design doc someone having written “we will need to rethink this if the system is to be asked to process 10ˣ⁺¹ things/period rather than 10ˣ”). And once you've hired those people, even if they get up to speed pretty much instantly, a system change isn't going to be designed/made/rolled-out in short order so patchwork fixes are still needed in the meantime, and any roll-out probably needs to be gradual which adds further complication. Scaling isn't easy. Scaling fast is a royal PITA.

maccard 2mo ago

GitHub’s reliability issues started earlier than the boom in agentic coding they talk about. I have comments on here from 2025 talking about massive outages with actions and issues.

fooey 2mo ago
When you're the uncontested market leader, you don't budget operations to be perfectly reliable, you budget to not be too unreliable. All those 9's are expensive after all. The AI boom caught them napping though, and they're slipping well into the too unreliable category
- aleph_minus_one 2mo ago
  > The AI boom caught them napping though Microsoft, who invested eyewatering sums into AI, even near to betting the whole company on an AI boom, is surprised about the AI boom?
  - minraws 2mo ago
    Microsoft bet everything on possible AGI, no one thought the result would be a bonafide commit printing machine.
    - samudrijan 2mo ago
      PRRRR
    - jasonvorhe 2mo ago
      That's not what Nadella said 1-2 years ago. Everyone seemed pretty aware of LLMs soon being able to handle a lot of steps of software development. I bet it's mostly due to Azure being brittle and the AI boom is just a nice cover story (not saying it's not challenging to handle all that new load) to save face and sell more Azure services.
      - redshiftza 2mo ago
        100% if this post (https://isolveproblems.substack.com/p/how-microsoft-vaporized-a-trillion https://isolveproblems.substack.com/p/how-microsoft-vaporize...) is anything to go by, it could explain much of the scaling & reliability issues.Perhaps github not being built to scale this fast + being hosted on azure is closer to the truth.
  - creshal 2mo ago
    Microsoft considers it a bonus if a hype lets one department cripple another department.
chocrates 2mo ago
GitHub uptime is cyclical. It's bad because they won't focus on the core, and then the right users complain and GitHub focuses on core again before it repeats. This is just the latest round exacerbated by vibe coding
- naikrovek 2mo ago
  [dead]
- infamouscow 2mo ago
  Another commenter replied, but was downvoted by angry employees of Microsoft for a fairly anodyne statement. Microsoft management knows this. It's not like there haven't been attempts to solve these core problems. The issue is GitHub was built by exceptional engineers, and none of those people are still around. I maintain all these problems began when they migrated off Erlang and egitd. :)
dgellow 2mo ago
It was related to their migration to Azure as far as I know
- chanux 2mo ago
  So they completed the migration? Didn't see that being reported.
  - EffrafaxOfWug 1mo ago
    Probably not. Also they still haven't added ipv6 support
  - dgellow 2mo ago
    I don’t know
Klonoar 2mo ago
Nobody disputes they had issues before, it’s the rate of issues that is even more noticeable now.

rcxdude 2mo ago
>Spare me the "Azure sucks" comments. Please. Microsoft Azure is the second-largest computing system on the planet The fact that Azure is big and important has very little to do with the quality of the implementation (and my experience with the services on top of it suggests that quality is not great).
- greatgib 2mo ago
  Anyone that has to work with azure knows that the bad stability and quality of the platform impact them. Often they just don't have the choice because only choice available on bigco huge framework contract.
altcognito 2mo ago
14x is a lot commits, but I seriously have doubts that some pushes are putting them over the edge. (as many have suggested) > GitHub Actions For me this is probably this biggest culprit along with the general pressure of AI against the hardware. Github actions became really popular. So many of those 14x commits are generating real CPU and network load. This is a really non trivial problem. They can't just wall it off. They have real customers on that stuff.
- martinald 2mo ago
  Why do you think that? GitHub Actions is 'simpler' to scale, it's just a bunch of runners, and Azure has a lot of capacity for VMs (mostly). Git itself is a compute intensive process and is far more interlinked.
  - altcognito 2mo ago
    Not sure how you see git repos as a compute intensive process. A single git commit takes milliseconds. It's mostly a hash calculation and a small file transaction. They've added some database stuff around the edges, but that is pretty tame as well. Actions are unbounded, take minutes to resolve - sometimes hours and there are megabytes, gigabytes of network transactions. Scaling commits is a load balancer and some more instances of a thing you’ve built over and over. Scaling actions is VMs, networking, imbalances in load, memory, insuring that folks aren’t running VM training against your infrastructure. Oh, and maybe you want to make sure folks aren’t trying to hack that infrastructure as well. It is a very, very hard problem.
    - farfatched 2mo ago
      I'm sure GitHub Actions are more intensive, but that doesn't make it harder to scale, which is what martinald said. > Scaling actions is VMs, networking, imbalances in load, memory, insuring that folks aren’t running VM training against your infrastructure. That infra scaling is what the cloud provider can easily offer. Meanwhile, GitHub repos/issues/PRs is all the business logic that they've written, evolved over time. Many coupled distributed DBs, indexes, consistency concerns, etc. That's trickier to scale.
discodave 2mo ago
Lots of good points in there but when you say "swapped out data center" that sounds very un-cloudy and unreliable to me - workloads in AWS are spread across many buildings unless you try really hard or pay $$$$
- VirusNewbie 1mo ago
  yes, it does sound weird to brag about that. 'Swapping out a data center' is something engineers working for AWS or GCP are do practiaclly every week.
cantalopes 2mo ago
I mean i'm sure github employs great engineers, nobody is telling they're nor. But then you look at a graph like this... https://ibb.co/Lffpmfd https://ibb.co/Lffpmfd
VirusNewbie 2mo ago
> Spend a minute or two thinking about the product journey GitHub would have taken under Google, and then think about how many non-search, non-advertising products have succeeded there. Android Youtube Waymo Chrome Workspace GCP >Which amazing developer tools from Google do you use regularly? Kubernetes Jax Tensorflow gRPC Go Istio