9 ms·
GitHub was having issues
- meepmorp 1y agoActually, the big problem for me is that github doesn't have issues at the moment.
- legobmw99 1y agoDay one of the new management structure off to a rocky start
- xyst 1y agoalways been "rocky" since the acquisition. This is just perfect comedic timing
- adhamsalama 1y ago[flagged]
- foobear9867 1y agoAI probably caused these issues.
- arp242 1y agoStop posting this. You've been told off by a moderator at least once already for literally exactly this: https://news.ycombinator.com/item?id=44640416 https://news.ycombinator.com/item?id=44640416
- acedTrex 1y agoDay ends in Y
- Thev00d00 1y agoVibe coded a bit too hard.
- belter 1y agoDid they upgrade to GitHub 11 ?
- CoastalCoder 1y agoThanks to internal politics, that only possible if they have a Technical Product Manager II (TPM).
- yabones 1y agoThe only thing worse than "it's down" is hearing "it's down again" Godspeed, IR workers...
- jeltz 1y agoLast time it was down was Friday last week when they broke some of the raw content URLs.
- rs_rs_rs_rs_rs 1y agoJust Github? I'm having problems logging in into Dropbox as their captcha setup fails.
- halfmatthalfcat 1y agoGithub reliability has been abysmal the past couple of weeks.
- thrdbndndn 1y agoCoincidentally reddit is also down.
- mock-possum 1y agoWell at least there’s a silver lining then
- amelius 1y agoWhen a cool job becomes boring IT work.
- zanfr 1y agowell I wish nothing but the best for the great people at Microsoft, this small family business.
- bootsmann 1y agoI am surprised at how much github is willing to let themselves go for being a product that can nowadays be replaced by someone over the weekend. This is the second major outage within like 2 weeks.
- taude 1y agoDepends how much customization you've done to your cicd and how heavily you use custom GHA, and other plugins. No way my org could move off in a weekend.
- tracker1 1y agoAgreed... you could probably move a single project over pretty easily on a weekend, or several depending on how integrated you are.
- reaperducer 1y agofor being a product that can nowadays be replaced by someone over the weekend. If one person can move a company move off of GitHub in a weekend, you're too small a company for Microsoft to care about.
- bootsmann 1y agoIt was less about moving your company away and more about the core functionality being extremely simple to reproduce.
- nimbius 1y agorecent developments in the C levels not withstanding, github has been a dumpster fire for quite some time now with no sign of improvement. https://www.githubstatus.com/history https://www.githubstatus.com/history
- jeltz 1y agoThose are not even all. They had an incident with raw content on the 8th which is not listed.
- metalliqaz 1y agoDear sages of HN, what is the scrappy new upstart project that will rise up to take Github's place? Because this is becoming intolerable.
- john01dav 1y agoI think that the answer here is eventual widespread use of many smaller options, from FSF's Savannah to self-hosted gitea/forejo/gitlab. Git is in a good position to support this.
- cAtte_ 1y agonot gonna happen. society loves monopolies because they're convenient
- john01dav 1y agoWe're starting to see the pain of such monopolies. Note that I included a hosted option (Savannah) in my list. It doesn't take everyone leaving github to break the monopoly, just enough to make it not a monopoly.
- meepmorp 1y agoIt's kind of weird that we've collectively decided on a distributed version control system, while centralizing where we keep the repositories and metadata.
- ryandv 1y agoThis is because the dream of true P2P decentralization died with NAT and IPv4 exhaustion.
- metalliqaz 1y agoI think it's just a simple reality that most projects don't actually need or want a decentralized development process. In my experience, most projects are looking for a single, high-reliability canonical source that is in control of project leadership. Most projects are only developed by a small group, maybe even only one person.
- evilmonkey19 1y agoI'm a fan of github but lately i'm seeing a lot of issues like these... Also they don't have yet support for IPv6 (surprising).
- awestroke 1y agoThe refusal to support IPv6 is embarrassing at this point
- geoffeg 1y agoI thought a recent downtime was contributed to rolling out the initial prep for IPv6 support.
- kevingadd 1y agoHad to buy an IPv4 address for a VPS the other day in order to clone some git repositories. Couldn't believe it. Costing their customers money when they should be able to support v6 by now.
- hk1337 1y agoWhat VPS are you using that doesn't come with both IPv4 and IPv6?
- jraph 1y agoThere are options where you pay 1€/IPv4/month and IPv6s are free.
- miyuru 1y agoThere are plenty of low end providers that support IPv6 only. At that scale price of IPv4 is the highest cost of the VPS. Here is a list of providers I created back in 2022. https://blog.miyuru.lk/ipv6-hosting-2022/ https://blog.miyuru.lk/ipv6-hosting-2022/
- e3bc54b2 1y agoHetzner charges extra for IPv4 address, as I believe most of them do. I know because I went through the same crap.
- rvz 1y agoNo CEO of GitHub to contact about this outage this time.
- numbsafari 1y agoContact Satya Nadella.
- siva7 1y agoThe guy who framed Github as a stellar example of how Microsoft will let operate its aquisitions independently?
- numbsafari 1y agoHe is (and was) literally the CEO. Yeah, he can give whatever title he wants to his subordinates, but the "CEO" of GitHub has been a mid-level VP for quite some time.
- dustypotato 1y agoWhy does it show me a cookie banner from Atlassian?
- alexjplant 1y ago...because they use Statuspage, an Atlassian product?
- j45 1y agoThe cloud is someone else’s computer. I wonder if self hosting is more reliable. How much does a private and firewalled git* instance need updates?
- taude 1y agoBefore gitlabs, github, etc. it was common to host your own code repos on-prem. The thing now, though, is that there's a lot of add-on functionality for how teams flow from using github over just hosting a git server.... so it's not really apples v oranges, anymore.
- john01dav 1y agoYou can use a self hosted forge, like gitea, forejo, or gitlab.
- marcosdumay 1y ago> The thing now, though, is that there's a lot of add-on functionality I don't get why people use those. I understand free software projects that don't want to run any infrastructure, but why companies push their building and deployment out of premises when all you need is one trusted computer somewhere. Why do people insist on trusting cloud computers more than the ones they can kick?
- j45 1y agoGitlab can handle a lot of CI/CD hosted locally for free that github actually charges for. Git was originally local only too. People would run their own source code repos, it was trivial to run and maintain for the most part for most basic to intermediate use cases. I had clients who insisted source code (mine or theirs) couldn't be on a public cloud provider. It's not that unreasonable or uncommon.
- john01dav 1y agoI have gitea running in my basement. You can see how often it updates on the gitea github (since they don't self host since they haven't written an importer yet). You could automate this, but I haven't spent enough time updating to make that make sense.
- ryandv 1y agoFunny, because just yesterday I was downvoted for pointing out (among other things) GitHub's less than stellar reliability these past few years. When are the AI vibe coders going to create a GitHub replacement? With 1000x AI productivity a lean startup should easily outcompete the incumbents, no?
- numbsafari 1y agoYeah, especially if you deploy it on Azure...
- gmerc 1y agoIs this a good time to mention forgejo as a self hosted alternative?
- numbsafari 1y agoIt's a great time.
- phkahler 1y agoYou got my attention, so why didn't you provide a link?
- sigio 1y agohttps://forgejo.org/ https://forgejo.org/ or https://codeberg.org/ https://codeberg.org/
- deleted 1y ago[deleted]
- jeltz 1y agoHow does it compare to Gitlab?
- dzogchen 1y agoMuch simpler. Much less features. Completely open source and not only the core.
- jeltz 1y agoAny important feature from Gitlab you feel is missing? I personally think Gitlab has way more features than I need but maybe there are some important ones I would miss.
- makkesk8 1y agoRunning Gitlab in any kind of scale beyond a single server is a major PITA. And it's very poorly optimized.
- everfrustrated 1y agoEnterprise customers, remember to email your sales rep and ask for them to report on their contracted uptime with you that you are allowed to do as per contract. They wont do this unless you ask hoping you don't notice the outages. It creates lots of internal pain - they have no automation internally for reporting on this. This is the only way anything will ever change. GitHub is _easily_ the most unreliable SaaS product. There's not a week whereby we aren't affected by an outage. Their reputation is mud.
- sieabahlpark 1y ago[dead]
- petcat 1y ago> GitHub is _easily_ the most unreliable SaaS product. Some of us are stuck using Atlassian and BitBucket and it is by far worse in every way.
- __natty__ 1y agoAgree, atlassian products are much worse in terms of reliability.
- enraged_camel 1y agoWe use Jira. It's a horrendous product, but I cannot remember the last time it was down, unlike Github.
- ta1243 1y agoI use self-hosted jira, it's a great product, but I have full control over my teams tasks and workflows and as a tiny team we make them work for us (subject, description, comments, occasional linking to other tickets, assigned to, and status of "open", "blocked" or "done") Most of the problems I hear about are micromanaging product managers. That's not the fault of the tool itself per-se.
- 1y ago
- ab71e5 1y agoAh was wondering why pull requests did not show up
- deleted 1y ago[deleted]
- gethly 1y agoI've always used Bitbucket as it allows private repositories, so Github was never something I wanted to use. But it is one of the most important websites in the world for tech people and should be run better than this, especially being now owned by the largest tech company in the world. On the other hand, it just shows that centralisation, or over-reliance on one thing or service, is always the worst idea. But people are very lazy and so we keep running in these circles ad infinitum.
- clintonb 1y agoDo you mean free private repos? GitHub has allowed free private repos for a few years now.
- gethly 1y agoI have been using bitbucket before it got acquired by atlassian, which was in 2010, so i am sure github changed quite a bit since then. Even though i do have a github account, i don't use it for anything else other than creating or commenting on issues for other repos. So i have no clue about present day github capabilities as i am satisfied with bitbucket and had no need to explore github in depth.
- masklinn 1y agoGithub switched from unlimited users and per-repo fees to unlimited private repositories and paying per-seat 10 years ago or so: https://github.blog/news-insights/product-news/introducing-unlimited-private-repositories/ https://github.blog/news-insights/product-news/introducing-u...
- mattbruv 1y agoMan, I remember feeling like that change was somewhat recent. 10 years ago was about 2016. I'm not ready to accept this.
- notnmeyer 1y agoanybody have a GH alternative they like? bonus points awarded for not being a straight clone of GH.
- astrolx 1y agoforgejo [1] works well for me. It can be self-hosted, otherwise e.g Codeberg [2] runs a good instance [2] [1] https://forgejo.org/ https://forgejo.org/ [2] https://codeberg.org/ https://codeberg.org/
- nerdypepper 1y agohttps://tangled.sh https://tangled.sh? social features via atprotocol, native jujutsu support, stacked PRs, interdiffs, light weight to selfhost, nix-based CI.
- notnmeyer 1y agoamazing
- Aldipower 1y agoGitlab of course. It's mature.
- havkom 1y agoDoes this matter much? Git is a ”distributed source control system”. Are you using git in the wrong way? :)
- tracker1 1y agoWell, considering git is only version control, and github does much more... pull requests, social interaction, workers/workflows, ci/cd etc. It's kind of a big deal. Unless you have some sort of decentralized method of doing CI/CD and pull requests I'm unaware of?
- ngoldbaum 1y agoI wonder why the github status page has an atlassian cookie request pop-up.
- mbb70 1y agoThe status page is a SaaS product called StatusPage, acquired by Atlassian in 2016.
- taude 1y agogit was designed as a distributed vcs for high-latency connected developers with plenty of ability to work offline. I don't think I've really been impacted by any of the outages. Maybe I wait an extra hour to merge a feature or something, in which case I actually get to eat lunch and browse HN, doesn't feel quite as catastrophic for me, as some of you.
- petcat 1y agoThe problem is that people design their entire development and release lifecyle to be dependent on Github. A lot of times they can't even push code hotfixes to production without it. It's a terrible SPOF for a lot of engineering orgs.
- taude 1y agoThe hotfixes makes sense.
- lucb1e 1y agoWe also started having customers since a few years that declare GitHub fully trusted, as in, it is simply not worth considering what the impact would be if that vendor gets compromised. I can't name names, but this includes a vendor that aims to prevent supply chain attacks (technically language-agnostic; in practice aiming to be the solution chosen by one of the biggest programming language's package manager) > can't even push code hotfixes to production without it. It's a terrible SPOF GitHub's availability impact is the least of my concerns these days. It'll be a really tough year for society worldwide if we need to rebuild loads of infrastructure after some threat actor got into github and managed to change key pieces of code without being detected a couple of years. Having seen how hospitals handle updates, they might get lucky and be old enough to not be affected yet, or have a really tough time recovering due to understaffed IT No clue how to even begin solving this since our OSes are likely all pulling dependencies from GitHub without verification of the developer's PGP key, if the project even has that and applies it correctly. I guess I can only recommend being aware of the problem and doing what you can in your own organization to reduce the impact
- dymk 1y ago
- adobrawy 1y agoIn meantime, I recommend to read their availability reports on their blog, published monthly: https://github.blog/news-insights/company-news/ https://github.blog/news-insights/company-news/
- bob1029 1y agoI've been working on a self-hosted alternative to GitHub and I am curious what HN finds to be the most important features. I think Code, Issues and PRs are the critical aspects, but I don't know what typical workflows look like for others these days. It seems like there are some teams that have figured out a way to turn GH into a labyrinth of CI/CD actions that allegedly produces a ton of business value, but I don't know how typical this kind of use case is. If the average GH user just needs the essentials, I could easily focus those verticals down and spend more time on things like SAML/OIDC, reporting APIs, performance, etc. I suspect there aren't a whole lot of developers who are finding things like first party AI integration to be indispensable.
- justinrubek 1y agoI don't think it's the features that are lacking from the alternatives. It's the network effects.
- iN7h33nD 1y agoHave you seen: https://forgejo.org/ https://forgejo.org/
- neilv 1y agoI like what I've used of Forgejo (Git, Issues+Board, Wiki), and have hosted it on servers and localhost easily. I haven't tried its CI features yet. Codeberg is a cloud site for open source projects that runs Forgejo. Forgejo is a fork of Gitea, which is another option, especially if you want commercial support, but I haven't tried it yet. I also kinda like GitLab, both the cloud one and the enterprise on-prem version. And their issue label features work more easily with the board than Forgejo's (automatically moving issues between columns based on scoped labels). Though their pricing tiers have been unfortunate at times (I don't know latest).
- gazook89 1y agoAny time I see this topic brought up, two things are always mentioned: the "hub" part, meaning the discoverability and social aspect, and the "network effect" of having everyone use a single service (so everyone already has an account and they don't have to create additional account for every self-hosted project.
- sashank_1509 1y agoI’m at Google, we have a million + file codebase. Every piece of code is snapshotted (no need to commit, automatic snapshot whenever a change is made). Every line of code has its own unique URL (this is for your branch too, not just overall). Background tests running nonstop hourly. Never seen any downtime. The infra at Google is insane
- chaz6 1y agoWithout manual commits, is there any way to record the intention of each change?
- sashank_1509 1y agoYes you can still commit as a logical delineation of changes. And every commit is automatically a PR, folding multiple commits is possible but not the common workflow, they prefer just reviewing and approving each commit.
- Balinares 1y agoThere are manual commits, they're just called changelists.
- sequin 1y agoGitHub is a completely different scale. If Google hadn't wasted its opportunities to become the prime source code platform they'd also have issues.
- sashank_1509 1y agoThat’s what I disagree with, I think Google codebase scale is massive. It has android + chrome + Google + Waymo + tensorflow etc, this is a lot of code. And also consider that every third party library is put inside Google3 codebase too, there are no truly external dependencies, all the code is there in one repo, with their version one version principle. I think Google is roughly similar scale to GitHub and is managing things far better
- CoderJoshDK 1y agoWith continued reliability issues and the CEO stepping down, now feels like such an opportune time for a competitor to start taking market share. I sure am rooting for it! For the longest time, I thought that there was absolutely no way for some of these corner stone companies (slash tech) to be toppled. And I’m very impressed with their ability to destroy consumer trust!
- colechristensen 1y agoSadly gitlab lacked focus especially on software quality.
- shrinks99 1y agoBetween Tangled, GitLab, Codeburg (Forgejo), and Gitea there's quite a lot of decent alternatives now compared to when GitHub first sold to MS. Having the entire world of FOSS integrated in one development platform was convenient but I'm more excited by the possibilities for more innovation in the space.
- Lammy 1y agoAnd Phorge! (former Phabricator, and my favorite) https://we.phorge.it/ https://we.phorge.it/
- alphazard 1y agoEveryone impacted by this outage should take a look at Radicle, which was also on the front page today. https://news.ycombinator.com/item?id=44874945 https://news.ycombinator.com/item?id=44874945 I'm not affiliated with Radicle in any way. I am just impressed with the technology.
- joshred 1y agoSooo.... What's the consensus on gitlab?
- tracker1 1y agoI've used their public site for a few private projects, mostly in habit from when private projects in GH were limited to paid accounts. The collaboration was a bit better at that time imo. I'm not sure that I would choose it for self-hosting over gitea, forgejo or straight up ssh+git on a remote system, which works well enought for a personal backup target.
- mattbruv 1y agoThis is just embarrassing for them, especially whoever is in charge of pull requests.
- sigio 1y agoJust completed one or our (quarterly) github exports before this hit... If people are looking into extracting everything from their organisations, i've published the scripts I use: https://github.com/sigio/github-export https://github.com/sigio/github-export
- irthomasthomas 1y agoAny one else smile for a moment when they saw their issues list empty? Don't rush to fix it ;)
- gdcbe 1y agoMaybe they should focus less on "agentic" and more on just keeping their core product solid... I suppose that doesn't rhyme with growth at all cost... zzz sad... it is
- asim 1y agoToo many unnecessary features at this point perhaps. Self hosting might be a good alternative. Don't need the UI.
- JyB 1y agoI’m really wondering what internally causes this. No one likes it when they have outages, but it keeps happening. Is this a culture thing? Like pressure to ship features fast? Some under staffed teams or lack of ownership on some crucial components?
- tracker1 1y agoDefinitely all good questions. I've noticed that there seems to be a bit of convergence between Azure DevOps and Github (Enterprise) and am curious how co-mingled the teams or management are at any given level or not. I'm mixed on the cultural changes at MS and have historically preferred GH's approach. I'm hoping MS moves closer to GH than the reverse. I'm not working with/at either company and don't really have a lot of insights to offer other than observations from the outside.
- Seb-C 1y agoGitHub is implemented in Ruby (on Rails), and now there is apparently a strong internal push from Microsoft to code everything using AIs/LLMs. IMO mixing a dynamically typed language, a framework based on magic conventions and vibe coding sounds like the perfect recipe for a disaster.
- jewel 1y agoI used to work at a place that had a second copy of all the git repositories on a server, available over ssh. We could push there and then, at deploy time, instruct the servers to pull from that repo instead of bitbucket. If I were to set up the same thing again today, I'd add some automation to automatically keep it in sync with github as well as automate the servers so that they'd attempt to pull from both on every deploy. I share this as an idea for those who need an emergency or backup deploy mechanism.
- yoyohello13 1y agoJust saying. Our company has been on self-hosted Gitlab for years. We have one devops guy who spends like an hour a month managing the server. Never one outage.
- jonator 1y agoIf you're reading this there's a great opportunity to pull a Linear move and disrupt the entrenched players with a 10x better UX. Although the hardest nut to crack here are the network effects.
- demarq 1y agoWhat’s the hardest part of making a git hosting service?
- Culonavirus 1y agoThere's a book about this. https://www.goodreads.com/book/show/9211847-how-hard-can-it-be https://www.goodreads.com/book/show/9211847-how-hard-can-it-...
- Aldipower 1y agoUsed and using Gitlab with success in all of my companies. It is maybe a bit boring and a little bit slow, but it is enterprise and has everything you need. Included time on the market..
- e-dant 1y agoI like github
- WhyNotHugo 1y agoThe linked page doesn’t say much. Just “increased latency”. No more details than that. Am I missing something?
- alfiedotwtf 1y agoDidn’t GitHub fire a bunch of developers recently because AI was a silver bullet?
- dakoosha 1y ago[dead]