8 ms·
GitHub having issues today
- hunkins 3y agoGetting intermittent 500s browsing repositories right now. Hugs to the GitHub ops team.
- efrecon 3y agoAlso actions won't be able to checkout.
- cupofjoakim 3y agoHappy I'm not in a hurry with any specific work items today. Hope it's not too much of a mess to figure out for the github peeps. Much love to them.
- Narciss 3y agoRun for your liveeeees!
- robinhoodexe 3y agoAnd just as we're about to migrate 4 kubernetes clusters with a total of ~4k pods. Terraform in github actions on selfhosted runners and argoCD is failing.
- mrweasel 3y agoOh that sucks, there's always going to be those who will say that it's the price you pay for using Github, but locally hosted VCS and CI/CD systems have issues as well. External dependencies are always problem, but do you have the capacity and resources required to manage those dependencies internally? Most don't and will still get a better product/service by using an external service.
- marginalia_nu 3y agoRate of outages on github last few years has been orders of magnitude higher than anything I've encountered on a locally hosted VCS. Local also means you can orchestrate maintenance windows to avoid outages at critical phases.
- darkwater 3y ago> Most don't Define "most". There is a surprisingly high number of small/mid-sized companies which have dedicated people for this kind of things.
- mrweasel 3y agoPerhaps "many" would have been better. If you only count companies that view themselves as IT companies then number of companies with self hosted/managed solution grows, but if you include everything then I'd guess that more than 50% don't run these services internally. If you're every small company with one or two developers that doubles as the IT staff then the numbers add up pretty quickly.
- snarkyturtle 3y agoThat's where I feel like it's actually pretty nice to not have CI tied to your source code. It's probably more expensive to use Travis/Circle but at least you don't have a single point of failure for deploys.
- sleepychu 3y agoDoesn't this give you 2xSPF? Or can I use my local copy of the source to kick of Travis/Circle?
- jpgvm 3y agoIdeally you don't ever rely on CI specific automation tooling to actually accomplish anything and instead just use it as a dumb "not my dev machine" to execute workflows. You should always engineer things so you can fall back to something akin to: ./scripts/deploy_the_things Ideally backed by a real build system and task engine ala Bazel, Gradle, whatever else floats your boat. It also means you are free to move between different runners/CI providers and just have to work out how to re-plumb secrets/whatever. GH actions/friends really provide minimal value, the fact they have convinced everyone that encoding task graphs into franken-YAML-bash is somehow good is one of the more egregious lies sold to the development community at large.
- snarkyturtle 3y agoWell, git ops might not be impacted on Github, usual Github outages tend to be through Actions/the site, not the actual git operations. Doesn't seem like you can use a local copy but you can use Gitbucket/Gitlab.
- acedTrex 3y agoThe fact that Github has been so unstable for so long is absolutely insane to me. I know ops is hard, but this level of consistent outage points to an endemic problem. Is it the legacy rails/mysql stack that is the largest culprit or is there systemic rot in the engineering org?
- indymike 3y ago> Is it the legacy rails/mysql stack that is the largest culprit or is there systemic rot in the engineering org? The culprit is change. Infra changes, config changes, new features, system state (os updates, building new images, rebooting, etc...), even fixing existing bugs all are larger changes to the system than most think. It's really remarkable at this point that Github is as stable as it is. It is a testament to the Github team they have been as stable as they are. It's not "rot" it's just a huge system.
- izietto 3y agoWhy people think it can be related to Rails when there are tons of companies out there using Rails not affected by this degradation?
- dtech 3y agoOne difference is that Rails and MySQL on a Github scale is rare, even when taking into account Github scale is rare.
- izietto 3y agoyou mean that most of the other popular huge Rails companies (GitLab, Shopify) use PostgreSQL? Basecamp uses MySQL tho
- berkes 3y agoGitlab is, AFAIK, nowhere near the scale of Github. Shopify IDK, but I'm fairly certain their type of usage is very different. Basecamp is also of another scale, and certainly a very different usage/performance type.
- rvz 3y agoStill having regular incidents at GitHub in 2024, even with Microsoft's infrastructure after 5 years since the acquisition with something always going down. Just expect GitHub to go down at least once every month as it is that unreliable. This certainly has aged well: [0] [0] https://news.ycombinator.com/item?id=22868406 https://news.ycombinator.com/item?id=22868406
- dartos 3y agoThe price teams will pay to offload their ops. People really like avoiding ops
- deleted 3y ago[deleted]
- lillecarl 3y agocopilot is purring like a cat, best wishes to their infra team!
- abhinavk 3y agoIt really feels like the frequency of this happening has increased lately. Are they facing high employee turnover? _Maybe it’s time for rewriting it in Rust._ Edit: RIIR was said in jest. I forgot HN doesn’t support markdown.
- sumtechguy 3y agoFull stack re-writes are not always the best way. Sometimes you end up with worse. Sometimes you end up with better. If you do go the 'full stack rewrite' you better have a decent plan in place. Because you are about to get to support 2 code bases for awhile. edit: fair enough
- vitiral 3y agoI'm pretty sure (hope?) the comment was said _in jest_ (though I'm not familiar with _this internet standard_)
- abhinavk 3y agoYeah. I thought markdown would work here.
- hiccuphippo 3y agoItalics work using asterisks.
- chuckadams 3y agospongecase works wonderfully for sarcasm: mAYBe IT’S TimE fOr rewrItING it iN rUsT
- djtango 3y agohttps://news.ycombinator.com/formatdoc https://news.ycombinator.com/formatdoc
- bombcar 3y agoWhoops guess that was one too many PRs. Sorry everyone!
- ftkftk 3y agoSomebody is having a bad Tuesday.
- jdthedisciple 3y agoGitHub having issues? Sounds like all in good order then ...
- amelius 3y agoIn the old days we had mirrors for many online repositories.
- sph 3y agoWe've lost the technology for decentralised Internet services in the early 2000s. We just hope SMTP keeps ticking along somehow or we're fcuked.
- berkes 3y agoTeams, Slack, Discord, whatsapp, imessage, etc are trying hard, to replace that too, though,
- djbusby 3y agoYep. Now I'm glad I vendor-mirror all my dependencies so I can keep doing tests and all that.
- 8organicbits 3y agoI wonder if projects should proactively mirror to another git site (github as main and gitlab as mirror, for example). Collaboration on the project may stop when the main is down, but consumers could proceed using the mirror. I'm not sure how well various tooling supports fallback origins. This would reduce (but not eliminate) the need for users to vendor their dependencies.
- richardwhiuk 3y agoEveryone with a git clone has a mirror....
- amelius 3y agoBut how do I find someone with a clone of the repo I'm looking for?
- 3y ago
- aaomidi 3y agoWe have a slack channel that monitors GitHub availability. There’s content there nearly 3-4 times a week. It’s amazing how awful this has become.
- alexnewman 3y agoPhew, I thought it was because I am in Puerto Rico. Has GitHub or Microsoft done any bigelayoffs?
- imdsm 3y agoTime to sword fight outside the offi... oh we all work remotely now.
- inhumantsar 3y agowell, at least that's one good reason to buy into zuck's metaverse
- pklack 3y agoTime to do the laundry and dishes then, I guess... Sometimes WFH has its boring sides.
- chaxor 3y agoWouldn't it be wonderful if the most popular version control system was is decentralized? This is achievable, and is the correct solution. This way your git repo could be located on: - GitHub - Your Closet (...) - UCLA's supercomputer - JBOD in Max Planck Institute (...) - GitLab Doing this with a simple file with "[ipfs, github, gitlab]" on it would be revolutionary, especially for data version control, like nn weights or databases that are too large for git and cost too much on other services, as they would be free on ipf/torrent. Then no one is phased by the inevitable failure of various companies.
- Bognar 3y agoAh yes, because version control is only about file storage.
- blauditore 3y agoCan't tell if this comment is sarcastic, but that's exactly what git is: Every clone of the repo is independent, and acts as a full backup. Likewise, a local repo can be pushed to various remotes, there is no inherent strong server-client coupling (even though it's often used in such a way).
- chaxor 3y agoYes but not having to specify every single user who pulls the git repo from GitHub by IP address as an additional remote is a huge win no? If "ipfs" can be added as a remote, and it automatically pulls from thousands of different devices without having to specify them, that's a pretty big win for redundancy right?
- blauditore 3y agoNot sure what's the big benefit here, backups? Otherwise, the more devs a repo has, the more redundancy exists anyway, if that's your main concern.
- chaxor 3y ago
- iddan 3y agoIt is back