6 ms·
GitHub Is Down
- deleted 7y ago[deleted]
- IceWreck 7y agoI was building a github app and was scratching my head about where I messed up. The website and front end were loading but github isn't sending webhooks.
- zackb 7y agoI was deploying code to production as it died!
- bill_mcgonigle 7y agoWell that is just the worst luck. Condolences as you revert locally?
- ahnick 7y agoYeah navigation is slow or timing out on my repos.
- awinter-py 7y agohave never seen the github 500 screen before! the 404 screen is delightful, this one is as well silver linings
- applecrazy 7y agoI'm not even getting the pretty errors :) Just the generic nginx 504 timeout.
- Stratoscope 7y agoI was trying to authorize a third-party client and got an angry pink unicorn.
- ahnick 7y agoSame here. Never seen that before.
- TAForObvReasons 7y agoYou can reliably hit the 504 screen (no real styling) by trying to pull up the blamelog for a large file that has a complex history
- applecrazy 7y agoYep, hitting Nginx 504s when accessing my (and others') repositories.
- badloginagain 7y agoI too went straight to HN upon realizing this
- gpsz 7y agoYep. git push/pull are throwing permission denied errors. I thought I'd check ssh itself. ssh -T git@github.com throws the same permission denied error.
- HugoDias 7y agoThat's great. 2 minutes before it went down a pipeline workflow ran in Heroku deploying a bug to my app. Now I can't deploy the hotfix since it depends on Github and I'm not able to revert to the last successful build on the Heroku interface. Any ideas?
- hakunin 7y agoHow about git push heroku master from local branch?
- HugoDias 7y agoHo! I just forgot about it. Thanks a lot. It's running ¯\_(ツ)_/¯
- corndoge 7y agoRemove spiderweb dependencies from your deployment pipeline?
- thanksforfish 7y agoThis is a good point, but it could be worded better. One of the things a site reliability engineer should think about is how well the site can be operated when dependencies have issue. After an incident like this, even if you were able to recover, it's worth thinking about how things could have gone better. In the past I had a painful experience with one application I was supporting that needed to install NPM packages on deployment. We couldn't successfully deploy (or scale up) for the duration of that outage. In that case we realized it was safer to switch to server images with all assets pre-installed and an NPM cache to give the build a better chance of succeeding. The next NPM outage we only noticed after the fact :) Not certain how this particular deployment pipeline is failing due to the GitHub outage, but a post-mortem to discuss may be helpful and protect against future issues.
- corndoge 7y agoTo your point on NPM dependencies, CDNs for JS libraries is another thing that I recently learned is common practice, which I don't fully understand. It seems like if the CDN goes down then your application stops working, but I am convinced I'm missing something here because it seems like such poor engineering judgement. This seems to be really common in SPAs, which is precisely where (it seems to me) you shouldn't be using a CDN. It seems the convenience of cloud based deployment pipelines is not really worth situations like this.
- Diggsey 7y agoWow, it's totally dead, not even a read-only view.
- newcrobuzon 7y agoLol. Couldn't work on github, I thought it was just a slow connection so checked HN instead to see whether the page loads - and now I guess it wasn't just an issue on my end :D
- gargarplex 7y agoI wrote some open source code to automatically mirror all my commits to Keybase. </shameless plug> I think read replica mirrors are good practice for fault-tolerant systems
- ahnick 7y agolink?
- fezzez 7y agoit's probably on github
- rad_gruchalski 7y agoHopefully does not depend on github webhooks for replication!
- gargarplex 7y agoIt does, but it would've mirrored your latest commit before it went down. Anyway, here's how you connect Keybase to GitHub: https://blog.codefor.cash/2019/08/30/free-automatic-github-backups-with-keybase/ https://blog.codefor.cash/2019/08/30/free-automatic-github-b... https://github.com/codeforcash/github-to-keybase-mirror https://github.com/codeforcash/github-to-keybase-mirror Note: this was a v1, and I'm sure there are more resilient ways to do this. Critical feedback strongly encouraged (ideally along with suggestions).
- rad_gruchalski 7y agoHooks (https://git-scm.com/docs/githooks https://git-scm.com/docs/githooks) together with templates (https://coderwall.com/p/jp7d5q/create-a-global-git-commit-hook https://coderwall.com/p/jp7d5q/create-a-global-git-commit-ho...) sound like a good alternative. Edit: I can see where your solution has an advantage. When one accepts contributions, all of that can be automated with web hooks via github. Still, having to run lambda/similar for this is a bit heavy weight. Edit 2: Considering that for a release, I would opt in for release from a tag or explicit commit, that would be a non issue. Pull when needed, for redundancy, use another, self hosted bare repo. Push there in case of github down.
- ogre_codes 7y agoApparently the earlier thread about Microsoft's cloud stability was a bit prescient.
- amarshall 7y agoHas GitHub moved their infrastructure to Azure? Last I recall they ran mostly on their own servers.
- geerlingguy 7y agoI'd imagine the higher volume due to remote work (e.g. people collaborating more on issues and such who may have communicated in person previously) is also a contributing factor. From the sound of their earlier 'not postmortem but we'll do one soon', it sounded like they currently have a database tier that's creaking at its foundations, and I'd guess any time there's extra strain or someone pushes a fix to the GitHub codebase that adds a tiny bit of unoptimized code, it strains it past the limit.
- ogre_codes 7y agoI'm not sure. My post was mostly in jest. But being at Microsoft clearly hasn't helped.
- maallooc 7y agoIt's OK to rely on Github clones for dependencies, they said. Mind your own codes, they said.
- elwell 7y agoSeems to be back up now.
- sgt 7y agoStill down for me
- jakejarvis 7y agoEven our static assets from raw.githubusercontent.com are throwing 500s. Whatever it is, it sounds like a pretty widespread failure...
- enraged_camel 7y agoYeah, can confirm. I'm working on some visualizations for covid-19 where I pull data from raw.githubusercontent.com and I'm getting 500s.
- ngokevin 7y agoThere are some CDNs in front of rawgit like jsDelivr that may work.
- rvz 7y agoI don't see the point for major open-source projects or even company projects to depend on a third-party provider for git services in the cloud that is not on-premise. It's fine if the source code is self-hosted or on premise and the mirror is hosted on some cloud VCS like Github, but not the other-way round, which for the latter you risk getting problems like this. The case for self-hosting a VCS server for serious projects is made once again.
- mattrick 7y agoI can understand this sentiment, but a lot of projects and companies (especially startups) want something that "just works" and not something that they'll need to invest resources into maintaining. Something like this can happen on your self-hosted git server and you won't have an SLA or anything to fall back on.
- jayd16 7y agoGithub (and gitlab) has a large value add with regard to discoverability, issue tracking, user management, hosting simplicity, and usually solid uptime.
- notacoward 7y agoGitHub addiction is not just about git itself. It's about the entire ecosystem of pull requests, issues, forking, linking to related projects, CI integrations, hosting for docs, and so on, all integrated under one sign-on umbrella. It is awfully convenient. Yes, for sure you can self-host one of the alternatives like GitLab, or assemble an equivalent set of services from smaller parts. It is absolutely reasonable to suggest that people should - I'm even inclined to agree - but comparing against git all by itself is the wrong comparison.
- rvz 7y agoIt is indeed convenient but the whole ecosystem now runs the risk of being a single source of failure if even one part of the service goes down and it is out of your control to do anything about it and also defeats the purpose of decentralisation of using git. It may not matter for personal or toy projects, but for company or serious ones, it does. In the case of other risks of only using GH to host your project, Its like hosting your encrypted private key on Keybase because it is temptingly convenient but both run the risk of going down or getting compromised in a security breach. Serious projects like the Linux kernel, WebKit and Chromium aren't primarily hosted on Github but have Github mirrors instead which makes more sense.
- gramakri 7y agoWe use a self-hosted GitLab ourselves and I can honestly say we have more up time than GitHub. But 9/10 every project we use is on GitHub and our work is stalled. Centralization will be our downfall :(
- thanksforfish 7y agoYou may be able to mirror those to your GitLab and then do all your work from the mirror. https://docs.gitlab.com/ee/user/project/repository/repository_mirroring.html#pulling-from-a-remote-repository-starter https://docs.gitlab.com/ee/user/project/repository/repositor... Congrats on the self-hosted uptime :)
- gramakri 7y agoThe nature of our product is to follow 100s of projects. And most of the interesting stuff is actually not in the code but in the issue tracker, release notes etc (for us).
- Seirdy 7y agoThis is why it's a good idea to have multiple remotes and advertise those remotes in the project README (my examples: [0]). You'll lose centralized issues during downtime, but you'll at least be able to interact with some remotes and do basic collaboration with `git format-patch` and `git send-email`. If something like sourcehut [1] goes down, nothing will change; users will still be able to push/pull to mirrors and work with mailing lists and git's built-in email-based pull-requests. [0]: chattiest-channels, dotfiles, term-dmenu, and mpd-scripts at https://git.sr.ht/~seirdy https://git.sr.ht/~seirdy [1]: https://sourcehut.org/ https://sourcehut.org/
- gadrev 7y agoJust managed to push something, but the web interface is still down. EDIT: (21:00 UTC ) Just got in :D
- jjordan 7y agoWell, I guess I'm done working for the day.
- keeganwitt 7y agoHehe, somebody modified the XKCD "Compiling" comic. https://twitter.com/PonteIneptique/status/783190337602846720 https://twitter.com/PonteIneptique/status/783190337602846720
- spir 7y agoPSA: I just received a github phishing request that was not detected by Gmail. I almost fell for it. Phishing email looks like a "review your suspicious activity" alert, but the alert is the suspicious activity. https://imgur.com/a/zdtWmuN https://imgur.com/a/zdtWmuN
- RMPR 7y agoJust received a repository vulnerability notification as well, in the Pillow dependency but it seems legit.
- meowface 7y agoDid the odd capitalization, "April 01th", and unrelated sending address not tip you off?
- johnghanks 7y agoYeah I gotta agree with you. How is this any different from the other 10s of thousands of variations of phishing attempts? If it's not signed and it's not from github.com or a registered subdomain, or if the URL of the action isn't _explicitly_ github.com... it's not legit. It doesn't matter if it slipped through GMail's filter or not...
- meowface 7y agoI don't mean to shame people for falling for phishing. I fell for a convincing Steam phish once when I was a young teenager after a friend's account was compromised and messaged me with a link to a perfect clone of the login page and a similar domain. But as far as phishes go, this one just seems not too convincing.
- justinhj 7y agoIn the case of the github phishing it does have a valid certificate and the host name is something that looks legit enough it will probably fool a lot of people
- ksec 7y ago
- intsunny 7y agoUTC for status updates?? Not PST/PDT?? Amazing!
- ralphael 7y ago...its back now for me.
- keeganwitt 7y agoI didn't realize until today that https://www.npmjs.com/package/geckodriver https://www.npmjs.com/package/geckodriver downloads from Github.
- cmckn 7y agoTheir status page is quite pretty, does anyone know of an open-source option that looks similar?
- EvanAnderson 7y agoIf only there was some kind of distributed version control system. /s
- thanksforfish 7y agoOf course, there is. You just need to host it. Self-hosted GitLab mirrors work great when GitHub is down. https://docs.gitlab.com/ee/user/project/repository/repository_mirroring.html#pulling-from-a-remote-repository-starter https://docs.gitlab.com/ee/user/project/repository/repositor...
- juancampa 7y agoI think he meant git itself
- jakejarvis 7y agoOr Gitea, if you want to use a fraction of the resources! I use it solely as a GitHub backup — you can mirror Git repositories from anywhere and it pulls changes on a schedule. Literally set it up this week, thank goodness. https://gitea.io/en-us/ https://gitea.io/en-us/
- Seirdy 7y agoInstead of relying on GitLab's API, it's probably a better idea to just use git remotes properly. I put together a script [0] to automate the process to set the primary remote to Sourcehut (git.sr.ht) and mirror to GitLab and GitHub. And yes, that script is in a repo that's also mirrored to GitHub and GitLab (check the README). It combines well with my `git pushall` alias [1]. [0]: https://git.sr.ht/~seirdy/dotfiles/tree/master/Executables/shell-scripts/bin/git-remote-setup https://git.sr.ht/~seirdy/dotfiles/tree/master/Executables/s... [1]: https://git.sr.ht/~seirdy/dotfiles/tree/master/.config/git/config#L41 https://git.sr.ht/~seirdy/dotfiles/tree/master/.config/git/c...
- gumby 7y agoYou can just sync with someone else’s repo by specifying it on the command line. Nothing to install, no script, nothing. It’s built into git and is right there in the single line summary of the man pages for git-fetch, git-pull etc: they all take a repository argument.
- andrekorol 7y agoOkay... so, when is Amazon launching their GitHub alternative?
- ljoshua 7y agoOh, they have CodeCommit... good luck with that one though!
- miguelmota 7y agoThere's the AWS CodeCommit service but it's for private repositories only https://aws.amazon.com/codecommit/ https://aws.amazon.com/codecommit/
- Ophuscado 7y agoIt's working again.
- stared 7y agoToday I have a deadline. It heats up the emotions. Paulo Coelho: “And, when you want something, all the universe conspires in helping you to achieve it.” Murphy: [GitHub is down] https://twitter.com/QuantumGameIO/status/1245818849246748674 https://twitter.com/QuantumGameIO/status/1245818849246748674
- RMPR 7y agoWorking now
- ChrisArchitect 7y agoahh, angry pink unicorns. Reminds me of the old Twitter failwhale
- axegon_ 7y agoDidn't check the status page but I figured once I had my before bedtime peak at the trending repos.
- ur-whale 7y agoGitHub or how to re-centralize what was properly decentralized. Sad.
- gumby 7y agoI first thought I was having network saturation problems due to my gf’s kids playing games instead of doing their school online. I then checked some other sites and found they were fine but my first assumption was not that GitHub itself was having problems.
- reilly3000 7y agoThe BSOD for a new generation...