13 ms·
GitHub is down
- jakejarvis 7y agoI feel like this whole year has served as one big reminder of how fragile the internet really is...
- pdelgallego 7y ago2017 S3 outage was a big reminder as well https://aws.amazon.com/message/41926/ https://aws.amazon.com/message/41926/
- joshuaturner 7y agoThe great AWS crash of 2017 convinced me to distribute a bit more, thankfully
- ilaksh 7y agoIt's only fragile when you centralize all of the information in giant websites. It didn't use to be like that. And now there are a ton of decentralization technologies that we should be taking advantage of, but almost everyone ignores them. For example https://medium.com/@alexberegszaszi/mango-git-completely-decentralised-7aef8bcbcfe6 https://medium.com/@alexberegszaszi/mango-git-completely-dec...
- jakejarvis 7y agoAbsolutely. I'm (cautiously) optimistic that some good can come from all this downtime in that sense.
- noir_lord 7y agoIt certainly helps make the argument not to move some stuff to the cloud, there is still a case for on premise as always the answer is "it depends".
- pier25 7y agoWell, the internet didn't stop working, so it's no so fragile after all. You probably mean centralized internet based services.
- langitbiru 7y agoYup, I can not push my commit.
- deleted 7y ago[deleted]
- 0x006A 7y agoDid Microsoft switch GitHub to AGI as part of the OpenAI deal?
- omarhaneef 7y agoQ: Can it automatically reject bad code or would that be impossible because of the halting problem?
- EamonnMR 7y agoit could automatically reject bad-looking code perhaps
- hysan 7y agoJust came here to post this. I was reading through the css-modules docs[1] and am now stuck. I can't see anything other than the main README. Anyone know of a mirror? [1]: https://github.com/css-modules/css-modules https://github.com/css-modules/css-modules
- sarathyweb 7y agoHere you go: https://web.archive.org/web/20190701222409/https://github.com/css-modules/css-modules https://web.archive.org/web/20190701222409/https://github.co...
- deleted 7y ago[deleted]
- primitivesuave 7y agoBrings into perspective how essential git is for my workflow - I am waiting for `scp` to transfer my files from my laptop to work computer and can't push anything into the CI/CD pipeline.
- Twisol 7y agoFor what it's worth, the "D" in "Distributed Version Control System" is useful here. You can `git init --bare foo` on your work computer, `git remote add workcomp username@hostname:path/to/foo` on your laptop, and `git push workcomp master` to push everything over using pure Git. (And the first steps only have to happen once for these two machines.) (This creates a bare repo on your work computer, meaning there's no associated working directory -- you'd probably want to add that same repo as a remote from whichever existing repository on your work computer you have. The bare repo, in this scenario, is just a means of passing commits from your laptop to your work computer in a Git semantically-meaningful way.)
- huffmsa 7y agoYour comment is probably going to inspire a lot of people to start self hosting remote repos during the downtime today. As with most things, it'll start great, but a lot of those people will be in tears in a few weeks. Great power; great responsibility.
- Twisol 7y agoTrue! It's best to consider that kind of repo as a downstream fork, with all the responsibilities of upstreaming changes once the canonical repository comes back up.
- fragmede 7y agoHaving a bare repo is certainly one option. Some may find having two copies of a repo on one system to confusing. Here's an alternate setup that doesn't use a bare repo. It does require some git hygiene/discipline though. Setup: Desktop: git config --local receive.denyCurrentBranch updateInstead This will let the laptop push the desktop, updating its files, as long as the desktop doesn't have uncommited things (aka working dir is clean). On the laptop: git remote add desktop... Working with it: Desktop: commit everything Laptop: commit everything git pull --rebase desktop git push desktop (assuming there are no issues w/ the pull). Not saying either workflow is better, merely providing an alternative. imo the reason git took over has some to do with being unopinionated about workflow - there's some tooling, but whatever workflow is managable with that tooling is "supported" - as long as the team can agree to use said workflow.
- cmarschner 7y agoJust when it was time for beergarden. Thank you!
- jbverschoor 7y agogit is distributed, so no worries
- samcday 7y agoI'm just wrapping up the work to migrate my company away from Gitlab to Github and this happens. I did it because I figured Github has to have better reliability / uptime than Gitlab. Someone joked that as soon as the migration is done Github will have some major downtime. sigh
- deleted 7y ago[deleted]
- benceno 7y agoYou can't spell GitLab and GitHub, so you deserve it.
- celticninja 7y agoIf we are being pedantic they were both spelled correctly.
- shhsshs 7y agoHN is not typically receptive to snide corrections/remarks. There are several more productive ways to address a minor capitalization error - the best of which is to not correct it at all because it's so trivial and unimportant.
- hombre_fatal 7y agoBesides, why would I care if how I capitalize a company's name isn't perfectly in line with their marketing style guide? Github is lucky if I even feel like capitalizing the first letter. What a weird thing to hold over others.
- benceno 7y agoOK I AM SORRY
- q3k 7y agoI highly recommend running at least a local, self-hosted git mirror at any tech company, just in these cases. Gitolite + cgit are extremely low maintenance, especially if you host them next to your other production services. Not to mention, if you get the self-hosted route you can use Gerrit, which is still miles better for code review than GitHub, Gitlab, bitbucket and co.
- deleted 7y ago[deleted]
- deleted 7y ago[deleted]
- corford 7y agoKinda irritating when a binary you need is only hosted on github (...together with the files to build from source).
- rvz 7y agoEven worse if libraries / packages are only hosted on Github. Then the main ecosystem of a language dependent on those packages can grind to a full stop from there. I'm looking at all of you CocoaPods, SwiftPM, npm, Cargo, vgo.
- cameronbrown 7y agoInsanity. Having an emergency mirror on standby is not hard when it's critical infrastructure. Surely npm/Cargo/etc.. could do this.
- knd775 7y agoI think I'll set this up now https://github.com/local-npm/local-npm https://github.com/local-npm/local-npm
- michaelper22 7y agoAdoptOpenJDK builds are hosted there too; found out about the outage trying to download one.
- otterlicious 7y agoThe npm registry itself does not depend on GitHub. It's true that most developers rely on GitHub for pushing to the registry, but there are a handful that use GitLab or self host that would be totally unaffected.
- PascLeRasc 7y agoI'm in the same position trying to build Neovim, but I have all the sources downloaded already since libmpack always times out. Maybe someone browsing this thread who can't work right now can help: does anyone know how to force/suggest to Cmake that the dependencies it needs are already downloaded and in some src directory?
- TimWolla 7y agosee also: https://news.ycombinator.com/item?id=20498983 https://news.ycombinator.com/item?id=20498983
- js4ever 7y agoGitpocalypse
- dugluak 7y agopush/pull seems to be working for me. Getting 500 server error while trying to browse repositories on the website
- ilikehurdles 7y agoPush and pull doesn't work for me. For a second I thought this was my company's subtle way of terminating my employment.
- 8K832d7tNmiQ 7y agoThat explains a sudden access error when I tried to open an issue in a repo.
- mielecmichal 7y agoGood that git is a distributed VCS. At least our master branch history is available on our local PCs, and we can freely work. Yupi.
- altmind 7y agoHope you dont have any dependencies hosted in github :)
- mielecmichal 7y agoJava and JavaScript programmers have solved it years ago. Maven Central and npm do the job :)
- pmelendez 7y agoDo you do PRs? or have CI/CD pipelines hooked with GitHub?
- wongarsu 7y agonot being able to submit or review PRs for a day is much less impact than not being able to checkout for a day.
- pmelendez 7y agoI guess it depends on the definition of done of each team. We normally can't complete tasks until they are code reviewed, so checking out locally would be the analogous of working offline in a CVCS
- viraptor 7y agoYou can finish the first task as much as you can, then start on the next one...
- u801e 7y agoThe CI/CD pipeline could also be done with a post-receive hook on another remote.
- joshuaturner 7y agoYou've got to wonder what the regex that caused _this_ downtime is going to be
- harel 7y agoThis is the perfect time to take a break. Kick back, have a coffee, contemplate your life choices. That commit can wait, that PR (i was about to merge) can wait too. It's not the end of the world.
- deleted 7y ago[deleted]
- dekhn 7y agowell actually I spent the holiday weekend not working and now I have a time window to work with our east coast contractors before they stop work for the day (they don't work on the weekends or after 5PM) so having the system down isn't really end of the world but it will impact our productivity.
- mattnewton 7y agoWhy not just keep working? That next commit will work just fine without a central server.
- dekhn 7y agomerging work with contractor right now...
- mattnewton 7y agoThat works too, believe it or not. Just add another remote and pull it.
- dekhn 7y agoI only coordinate my remotes through github. I don't use git in "decentralized mode".
- giggles_giggles 7y ago>
- KenanSulayman 7y agoI wonder if all of these downtimes are related to the summer holiday season...
- huffmsa 7y agoThey're probably still moving stuff to Azure after the acquisition. If my memory serves, there was an uptick in outages ~6 months after the buyout.
- dswalter 7y ago6 months after an IPO is often the conclusion of a typical employee lockout period on shares, which can also be a time of more frequent team members leaving. I wonder what the retention incentive period was for Github employees; it's often a year, and the Github acquisition was mid-2019...
- SahAssar 7y agoI think the comment is referring to the other outages like Cloudflare, Stripe and so on.
- justadudeama 7y agoAnd all the interns pushing to prod...
- madsy 7y agoIs my project lost now?
- dmonitor 7y agoDamn. Thought I was going nuts when I couldn’t browse certain repos. Everything I checked said GitHub was up
- CapricornNoble 7y agoThere was a certain repo I was trying to access and I was concerned the content had been taken down. I pulled up the repo in Google Search cache, copied the git URL, and cloned it from my terminal. That worked just fine, only the webpage was completely down.
- gtrubetskoy 7y agoGithub goes down on Monday around 9am pacific time - must be totally random.
- matthewowen 7y agoThe funny thing about this is the idea that west coast engineers actually start work at or before 9am
- InvaderFizz 7y agoI'm rarely in before 9AM. The 10 minute standup at 10:30 AM is the only daily requirement for most staff.
- GhostVII 7y agoI bet a bunch of PR's were built up from the weekend which weren't deployed, and some guy who came in at 9 decided to deploy them and broke things. Always scary to be the one to deploy if no one has deployed in a while.
- bluejellybean 7y agoAlthough it's nice that Git is Git and we can all mostly still work, it still seems foolish to rely on a single point of failure like Github. I've been toying around with the idea of creating a tool that would map the Git api to work with two+ hosting services at the same time. The effect would be something like, run "$git push" and it pushes to Github, Bitbucket, and Gitlab. I can't imagine something like this would be too difficult and would eliminate having to twiddle your thumbs while you wait for things to come back up.
- TimWolla 7y agoYou mean like? git remote set-url --add --push origin git@github.com:Foo/bar.git git remote set-url --add --push origin git@gitlab.com:Foo/bar.git :-) see: https://git-scm.com/docs/git-remote#Documentation/git-remote.txt-emset-urlem https://git-scm.com/docs/git-remote#Documentation/git-remote...
- bluejellybean 7y agoHah, today I learned! It's a great feeling when you don't have to reinvent a wheel, thanks. Now I'm just annoyed that more teams I've been on haven't set this up!
- bartread 7y agoMe too. That's awesome: I've just suggested it to our team. Thanks for sharing GP! One serious question though: how do you deal with PRs when you do this? That's one area where it feels like things could be quite messy, especially if you have quite a few PRs going in throughout the day.
- madsy 7y agoIts up n running for me now, everything works. Location Sweden.
- nwatson 7y agoWorking now.
- acron0 7y agoDefinitely isn't.
- cpncrunch 7y agoSeems to be working for me.
- rhacker 7y agoAs of this post it was working as of 1 minute ago. Is this going to be a PR stunt like hotmail - they're going to say the fix was migrating to Azure or upgraded from Linux to Windows 10.
- joshypants 7y agoCan people stop with the "it's working" or "it's not" - because there is intermittent / unevenly distributed failures. Still have some pages not loading for me.
- carrja99 7y agoIt's what makes these posts cute. Thanks to multi-region and load balancing it will work for some and be broken for others
- deleted 7y ago[deleted]
- ben_jones 7y agoSkimming this thread I found only two important takeaways, are there any more? 1) It's really, really, easy to setup a second mirror that you can switch your CI/CD processes to in the event of downtime in order to avoid being affected 2) Unless you don't vendor your builds and some of your third party dependencies rely on Github as their sole distribution mechanism in which case you are SOL
- sirtoffski 7y agoIt's funny I thought pi-hole was somehow blocking github, but dig returned good responses. dig raw.githubusercontent.com +noall +answer 0 < 12:54:48 ; <<>> DiG 9.14.3 <<>> raw.githubusercontent.com +noall +answer ;; global options: +cmd raw.githubusercontent.com. 6 IN CNAME github.map.fastly.net. github.map.fastly.net. 19 IN A 151.101.20.133 Naturally the first thing I did was check here and nothing was posted. After 20-minutes of TSing I thought it was DNSSec screwing up as after disabling it everything works. And now I come back here and see this.. :D
- tptacek 7y agoThankfully, like virtually all major tech companies, neither GITHUB.COM nor GITHUBUSERCONTENT.COM are DNSSEC-signed, nor are they ever likely to be.
- rodrigopetter 7y agoVery nice page. Some one knows a open source project similat to this githubstatus page?
- thosakwe 7y agoTook this opportunity to set up my own Git server on a VPS I have. It was actually extremely simple to set up the server, limit the git account (ssh only) to only using git-shell, and adding gitweb with basic auth in an nginx setup with letsencrypt, HTTP/2, and FastCGI. Took maybe 20 minutes, and most of it was looking up the necessary commands. That being said, I am, and probably will still be a massive GitHub user, because I value its community aspect. But I also now have unlimited, truly-private Git hosting for peanuts (cheap DigitalOcean server), which is always nice.
- gfdgfdg 7y agoYou can also use gitea, which is setup very quickly as uses almost no resources
- thosakwe 7y agoI definitely considered it, but it struck as more of a "host your own Github," where really all I needed was "host your own Git," which of course comes out-of-the-box.
- tracker1 7y agodon't even need that much... just SSH + Files. On server: mkdir -p ~/repos/foo.git cd ~/repos/foo.git git --bare init On desktop: git remote add backup ssh://user@server/~/repos/foo.git git push --all backup Or something similar, been a while. Can use an absolute path too if sharing, need that path setup for +rwx for user/group. Used to do this as a quick and dirty remote for git usage. You can also setup a git user and add your public key, but find that isn't really necessary. edit: looked up commands, made a couple minor tweaks. note: this doesn't include any kind of LFS support if you're needing it, consider a more complete install, unsure on gitea or gitlab etc.
- smoyer 7y agoThe incident says it was resolved but I still can't get to my profile page.
- marmshallow 7y agoCould be that the fix has been deployed but it takes time to propagate thru.
- zymhan 7y agoYou're not crazy, it was only finally just resolved at 19:47 UTC
- jaequery 7y agoIt gives me the chuckles every time I hear Github or Gitlab goes down. Because one of the main points of Git was to provide a "distributed" version control system. Meaning that you are probably using it wrong if your entire company completely halts to a stop just because Github/Gitlab/Bitbucket are down. Maybe what we need is a service that automatically consolidates them for you so your repos are always online?
- NoodleIncident 7y agoI don't really understand your point. Everyone still has their copy of the repo, and they can keep working on their code. That's the distributed part. It's still important to be able to share that code, trigger builds and tests, etc.
- bob33212 7y agoThese organizations are not distributed hiveminds. Someone in authority has to approve the code change, and someone above that person has to verify that the code is being approved appropriately, and someone above that person has to monitor everything below them. If you are working on a solo project it doesn't matter if the remote repo goes down, just push to another Linux server
- count 7y agoDo you think Github is nothing more than a 'central git' repo? Have you even used the site in the last few years?
- dang 7y ago> Have you even used the site That's a variant of "did you even read the article", which the guidelines ask you not to do (see https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html). That's partly because it's a putdown, but mostly because there's no information in it. Readers who know less than you do are here to learn, so a better version of this comment would share some of what you know.
- 7y ago
- wyldfire 7y agoIt's a hassle, for sure. If this incident cost you time, I would advocate for you to invest a brief amount of time to hedge against GH. The next downtime may not be as brief. These steps will cost you < 15 minutes but you could end up with significant savings. 1. make an account on gitlab 2. create a new repo 3. configure it to mirror your existing GH repo. The mirror will periodically pull from GH without any interaction from you, so you always have another option. To be clear, I don't have any opinions about the operations of Github vs Gitlab. I have no idea which has superior procedures or equipment to avoid downtime. But I am sure that this small effort to diversify is worthwhile. EDIT: wowsers downvotes for practical advice? Downvoters, please chime in on how my advice could be improved.
- zegl 7y agoHow will this save you from outages like this one? The problem is usually not that people loose access to their code (you usually have a local cache), it's that you can't open or review PRs, and that you can't trigger CI builds. Using _both_ GitHub and GitLab simultaniously for more advanced workflows is a big hassle.
- dang 7y agoPlease don't break the site guidelines by going on about downvotes. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
- lightedman 7y agoThis is why I run my own internal development repository, so I don't have to worry about online failures.
- robotsquidward 7y agoI sometimes look forward to outages like this just so I can read the post-downtime-resolution blog post that almost always follow. I find reading about how companies deal with issues when shit hits the fan to be really interesting.
- tdhz77 7y agoI agree. I made a website which breaks down failures and resolutions.
- jshah111 7y agolink?
- deleted 7y ago[deleted]
- maximeblais 7y agoProbably not what he's talking about but there's a nice collection here: https://github.com/danluu/post-mortems https://github.com/danluu/post-mortems
- beebs93 7y agoAgreed. I almost always dial into any large scale events at my company - regardless if it concerns me directly - to listen how ppl critically evaluate data, triage errors, and determine how to mitigate the problem(s). Whatever skill level I have in this area, I cannot nail down if it's from multiple "baptism by fire" situations, learned naturally via university courses in the scientific field, strategy video games (half joking), or other sources I cannot think of. I've listened to some very impressive people handle serious crisis situations and I'm both in awe and curious how they achieved that level of deductive reasoning.
- joshbetz 7y agoThe same way anyone gets good at anything: practice. Knowledge of the system + experience in high pressure situations and you get pretty good at solving those problems over time.
- black_puppydog 7y agoNobody brought up radicle yet? HN what's up? There you go: fully IPFS based version control and collaboration. https://radicle.xyz https://radicle.xyz
- colemickens 7y agoI thought Radicle was cool too, but as I understand it (in its current state), it has a much "larger" SPOF in that changes can only be submitted when the single authoritative repo is online.
- black_puppydog 7y agohow so? If everything is stored in IPSS, then it should always be online. The IPFS daemon is local, so you can commit things without any network connection. Am I misunderstanding what IPFS is?
- colemickens 7y ago"Currently the owner of the project must be online in order to receive any proposed RSM updates from a contributor. Once received and processed, these updates will be written to IPFS by the project owner, and made available to all users who follow that project." -- http://radicle.xyz/docs/#faq http://radicle.xyz/docs/#faq
- colemickens 7y agoWhat the hell? I get downvoted for linking an authoritative answer to the question I was asked? I'm so god damn sick of participating here in good faith and getting wordlessly crapped on for it. The answer supports exactly what I said, and I provided a link. Pretty obvious someone didn't like one of my other comments and then proceeded to downvote the others they could since I commented in multiple threads at the same time. Why is this nonsense allowed? It would be easy to detect.
- 7y ago
- maddyboo 7y agoOver the past 2 or so weeks, I’ve been getting 500 errors from GitHub quite regularly. A page will take a while to load, then finally fail with an octocat error page. Reloading the page usually succeeds with the correct page, so it seems to be some sort of intermittent issue. In the years I’ve used GitHub, I can’t remember this ever happening – I wonder if this incident is related? Has their user base expanded a lot lately and they’re having difficulty scaling their system to meet demand? I know that I have been noticing a seeming increase in activity and new users as GitHub becomes more and more mainstream, I wonder what their growth looks like.
- ryanianian 7y agoGitHub has a really bad track-record on reliability. I worked on our devops systems for a while. Every `git clone` had to have multiple retries, and even then there were multi-minute outages multiple times a month that caused things to turn red and caused distrust in our CI pipeline. I tried hard to get my company to not rely on github as part of our CI process (as others in these comments indicate) but that's an expensive proposition for many - similar to relying on other third-party cdns like dockerhub, npmjs.org, etc.
- noir_lord 7y agoBitbucket isn't really any better, they just surf the line of "right, that's it I'll just run Gitea on a local server, I've had enough" and convenience.
- nanomonkey 7y agoTime to start going back to using a decentralized git archive, like git-ssb. https://github.com/noffle/git-ssb-intro https://github.com/noffle/git-ssb-intro
- tus88 7y ago3 hours later still down. That's a pretty long outage by most standards. Ouch.
- colpabar 7y agoCan we talk about how many services have had outages recently now?
- viraptor 7y agoWe can talk about the Poisson distribution, yes :-) https://en.m.wikipedia.org/wiki/Poisson_distribution https://en.m.wikipedia.org/wiki/Poisson_distribution
- carapace 7y agoGithub: the premier collaborative platform for FLOSS development, which itself is closed-source, made under a closed development process. Based on a distributed CVS, it has become a massive SPOF. As if that wasn't strange enough, it was acquired by Microsoft. Was that the victory of FLOSS or the defeat? How can you tell? WTF is going on? Mood: "Am I going crazy, or is it the world around me?" ~ https://fishbone.bandcamp.com/track/drunk-skitzo https://fishbone.bandcamp.com/track/drunk-skitzo
- Spivak 7y ago* Github being closed-source doesn't really matter. Restrictive software licensing hurts the user of the software which in this case is Github Inc. and only Github Inc. As long as the protocol is open (which it is) whether hosted services that speak that protocol are open or closed source is immaterial. [1] * Git isn't distributed. It's decentralized -- big difference. Distributed means there (usually) aren't any single points of failure. Decentralized means these are many single points of failure but failures are localized. Combine that with the reality that in most markets a "best option for most people" emerges and we get bigger and bigger points of failure. * Microsoft has been trying to get into the socal networking game for a while now and found their match made in heaven with one based on software. Time will tell if the recent Microsoft will be a good steward -- so far it's been pretty good. * Definitely a victory. The dominant VCS is open. More people than ever are contributing to OSS. It's easier than ever to share code. [1] Github's on-prem offering is a completely different story and is very much a problem that it isn't OSS.
- tyre 7y agoThis falls under very specific genre of Hacker News comment that has always struck me as profoundly uninteresting. Github built something that inarguably made development better. - They host some huge number (tens of thousands? hundreds of thousands? millions?) of repositories for free. - They built Github Pages for people to ship websites and host those for free. - They built an intuitive flow for managing issues and pull requests, which is included with every repository for free. - They integrate freely and openly with all sorts of third-party services, some of which compete directly with them. That's quite uncommon for a for-profit software platform of their size. - They have given back immeasurably to the programming community, to the open source community, and to developers including sponsoring conferences, donating office space for events, etc. - Their developers widely share their (i.e. done at/for Github) work and contribute upstream. There is some argument that all of the above are self-serving for Github, which is proven false if you talk to a single developer at Github. Do they adhere to every idealistic principle of FLOSS? No. But, honestly, who gives a crap? So, to answer your question of WTF is going on: real, positive progress.