10 ms·
GitHub had connectivity problems
- codingthebeach 11y agoPossibly just Round 2 of the massive DDOS from March: https://github.com/blog/1981-large-scale-ddos-attack-on-github-com https://github.com/blog/1981-large-scale-ddos-attack-on-gith... http://arstechnica.com/security/2015/04/ddos-attacks-that-crippled-github-linked-to-great-firewall-of-china/ http://arstechnica.com/security/2015/04/ddos-attacks-that-cr...
- disordinary 11y agoWell that's me not able to work.
- feld 11y agoIf only git were a distributed version control
- Zirro 11y agoIt would be nice not to have to rely on centralized servers at all. A P2P/torrent client built specifically for sharing code from git repositories, perhaps?
- madez 11y agoBut there is much less commercial incentive behind a p2p alternative. Centralized services are easier to monetize. I look forward to the day governments start to fund free software, for example through the GNU project. Then we'll all have better tools. Actually, there were already some fundings. Germany funded GnuPG to port it to Windows. What do I care what is the new world record in sprinting? But I very much do care about a new release of libreboot, libreCMC/openWRT or Debian! I hope one day we will see peaceful international competitions between nations of who can provide the best/most used free software.
- dTal 11y agoHear, hear. I've talked about this with people and it's surprisingly hard to convince them that government-sponsored software is a good idea. While a tiny fraction of the amount we already spend on commercial software would utterly transform the free software landscape, it's not politically viable as long as commercial software vendors have lobby power.
- danieldk 11y agoLet's not forget that it is not in many governments' interest to have distributed tools (with strong encryption). It is much easier to take down stuff on centralized services. BTW, it is hear, hear ;).
- wtbob 11y ago> it's surprisingly hard to convince them that government-sponsored software is a good idea That's because it's not. Governments are not reliably good, and their money comes with far more strings than private money. The FSF have it right: people should write free software because proprietary software is immoral.
- simoncion 11y ago> [Government] money comes with far more strings than private money. Have you ever worked on a government-funded project? The Tor folks have and do. :) > ...[P]eople should write free software because proprietary software is immoral. USGov does fund Libre and Open Source software. One big example is the Tor Project.
- mallamanis 11y agoThere is this https://code.google.com/p/gittorrent/ https://code.google.com/p/gittorrent/ although I've never tried it... edit: a more recent (?) link https://github.com/cjb/gittorrent https://github.com/cjb/gittorrent
- ywecur 11y agoIt would be really nice to get it to succeed. Decentralisation is almost always preferable.
- jlcx 11y agoI think they're two different projects, but this is exactly what I thought of. A separate Chinese DDoS was even mentioned by cjb in his announcement of GitTorrent a few months ago: http://blog.printf.net/articles/2015/05/29/announcing-gittorrent-a-decentralized-github/ http://blog.printf.net/articles/2015/05/29/announcing-gittor...
- david_ar 11y agoSee http://ipfs.io http://ipfs.io and https://github.com/cryptix/git-remote-ipfs https://github.com/cryptix/git-remote-ipfs
- hadeharian 11y agoIPFS, anyone?
- leni536 11y agoYeah, I'm hyped too.
- tomphoolery 11y ago> A P2P/torrent client built specifically for sharing code from git repositories, perhaps? Or, we could both set up HTTP servers on our respective machines and simply push to each others repos.
- JustSomeNobody 11y agoI thought that was the point of git...
- moviuro 11y agogithub is not only git... how many times will we have to tell that? PRs, wikis, issues are the reasons why we use github...
- liw 11y agoIf only it were possible to keep those things in git itself. Oh wait, it is.
- paulrouget 11y agoCan you explain? How do you store issues in git?
- deleted 11y ago[deleted]
- wtbob 11y ago> How do you store issues in git? An org-mode file named 'issues,' with each issue under its own heading? A directory named 'issues,' with an org-mode or CommonMark file for each issue? A directory named 'issues,' with a directory for each issue, with CommonMark, org-mode or restructured text files for each person's comments?
- a-priori 11y agoYou commit your issue database to the same repo as your code and track it alongside your code. Then you just need a tool to manage the database for you, and there's a few tools out there to do this: http://ditz.rubyforge.org/ http://ditz.rubyforge.org/ http://pitz.tplus1.com/ http://pitz.tplus1.com/ https://github.com/jeffWelling/ticgit https://github.com/jeffWelling/ticgit http://www.bugseverywhere.org/ http://www.bugseverywhere.org/ ... and probably others. That's just based on a quick Google search. One nice thing about this approach is that your issue's state follows your code through merges. When you fix the bug you mark it as 'fixed' and commit that change to a branch. When it gets merged into master, the 'fixed' status gets merged as well.
- gioele 11y ago
- benihana 11y agoOh cool, I didn't realize git being distributed solved the communication problems that arise on an engineering team. Back to work everyone, git is distributed version control so we don't need pull requests or gists or any kind of easy-to-read historical record that is also accessible for non-engineers.
- Gigablah 11y agoIf it's that critical then someone needs to fix the SPOF.
- gerbilly 11y agoIf it's really that important, why not host your own gitlab or pay for a self hosted github installation?
- chopman 11y agoIf only there was a way to do p2p communication that you could use to send patch files. Uhmmm.
- JustSomeNobody 11y agoI think the bigger point is to have a contingency plan in place so your whole engineering team isn't sitting on their thumbs with stupid grins on their faces.
- otis_inf 11y agoYou don't have to edit the files directly on github in the browser, really ;) Just edit locally, commit even! It's magic!
- allannienhuis 11y agohttp://stackoverflow.com/questions/11459475/should-i-check-in-node-modules-to-git-when-creating-a-node-js-app-on-heroku http://stackoverflow.com/questions/11459475/should-i-check-i... That might not be your specific problem, but I'm guessing a few people have this problem today. I'm glad I check my libraries into my local repository :) Using a local fall-back for popular libraries hosted on cdns is a good idea too. To be fair, I don't do it because I think it's stupid not to, I do it because I often work on my laptop while travelling and have to be able to continue to work without an internet connection :)
- juquinha123 11y agoGuys, what's the purpose to attack a service like Github?!
- webmonkeyuk 11y agoIt could be for something like this: http://arstechnica.com/security/2015/04/ddos-attacks-that-crippled-github-linked-to-great-firewall-of-china/ http://arstechnica.com/security/2015/04/ddos-attacks-that-cr...
- rethab 11y agoInstitutions that don't like code that is hosted on / distributed via Github.
- theallan 11y agoProof that you can take down a service as large and as well connected a Github perhaps.
- jsjohnst 11y agoWhat makes you think Github is "large and well connected"? Don't get me wrong, I love GitHub, but due to their architecture decisions, they aren't anywhere near a great example of best practices in the industry for preventing DDoS attacks.
- JosephRedfern 11y agoMaybe they're unable to take down a service larger or better connected than Github, perhaps?
- nly 11y agoI'm intrigued. What architectural decisions?
- jsjohnst 11y agoOne that would really help with DDoS attacks is if they didn't do everything on github.com. Layer 3 DDoS mitigation will always be cheaper and more effective than Layer 7. If it's not obvious what I mean, here's an example: Current: http://github.com/jsjohnst/project http://github.com/jsjohnst/project One (of many) better approaches: http://jsjohnst.github.com/project http://jsjohnst.github.com/project
- mambodog 11y agoI wonder if this is related to https://news.ycombinator.com/item?id=10101469 https://news.ycombinator.com/item?id=10101469
- pravj 11y agoI'm nobody to assure this but I can see one thing common in both the cases, 'Dragon'.
- deleted 11y ago[deleted]
- marinintim 11y agoAll these ideas like "let's fetch our deps directly from github" sound good until github is not down.
- ilghiro 11y agoYeh it's definitely preferable to have one of the two developers your small startup can probably afford spending a good portion of their time rolling out, securing and maintaining your own infrastructure
- biggestbob 11y agoYeh cos those are two only two options available. Good thinking.
- bad_user 11y agoGitHub is not meant for distributing dependencies. Maven Central on the other hand is, the difference being that it is mirrored and if repo1.maven.org goes down, it's not a big deal and your project can still be built and deployed.
- wereHamster 11y ago.. until they become a target of a DDoS. Also, if GitHub is down you can still fetch your dependencies from somewhere else.
- JumpJumpJump 11y agoYou have not understood the concept of 'mirrored'.
- chopman 11y agoWell I guess all mirrors could get targeted. Also, what stops github from getting mirrors themselves?
- deleted 11y ago[deleted]
- sajal83 11y agohttps://pulse.turbobytes.com/results/55dc40efecbe400bf8001468/ https://pulse.turbobytes.com/results/55dc40efecbe400bf800146... problems during TCP connect https://pulse.turbobytes.com/results/55dc40faecbe400bf8001469/ https://pulse.turbobytes.com/results/55dc40faecbe400bf800146... traceroute looks ok..
- axelerator 11y agoI'm wondering if there is a corresponding traffic spike on the hn servers ;-)
- Retr0spectrum 11y agoMy github windows GUI just crashed. Looks like the devs didn't consider suboptimal network conditions.
- stonewhite 11y agoTitle should be "Github _is_ under DDoS and having connectivity problems" edit: TIL my English wasn't as good as I thought.
- paublyrne 11y agoThat's subjective. In America companies tend to be referred to as singular entities, but in the UK they would be referred to as plural. Neither is wrong, just different idioms.
- muchcomment 11y agoTIL
- webmonkeyuk 11y agoAs you say - In English they would be referred to as plural. I'm English.
- OJFord 11y agoNo, 'are' is, I believe, also incorrect in BrE. It's commonplace here, but not correct. (Or so I was taught!)
- DanBC 11y agohttp://blog.oxforddictionaries.com/2011/09/agreement-over-collective-nouns/ http://blog.oxforddictionaries.com/2011/09/agreement-over-co... > In British English it’s absolutely fine to treat most collective nouns as either singular or plural – you can say my husband’s family is very religious or my husband’s family are very religious. http://itre.cis.upenn.edu/~myl/languagelog/archives/001874.html http://itre.cis.upenn.edu/~myl/languagelog/archives/001874.h... etc. With something like collective nouns it's probably wrong to make blanket statements about correct and incorrect, even if you're a prescriptivist not descriptivist. EDIT: And while I love (but am hopeless with) English usage this is the least interesting bit of the submitted article.
- 11y ago
- bitinn 11y agoTotally speculation, but maybe related: https://news.ycombinator.com/item?id=10115641 https://news.ycombinator.com/item?id=10115641
- ck2 11y agoYeah my first reaction was it is government based and probably China. I know, we should keep building up their economy by manufacturing nearly everything there, that should stop them.
- deleted 11y ago[deleted]
- brador 11y agoOver the long term, it will. An educated wealthy populace doesn't accept tyranny for long.
- ninjin 11y agoTo some extent, most likely, yes. But my perspective on the PRC is that the powers that be know that they will only remain in power for as long as they can provide, or rather, the illusion of them providing, a continuous increase in wealth. This in part would be why information is key in China. Assange had a very positive view of it when he said "I often say that censorship is always cause for celebration. It is always an opportunity, because it reveals fear of reform. It means that the power position is so weak that you have got to care about what people think." [1]. [1]: https://wikileaks.org/Transcript-Meeting-Assange-Schmidt#996 https://wikileaks.org/Transcript-Meeting-Assange-Schmidt#996
- geomark 11y agoYou know, the implication of that for the U.S. is pretty disturbing.
- 11y ago
- haosdent 11y agoChina!
- andrewvc 11y agoI've always wondered why github hasn't considered switching to a distributed sub domain layout to help ameliorate this problem. Surely if they spread their infra out that would make ddos that much harder. A subdomain per user or repo should work wonderfully
- elktea 11y agoPossibly for SEO reasons.
- tvvocold 11y agoI don't think that will solve the problem.
- andrewvc 11y agoWell if x% of repos remain up that goes a long way. Right now it's all or nothing. Also if you're going to respond include a reason why you disagree
- zrm 11y agoSeparating users by subdomain leaks information. An observer can trivially see your DNS queries but not which repository you access over https or ssh. There is also little benefit in it. If the subdomains only point to the same servers then the same level of traffic will still take them all down, but if different subdomains point to different servers then it makes attacks easier because the attacker only needs enough resources to overwhelm 5% of the servers instead of all of them.
- fallingfrog 11y agoUgh, how horrible! DDoS'ing github is like kicking a puppy in the face.
- dantillberg 11y agoGithub has certainly done an amazing PR job if that is a common perception.
- pavel_lishin 11y agoIsn't it more like barricading bus drivers inside their homes because someone you don't like takes the bus to work?
- konradb 11y agoI always say that making an argument using analogies is like trying to tie your shoes with laces made of butter. :)
- deleted 11y ago[deleted]
- rorykoehler 11y agoI haven't experience any connectivity issues on Github today. Just pushed some code and it worked fine and promptly.
- kevinbowman 11y agoDoes anyone else find the Github status "messages" page [0] a bit jarring in how it's organised? The way that, when read from top to bottom, time goes forwards within a day but backwards across days? I guess I normally wouldn't notice, but there are currently some messages on there from today and yesterday and I found it hard to read as a story (either forwards or backwards) - I had to jump around a bit to figure it out. [0] https://status.github.com/messages https://status.github.com/messages
- henkdevelopment 11y agoI just noticed as well, i got really confused for a few minutes. Bit of a weird design.
- mahouse 11y agoDesign looks very, very confusing regarding timestamps. https://i.imgur.com/wminZWg.png https://i.imgur.com/wminZWg.png
- hellbanner 11y agoIs there a repo you can make an issue / PR on?
- Someone1234 11y agoDoubtful, GitHub isn't open source software.
- 11y ago
- masukomi 11y agoi'm sorry but... GitHub IS not GitHub are Examples: GitHub IS a web site. GitHub is a SAAS app. The only plural aspect to GitHub is its employees and they are not being DDoSed. The web site (singular) is. I'm not going to bother explaining the problems with "under DDoS" or the other issues with this sentence.
- dhimes 11y agoIt's a British thing. Don't try to understand it- just let it be. :)
- toxican 11y agoIt makes sense, but "is" is too ingrained in my brain to ever go British.
- dopamean 11y agoI'd be surprised if no one at GitHub has said "we're being DDoS'd this morning." GitHub is a website. It's also a company of people who probably identify closely with the product they build, like many of us. I don't think it's that crazy to use "are" here.
- gtk40 11y agoPerhaps it is this? https://en.wikipedia.org/wiki/Comparison_of_American_and_British_English#Formal_and_notional_agreement https://en.wikipedia.org/wiki/Comparison_of_American_and_Bri...
- codeshaman 11y agoTechnically, the Github servers are being attacked. The "website" (or "app") is the abstract thing that runs on those servers. The servers are trying to service a flood of incoming tcp connections and http requests from rogue clients, which slows them down. So "are" is not entirely out of place, but of course we all got the message ;).
- friendzis 11y agoI wonder why are they providing time in FLE[S]T (same as EE[S]T)? Are their engineering team based in Eastern Europe or what?
- benmarks 11y agoGitHub DDoS, pointing out ironic single points of failure since the beginning!
- deleted 11y ago[deleted]
- luck87 11y agoAbout 3 hours in github twitter profile: https://twitter.com/githubstatus/status/636159212876361728 https://twitter.com/githubstatus/status/636159212876361728
- gcdgcd6 11y agoIt is the order from the leader of chinese communists XiBaozi --- New Hitler of NAZI in China
- nns 11y agoWhat do they possibly get by DDoS'ng GitHub ? Is this out of pure malice or are there any probable commercial gains to this?
- maxander 11y agoThere's two common flavors of comment on this post, namely "why would anyone do this?" and "well, I guess that means I can't do any work this morning." These could go together somehow. :)
- JustSomeNobody 11y agoIndeed.
- drzaiusapelord 11y agoConsidering its the Chinese government, yet again, its probably a mix of both. It makes it harder for their nationals to get VPN software that goes through their firewall and also is an assholic statement against the West's ideas about freedom of speech, assembly, and peaceful changes of power via popular elections and multi-party government. The Chinese are most likely hoping they can force github to get rid of these projects by threatening periodic DDOSing. They also probably feel emasculated that South Korea told their pet regime to knock off the stupidity and China and North Korea capitulated to their demands yet again. The timing of this is far from a coincidence. China's foreign policy is almost 100% hinged on having North Korea attack the west with impunity and they don't like that Park is actually pushing back against this dynamic. So now China is having its hissy fit on Github instead on the Korean peninsula. Yeah, lets keep rewarding them with factory contracts, right guys?
- rem7 11y agoSomeone doesn't wanna work today.
- TallGuyShort 11y agoMandrill reported they were investigating system slowness a short time ago, also.
- xedarius 11y agoI do wonder why GitHub gets hit, and people say it's a high profile target, yes almost definitely. Another thought occurs to me though, I guess one of the problems when writing a DDoS is measuring how well it performs. The GitHub status screen provides a lot of useful metrics for tuning such an attack.
- niuzeta 11y agoIf I may posit... 1. Github is very well known and cater specifically for the tech crowd. So, an attack on Github is more likely to be talked in the tech crowd, which I would assume the people who would try out DDoS attacks be more likely to be part of. Validation is a weird thing. 2. In a convoluted sense, taking down Github doesn't harm as many bystanders. If that makes any sense... 3. As you've mentioned, very clear useful metrics for the attacks. 4. Crowding effect? Maybe attacking Github has become some 'cool' things to try in that community. I'm just imagining at this point.
- mightybyte 11y ago> Validation is a weird thing. And bragging rights. Imagine the street cred you get amongst that community if you take down something like github...
- juliangoldsmith 11y ago>I do wonder why GitHub gets hit, and people say it's a high profile target, yes almost definitely. It could be related to things like this: https://news.ycombinator.com/item?id=10101469 https://news.ycombinator.com/item?id=10101469
- kzhahou 11y agoIt seems a certainty to me that github will be breached one of these days, and all internal data (i.e., private repos) made public. On that day, so many companies will inadvertently become open source! Do we have any info on what steps github takes to prevent this? I ask as a paying customer (with both personal and corporate accounts).
- atonse 11y agoIf such a massive scale breach would happen, it would probably have to take weeks because GitHub probably has so much data. (I would guess on the order of hundreds of TB or a few PB). It would be more likely to just have handful of high visibility repos.
- mrks_ 11y agoYou're probably right. It's unlikely that a hacker would care about my private repos. That being said, I'm also curious about GitHub's efforts to prevent such a scenario.
- stingraycharles 11y agoWhich makes me wonder: how would one prevent such a scenario, where you cannot simply encrypt with a shared key given github's auth model ?
- Rafert 11y agoLike http://homakov.blogspot.nl/2013/03/hacking-github-with-webkit.html http://homakov.blogspot.nl/2013/03/hacking-github-with-webki... or http://homakov.blogspot.nl/2014/02/how-i-hacked-github-again.html http://homakov.blogspot.nl/2014/02/how-i-hacked-github-again... ? ;)
- dubcanada 11y agoYou would probably get a better answer if you asked Github this question.
- 11y ago
- lai 11y agoIs this because of shadowsocks and China is pissed again?
- ElegantGiraffe 11y agoBitbucket is currently experiencing some issues too [0]. Is it related? [0] https://bitbucket.statuspage.io/ https://bitbucket.statuspage.io/
- dubcanada 11y agoI'm going to assume yes. And I'm going to assume what ever is the issue has a mirror on bitbucket. But I don't think anyone can say for sure.
- dsmithatx 11y agoI'm getting 500 timeouts on Bitbucket. Didn't Google Code go read-only today? Don't you guys think Bitbucket and Github might be having issues due to people migrating off of Google Code at the last minute?
- smpetrey 11y agoPossibly, but they are having degraded API performance and high load on their front-end systems. http://status.bitbucket.org http://status.bitbucket.org
- tuxt 11y agoAnd a famous break-through-GFW software called goagent is deleted from github. *sigh https://github.com/goagent/goagent https://github.com/goagent/goagent
- bdcravens 11y agoYou may want to explain how this is related?
- LinuxBender 11y agoIf GitHub folks are technical in nature, couldn't they simply have a secondary mirror of their own that they host on their own servers / clouds and then reference both primary and secondary? Perhaps redirect the secondary to the primary if it is reachable to avoid bandwidth issues?