9 ms·
GitHub was down
- tdonovic 6y agoGetting 500s all over the api, gl whoever is on call
- benbruscella 6y ago"We have identified the source of elevated errors and are working on recovery." A day wrecker!
- luiseduardo 6y agoYeah, our builds are failing for a couple of hours now, and we just use Github because one of the dependencies in NPM downloads a binary from a public repository. We're already forking what we can to our gitlab self-hosted server, but even a simple git clone or even browsing the website can lead to a HTTP 500 right now. Lots of incidents lately, but it's becoming increasingly hard to get away from Github.
- jonny383 6y agoSeriously, again!? I am growing seriously impatient with this. It's been downhill since Microsoft took over. GitLab is starting to look good (or even Gitea self-hosted).
- navanchauhan 6y agoI might just self-host Gitea on my Pi today
- jonny383 6y agoI was going to warn you about SD card corruption, but then I realized even this is probably more reliable than GitHub at this point.
- navanchauhan 6y agoI personally haven't experienced any SD card corruption, even though I am running a Maria-DB instance, Home-Assistant instance and a Pi-Hole instance on a Pi Zero with a 32 gig SD Card ( 23 days uptime as of writing )
- _ikke_ 6y agoI've had 2 sd-cards go corrupt (read-only, any writes were lost). Had a mariadb instance running on it as well. It will hold for for some time, but eventually, the card will give up.
- navanchauhan 6y agoOh no! What will you recommend as a long term solution? I have ordered a Pi 4, should I use an USB Flash Drive / HDD / SATA SSD?
- m0xte 6y agoHonestly throw the thing in a bin and use an old mini pc instead. Lenovo thinkcentre tiny. You can pick them up on eBay for about the same as a fully equipped pi. Pi is 100% not suitable for 100% duty work. It’s just a toy.
- kelnos 6y agoAs someone who's been running a media server and home automation server on his Pi 24/7 for 3 years now, I beg to differ.
- m0xte 6y agoI’ve had 6 so far and all have had reliability issues or weirdness. Mostly related to SD corruption, crashing or power brown out. The power issue was not solved by running them off a proper keysight bench supply. SD cards are quite frankly horrible boot media as well.
- chr15m 6y agoI run a cosy Gitea hosting service over at https://hostedgitea.com/ https://hostedgitea.com/ for those who want their own private Gitea box without the hassle of deploying or maintaining it. Basically I handle all that for you myself. Just starting out but happy to get feedback!
- kohtatsu 6y agoI changed the i18n of Gogs to s/Organization/Folder/, which I think works pretty well, though I'm becoming more interested in gitea's fork. I reckon the same trick will work.
- lprd 6y agoThis is slowly becoming a weekly occurrence ever since Microsoft entered the picture... https://www.githubstatus.com/ https://www.githubstatus.com/
- kenhwang 6y agoI remember it being a weekly occurrence before Microsoft as well. edit: though their historical data doesn't back it up. It's just a 2020 thing it seems; coincidentally picked up right near the middle of February which correlates way too closely with the beginning of COVID.
- Insanity 6y agocorrelation != causation though.
- kzrdude 6y agoFortunately, otherwise covid outbreaks might be caused by github outages
- avancemos 6y agoAnd yet (Correlation != causation) != (!causation)
- p0llard 6y agoThe whole "correlation does not imply causation" thing is completely misunderstood. The issue is with the meaning of the word "imply"; when used in the formal sense as it appears is Classical Logic, correlation does indeed not imply causation. In common parlance however, "imply" is often used to mean "provides evidence for", and correlation can indeed provide (potentially strong) evidence for a hypothesised causal link; the problem lies in people reading "correlation does not imply causation", assuming the informal meaning of "imply", and then going on to reject any notion of causation which uses observed correlation as evidence. Pretty much every empirical science uses notions of correlation (in its various formal statistical guises) to provide support for causation, indeed to reject such reasoning would be to invalidate huge swathes of mainstream accepted science; half the battle in these instances is making the leap from correlation to causation in a manner which is considered scientifically sound.
- quantummkv 6y agoSurely it can't be a coincidence that Github is down every other week after the Microsoft acquisition? Is Microsoft interfering too much? Or did the core technical expertise leave for other greener pastures in Microsoft or outside?
- pcr0 6y agoI believe GitHub's been adding new features much faster after the Microsoft acquisition. Moving faster can lead to breaking things more frequently.
- goldfishlover 6y agoprobably overly complicated layers-upon-layers of services, infrastructure and feature creep fighting with each other
- nicc 6y agoPeople are advocating hosting their own Git repos, but wouldn't those go down, too, and wreck the day even more? Or, are you guys all devops geniuses better than those who work at GitHub?
- dnet 6y agoBut those wouldn't take most of the world's public git repos down all at once just because of a single issue. Single points of failure have a bad reputation for a reason.
- deleted 6y ago[deleted]
- realusername 6y agoOf course everything goes down one day or another, it's just it's been a very common occurrence with Github recently. When my own side-project has more uptime than Github, there's something wrong somewhere.
- zorpner 6y agoGit is notionally decentralized. It should be neither as difficult or uncommon as it is to have multiple repositories, but we have collectively let convenience get in the way of reliability.
- thayne 6y agoWell, if you host it yourself, and it goes down, then you have control over getting it back up, rather than waiting for someone else to fix it. Though really, if it is that important for it to be up, you should mirror it to at least one other provider (ex. self-hosted and github).
- thaumasiotes 6y agoI did work somewhere once with GitLab self-hosted in a VM. The person in charge believed it was important to reboot the VM every so often. One day, for whatever reason, he couldn't bring the VM back up. Self-hosted GitLab was out for the rest of the day. I found this pretty funny.
- a012 6y agoI know Github is down because I'm trying to update a kops cluster, but Github breaks it > kops update cluster error reading channel "https://raw.githubusercontent.com/kubernetes/kops/master/channels/stable": unexpected response code "500 Internal Server Error" for "https://raw.githubusercontent.com/kubernetes/kops/master/channels/stable": 500: Internal Server Error
- breakingcups 6y agoSee, I like the idea of K8S quite a lot. Enough that I decided to set up a cluster from scratch to figure out how everything ties together. I've since grown to strongly dislike how much of the entire ecosystem seems to depend on random unversioned Github gists or direct links into a raw file in a repo somewhere.
- wg0 6y agoThis actually is testament to the fact that the github used to be that damn reliable at one point.
- capableweb 6y agoEh, I'd love to see numbers on this but I don't think this is true. I remember stuff like homebrew breaking even in the beginning of the project because of the dependency on github to pull down formulas and stuff.
- IshKebab 6y agoNPM is the same. You might think that all of the packages are stored on NPM servers. Mostly true, but still when you install some of them they'll download precompiled binaries or whatever directly from Github.
- miyuru 6y agoIf you have IPv6, remove IPv4 to see how many tools break, because github does not have IPv6. Composer is useless in a native IPv6 only servers.
- rvz 6y agoI said this before many times and I'll say it again, consider self-hosting your projects on a solution like GitLab or Gitea to avoid this sort of situation. [0] GNOME, Xfce, Redox, Wireguard, KDE and Haiku all have self-hosted on either cgit, Gitlab, Phabricator or Gitea. [0] https://news.ycombinator.com/item?id=23676072 https://news.ycombinator.com/item?id=23676072
- sparkling 6y agoHow about a more low-tech solution? I don't want/need a full-blown web application like Gitlab, just need somethings than auto-syncs my public and private repos so that i can still access everything when Github is down. Any ideas?
- cyberbanjo 6y agoSSH and git-shell might be enough. https://git-scm.com/book/en/v2/Git-on-the-Server-Setting-Up-the-Server https://git-scm.com/book/en/v2/Git-on-the-Server-Setting-Up-...
- hamandcheese 6y agoThe lowest tech solution is to just wait until the outage is over. It’s 100% free! Seriously, though, any repo that I work on regularly will be cloned to my local dev environment, so it’s not a hard blocker. That said, a cron job on a cheap VPS would probably do the trick.
- jopsen 6y agoMoreover, GitHub will eventually get these outages under control. Or we'll all be driven to gitlab :)
- remus 6y agoIt's still a full full blown web app but I can highly recommend gitea. It's just a single binary so getting setup is really easy. I run an instance and it literally requires no maintenance.
- 6y ago
- quyleanh 6y agoI still don't understand people who always mentions to Microsoft's acquisition. Until the official statement, it isn't Microsoft failure. Don't blame them.
- lopis 6y agoIt's not MS fault, of course, but since MS acquired GH, GH has been much more relaxed. New features are added all the time, clearly not ready for the spotlight. It gives you a different comfort knowing your daddy is there as a safety net.
- tester34 6y agoBut also they have a lot of new customers/users, so maybe the scale's the problem? I don't know.
- ivalm 6y agoThis is a nice reminder that you can have multiple remote repos and push/pull to all of them at the same time. For my side projects I usually use both github and google cloud source (I use gcp). If one is down the other is still available and then just resync when service is recovered.
- mpweiher 6y agoAdditionally, I have a machine on my local network that I push to (and that is backed up separately). Really quick pushes, too :-)
- deleted 6y ago[deleted]
- mosselman 6y agoNice, good tip, I will set that up for my local projects. Not using Github for those, but it nevers hurts to have redundant Git servers.
- kingo55 6y agoAre you thinking like this?: https://jigarius.com/blog/multiple-git-remote-repositories https://jigarius.com/blog/multiple-git-remote-repositories
- ivalm 6y agoYes, push/pull all.
- anamexis 6y agoDo you have something set up for the remotes to sync with each other?
- ivalm 6y agoNo, I do not. Usually though it is easy since whatever reconciliation you did in one repo you can do in another. The only difficulty is if you rewrite remote history, so don’t do it!
- INTPenis 6y agoI just had to access my Github stars to find an old app I bookmarked. No dice. Otherwise I've moved all my current projects to Gitlab so Stars and contributing to other repos are my two most used features atm.
- m0xte 6y agoYay this crashed our Jenkins instance as well.
- YetAnotherNick 6y agoSame mate
- mindfreeze 6y agoAnother Discussion: https://news.ycombinator.com/item?id=23817794 https://news.ycombinator.com/item?id=23817794
- chvid 6y agoCool thing about git is that it is distributed and there is no single point of failure ...
- Symbiote 6y agoIt is. We all have our own copy of the repository, and can still distribute changes using any of the other methods: - A different central server - Email - A shared on-filesystem copy, e.g. local network drive - HTTP or SSH between developer computers (put your repository somewhere where your NginX or Apache serves it, the other developer can "git remote add chvid http://chvid.example.com/repository" http://chvid.example.com/repository").
- holler 6y agoHopefully it's just them reverting their latest UI changes
- mckinney 6y agoHa!
- whatch 6y agoWhat's wrong with them? Maybe it's because I do not use project management features much, but I like new UI more.
- meditative 6y agothe repo top bar is wider than the rest of the content. It feels unbalanced. Personally I'm not a fan of that part. No complaints about any other change.
- misnome 6y agoMy main dislikes are slow, lazy loading, and that you can’t read the last commit messages without extra clicks of the “...” button. Even in pull requests I’ve seen new commits show up without the message (and clicking the expand button, it closed itself after a few seconds). There have been some other annoyances/changes in behaviour that have bugged me too, but mostly stopped remembering them because am resigned to it now.
- thecopy 6y ago>you can’t read the last commit messages without extra clicks of the “...” button You can opt-in to this now. It is a preview functionality, i guess it will be GA in a couple of weeks.
- maxbaines 6y agoSavage Unicorn, Github who knew.
- bezmenov 6y agoThey’re minutes away from dropping below two nines looking at Issues, Pull Requests, Projects. Other services look comparably unreliable.
- Beldin 6y agoIt is ironic that a version control system engineered to be distributed is now typically used in such a centralised way.
- deleted 6y ago[deleted]
- bamboozled 6y agoI guess it's time Github realizes that they're not longer just relied upon for git, but for so much more.
- jschulenklopper 6y agoThis can be merged with https://news.ycombinator.com/item?id=23817794 https://news.ycombinator.com/item?id=23817794
- yuppiepuppie 6y agoITT: Lots of Github hate. Dont forget that change in software is inherently risky and will result in bugs, etc. Id rather have a platform that is always looking to make things better and risking a bit of downtime, than a stale platform that we all know we depend on.
- kkapelon 6y agoI think the issue here is that Github is considered a mature product and it just works. So is there any actual pressure to move fast and break things?
- robertlagrant 6y agoYeah it feels they're just being pushed to out-everything Gitlab, to kill a competitor, before relaxing and stagnating.
- hhas01 6y agoHuman sacrifice, dogs and cats living together… mass hysteria!
- mxschumacher 6y agoin my build pipeline, I query several different package hosts (npm, pypi, docker-hub etc) and Github/Gitlab. If any of them is unavailable, the build fails. What's the best way to keep my own copy of the packages my software needs (and their dependencies), so that my build process is less fragile? Ideally, I'd only have to rely on those 3rd party platforms to download new versions or have them as a backup. When relying on my own copy of required packages - can I expect much faster builds?
- benalfarley 6y agoI've used Nexus for a while without any issues.
- Evidlo 6y agoEverything seems up now. Does anyone know if Github pages went down too?
- maxheadrum 6y agoEmbrace, extend, and... extinguish!
- FBISurveillance 6y agoFrom what I understood, they are having capacity issues with one of their MySQL masters or something. I've read that they are in process of sharding/resolving that but it takes time. To GitHub SRE/oncall people: hang in there, you're awesome.
- terom 6y agoThat's the picture you can get from their recent blogs: * https://github.blog/2020-03-26-february-service-disruptions-post-incident-analysis/ https://github.blog/2020-03-26-february-service-disruptions-... * https://github.blog/2020-07-08-introducing-the-github-availability-report/#availability-report-for-may-and-june https://github.blog/2020-07-08-introducing-the-github-availa...
- ashishb 6y agoWhat if we had a smart failover. Use GitHub and GitLab simultaneously. All issues, all comments, all PRs duplicated. If one goes down you use the other one in the time being with no interference at all. One can probably then build a frontend which magically does this failover for ci/CD etc. Isn't that's how much redundant this should be?
- mro_name 6y agoWho would really rely on a single, external, free, no-guarantees service and not have redundancy to tolerate some hours of downtime? Make github a mirror (at least source-wise) and you can benefit from it's outreach without being held hostage. Am happy with that e.g. https://notabug.org/mro/ShaarliOS/src/master/doap.rdf https://notabug.org/mro/ShaarliOS/src/master/doap.rdf Inspired by https://indieweb.org/POSSE https://indieweb.org/POSSE
- bamboozled 6y agoA lot of people pay for Github?
- mro_name 6y agoseemingly not enough :-)
- sairamkunala 6y ago(copied over from other thread - https://news.ycombinator.com/item?id=23817794 https://news.ycombinator.com/item?id=23817794) Github started doing availability reports. Last month's details in the blog post below with summary of the issue. Stay tuned till next month for the current outage. https://github.blog/2020-07-08-introducing-the-github-availability-report/ https://github.blog/2020-07-08-introducing-the-github-availa...
- spians 6y agoHere's the RSS feed of the blog https://github.blog/feed/ https://github.blog/feed/ I am using https://github.blog/category/engineering/feed/ https://github.blog/category/engineering/feed/ for engineering category