15 ms·

Tell HN: GitHub is down again

442 points by pupdogg 5y ago

Yet somehow https://www.githubstatus.com https://www.githubstatus.com is ALL GREEN! smh

bredren 5y ago
There goes my docs PR for `drf-turbo`. Guess I'll work on my stuff.
dgellow 5y ago
Oh wow, EVERYTHING is red or orange in the status page. That’s unusual.
carbocation 5y ago
And no updates from https://twitter.com/githubstatus https://twitter.com/githubstatus
- nickradford 5y ago
  I mean... it literally just went down.
  - yellow_lead 5y ago
    It's been like 30 mins at least...
    - nickradford 5y ago
      My coworker and I just reviewed and merged a PR, at 12:38 Pacific
  - carbocation 5y ago
    Yes, if it's a manually run account, then the fact that it just went down a few minutes ago would be a good reason for it not to be updated.
- yellow_lead 5y ago
  https://twitter.com/githubstatus/status/1464696434645741574 https://twitter.com/githubstatus/status/1464696434645741574 Got one after you posted at least
milankragujevic 5y ago
I rebooted my modem two times before I realized (with VPN) that Github truly is down. And I need it right now. Guess I'll be setting up a local git instance very soon... edit: by local I meant in the intranet. not on my local machine.
- maximilianroos 5y ago
  > Guess I'll be setting up a local git instance very soon... The incorrectness of this highlights how useful DVCS can be — a git server going down doesn't affect working locally at all.
  - markozivanovic 5y ago
    What if he needs to pull some changes he needs to continue with his work?
    - julianlam 5y ago
      Then a local git instance would preclude him from needing or receiving those changes anyway.
      - markozivanovic 5y ago
        Hmm, I think we understood 'local' as two different things. I understood it as a self hosted git server, like on premises in parent's company that he and his colleagues would use. Now that you mentioned it, I'm starting to think I understood it wrong.
        milankragujevic 5y ago
        Nope, I may have misspoke, but I meant self-hosted git instance in the intranet.
    - Jweb_Guru 5y ago
      You can merge from anywhere, not just Github master. It's a little more work, but a coworker could share their repository without much difficulty. You can even apply patches sent over email, if you're so inclined.
yellow_lead 5y ago
https://downdetector.com/status/github/ https://downdetector.com/status/github/ Yep, can't even push
pupdogg 5y ago
For record keeping: https://imgur.com/a/b6JIQT3 https://imgur.com/a/b6JIQT3
- svnpenn 5y ago
  https://i.imgur.com/SJDJRyo.jpeg https://i.imgur.com/SJDJRyo.jpeg
  - breakingcups 5y ago
    Showing (nearly) all-red for me, 18 minutes later: https://imgur.com/a/plZbFky https://imgur.com/a/plZbFky
    - svnpenn 5y ago
      https://i.imgur.com/3TyNW9A.png https://i.imgur.com/3TyNW9A.png
jmeyer2k 5y ago
Thought I might be the only one seeing this (account-related or something), but nope, just Github's misleading status page...
deleted 5y ago
[deleted]
spapas82 5y ago
Can't they at least fix their status page? https://www.githubstatus.com/ https://www.githubstatus.com/ It returns `All Systems Operational`. I mean what's the point of having a status page if it returns wrong info?
- res0nat0r 5y ago
  These things don't/can't get updated instantly. I was doing some work as of ~5 minutes ago and it was working fine, and is unavailable now. If it is a major outage it will likely be updated shortly.
  - MrStonedOne 5y ago
    My status page for an open source multiplayer game updates the status page for each server on a minute by minute bases. They can do better, they just don't want to.
    - res0nat0r 5y ago
      Large orgs (like Github) don't want or use automated status updates. There is usually always some service having issues in complex systems and immediately updating a public status page does more harm than good, including false alarms which may not affect anything public facing.
      - Dylan16807 5y ago
        If you don't want to report sufficiently small issues, you can put that into the code, can't you? Besides that, how are you going to cause "more harm than good"?
        derefr 5y ago
        More harm than good to the company's own long-term reputation. A status page is a kind of PR. Think of it like a policy for a flight attendant to come out into the cabin to tell everyone what's going on when the plane encounters turbulence. That policy is Public Relations -driven. You only do it if you expect that it's positive PR, compared to not doing it — i.e. if telling people what's going on is boosting your reputation compared to saying nothing at all. If a status page just makes your stakeholders think your service is crappy, such that you'd be better off with no status page at all... then why have a status page? It's not doing its job as a PR tool.
yellow_lead 5y ago
Maybe companies should alert on an increased amount of traffic to their status pages.
- kelnos 5y ago
  That might be a good idea as a last-resort measure, but if you're only finding out problems because customers are telling you about them (even indirectly through a signal like this), your monitoring and alerting is woefully inadequate.
  - hnarn 5y ago
    It’s not last resort, you should do both because you’re alerting on different things.
  - szundi 5y ago
    fun fact here that no matter how short the automated system is set up to alert, customers are faster - no one knows how is that possible
    - therein 5y ago
      Yup, that's another reason why it pays off having a direct line of communication to at least some of your top customers. Can't tell you how many times I've had a customer write to me, only to be receiving some automated alerts 30-40 seconds later.
      - lanstin 5y ago
        And the best customers for this are the pickiest annoying ones
    - strictfp 5y ago
      Generally I agree. Humans are much better at recognizing patterns than machines. But as a counterpoint I was once working in a project where our monitoring dashboard showed an anomaly in incoming traffic. It turned out that it was an ISP problem and that we were the first ones to notice according to them. So maybe part of the answer as to why customers are faster is that different monitoring systems are monitoring each other :)
thebradbain 5y ago
Right in the middle of replying with a lengthy comment on a code review, too :). Guess we'll find out if the server persisted it or not!
- szundi 5y ago
  When it's more that 5 minutes of typing, I always select all and press ctrl-c. And then submit. Saved my time several times.
  - brundolf 5y ago
    Usually just clicking the back button, your browser will remember the form field contents
gray_-_wolf 5y ago
Github.com does not work properly (borderline on "at all") in a browser, and I cannot download a release of one project via curl. This sucks. And is definitely not just "degraded performance for GitHub Actions".
- staticassertion 5y ago
  Yeah I noticed because I'm trying to pull down a protobuf release binary.
- cube00 5y ago
  It's funny seeing the description say "degraded availability" but if you hover over the red icon you get "major outage"
  - noir_lord 5y ago
    degraded covers everything from 1 in 100,000 requests is slow through to "the data centres all simultaneously caught fire". It's weasel words in their purest form. Right up there with "We apologise for any inconveniences you may have experienced". If you are apologising it's because you know you inconvenienced someone. I'd genuinely have more respect if instead they just said "We fucked up, we'll do better". It's at least honest.
revskill 5y ago
Rails is hard to scale.
- activitypea 5y ago
  onejoke.png
aliswe 5y ago
The error message for me, when trying to push, was "Make sure you have the correct permissions" and also something about repo not found ...
sidarape 5y ago
All red now.
ejanus 5y ago
Not only GitHub from my side. I can't access Telegram from browser.
13415 5y ago
Of course it's down. That is the one and only time I wanted to check something on it on Saturday evening.
- tisryno 5y ago
  This is exactly what's happened to me also, oh well, back to procrastinating
zapf 5y ago
Wish we could stop using such msft run nonsense, and build a p2p decentralised PR system. What's the challenges we need to address to make it happen?
- dataflow 5y ago
  > What's the challenges we need to address to make it happen? I imagine the ability to resist $billion offers when you're successful would be one.
- sodality2 5y ago
  > p2p decentralised PR system My first thought was https://radicle.xyz/ https://radicle.xyz/. What a cruel twist of fate. Their downloads page is hosted on Github Pages, which is down right now! https://radicle.xyz/downloads.html https://radicle.xyz/downloads.html In any other situation I'd recommend it since it really was the tool of choice for decentralized git repos with lots of cool features. (Despite the "Empowering developers with DeFi building blocks like NFTs, token streams, and more" mentioned on the home page which throws me off) Edit: It seems like the download page is up and working now.
- pshc 5y ago
  That would be great. But Github is a massive coordination point with network effects and free tiers so it's not so easy to compete. Some other challenges: - despite being decentralized, you still need moderation and access control - open source projects typically aren't great at frontend UI/UX It would be neat if I could publish git repos on IPFS and receive patches from people. It's just hard to compete with a centralized, CDN'd service where you can click and get a result in 100ms. edit: cribbing from sibling comment, it looks like Radicle has thought hard about decentralized git: https://radicle.xyz/blog/radicle-link.html https://radicle.xyz/blog/radicle-link.html
- vimda 5y ago
  > p2p decentralised pr system You mean Git? The Linux Kernel for e.g. does this just fine.
  - 5y ago
jerbearito 5y ago
Thankfully command line / desktop app still work
- minedwiz 5y ago
  Thanks for pointing that out. I needed to clone a new repo. EDIT: nope, command line is down, too.
  - jerbearito 5y ago
    Odd, sorry. I'm able to push and pull from one of my private repos. [Edit: no longer accurate. everything has stopped working]
Gsydvdndh12876 5y ago
Would be interesting to know PRE-MS and POST-MS acquisition statistics on downtime...
- meibo 5y ago
  Even if you were to find the data somewhere, GitHub grew a lot after MS and added a lot of features, including the now most popular CI, so it's not a great comparison.
deeblering4 5y ago
The tone of this post could be improved by describing the problem objectively, and there is no need to say “down again”
- julianlam 5y ago
  Not to mention the vitriol over a manually updated status page not updated the instant a service is down. I mean, the moment it went down I'm sure Github's SREs were paged. Give them a minute to process it first, jeez.
  - MrStonedOne 5y ago
    I have a status page that is automatically and manually updated and it costs 10 bucks a month to run, updates within 2 minutes of a downtime. Github has no excuse. We can and will hold them accountable for their decision to hide information from their paying customers and any other case they prioritize their ego over their users.
  - Dylan16807 5y ago
    Mockery, not vitriol. Nobody asked them to make it manual.
    - naikrovek 5y ago
      it SHOULD be manual. you're foolish if you automate your red/yellow/green status indicators, or even host any of that kind of thing on your own infrastructure in any way. the status page needs to be updated by a human that knows what the issues are, and not automation that will flag everything red the instant anything trivial in the automation fails. you want static pages and images served up, nothing automated or scripted.
      - Dylan16807 5y ago
        Manual reporting tends to undershoot a lot. I find that worse than the risk of the status page going red when things are fine. > you want static pages and images served up Totally separate issue. You can update static pages with any method.
midasuni 5y ago
They should run it in the cloud /s (My self hosted gitlab is working fine)
- Longwelwind 5y ago
  I would wager that your self-hosted GitLab has less uptime than GitHub (or the quality of the deployment). We typically underestimate the downtime of the software we deploy ourselves, probably because we're attached to it, or because we're busy fixing it, instead of having to wait. I'd take the occasional downtimes of cloud solutions, if it means not having to use some of the self-hosted software I had to to use at work.
  - raro11 5y ago
    Nope. Been self-hosting for four years without any issues. We only do security upgrades or wait for x.1 upgrades. Never x.0 The hosted .com however I wouldn't recommend.
  - js4ever 5y ago
    I'm managing a Gitlab instance for 10k users since 3 years, we had zero unplanned downtime in the last 3 years, I just had to apply the upgrade every 3 months and it's running smoothly ever since
    - mishac 5y ago
      If Github had planned downtime every 3 months people would scream bloody murder, so the point stands that github's uptime is almost surely better than someone who selfhosts.
      - deepakkarki 5y ago
        Ironically enough, Github seems to have unplanned downtimes every 3 months!
      - midasuni 5y ago
        Not really, it’s far better to have downtime at a time of your choosing than in the middle of a key time. It’s why we run certain services in my department ourselves rather than central IT, because central IT thinks that the best time for planned downtime is 6pm, which is hilarious as for our department 5pm-11pm is the most critical time, and the best time to make changes is between 9 and 12 when the services are least important (not that we have a full outage, but we can have a degraded outage - our internet service for half of our equipment can take upto 20 seconds to fail over, and not every end device is dual connected)
seirl 5y ago
If you urgently need to retrieve a piece of software, it's likely archived in Software Heritage: http://archive.softwareheritage.org/ http://archive.softwareheritage.org/
- withinboredom 5y ago
  I ended up here because automated cluster deployments failed trying to download releases from GitHub... I wonder if that software is served there and I can update the URLs before GitHub fixes their issues :)
- davegauer 5y ago
  This is fantastic! It also has some projects I thought had been lost to the mists of time when Bitbucket stopped hosting Mercurial repos.
cube00 5y ago
Watermelon status.
qwertox 5y ago
I just had a very odd thing happen to me on GitHub. I accidentally closed my browser so I reopened it with Undo Tab Close, and GitHub's tab title was labeled "Your account recovery is unable to load" for a very brief moment. Then the GitHub error site with the pink unicorn loaded. The URL which was supposed to load was https://github.com/hlissner/doom-emacs https://github.com/hlissner/doom-emacs which I had tried to load about 15 minutes earlier or so, but which would not load because the site is down.
- sodality2 5y ago
  It sounds like some HTML was cached by your browser and it then tried to load some info, which of course results in a 5xx error. Then the existing HTML realized that and showed the error.
  - qwertox 5y ago
    Why would something related to account recovery be cached in such a way that it is the first thing which gets rendered? I really doubt that it is related to caching. Also, that tab was idle for a couple of minutes while I was trying to load GitHub in another tab multiple times before I accidentally closed the browser (just that window, I had other windows of the same browser instance open).
    - sodality2 5y ago
      Just the HTML of the existing page could have been cached, and it may check in the background to reload account info. Then, if this can't be loaded, it shows the error.
donutloop 5y ago
Dam!
activitypea 5y ago
Started learning Emacs tonight, and now I can't access Helm's documentation :\
- brabel 5y ago
  Good time to swith to the lighter (and cooler) Ivy [1]. This site is up, but the actual Ivy Docs Page [2] is also returning a 500! [1] https://writequit.org/denver-emacs/presentations/2017-04-11-ivy.html https://writequit.org/denver-emacs/presentations/2017-04-11-... [2] https://oremacs.com/swiper/ https://oremacs.com/swiper/
  - activitypea 5y ago
    Thanks for the suggestion, but for now I can only bookmark it for later. My priority right now is to reach basic competency ASAP, so I'm running Spacemacs. I love the "batteries included" thing, but just installing it took like an hour of fiddling before I just gave up and ingored the error message. I don't think it'd be wise to fiddle around with the settings if even the defaults are going haywire.
    - brabel 5y ago
      Ah sure, it's good to stick with the default packages in your distribution then... I started from scracth and found that Ivy was more than sufficient for everything I need (and it's quite a lot lighter than Helm). BTW what was the error mesage? Good luck!
- thumbellina 5y ago
  Helm might have info documentation, available offline within Emacs. Try `M-x info-apropos` and type "helm". Info docs form a tree: press '^' to go up a node, 'n' and 'p' to go forward and back on same level, '[' and ']' to go forward/back a page like a book.
rvz 5y ago
Again? Oh dear. I think I lost track of how many times they went down since, suggesting to self-host, repeatedly. [0] So they seem to continue to be very unreliable, I expected them to be up without any issues for a month. Anyway, going all in on GitHub once again makes no sense. So at least have a (self-hosted) backup solution to this. [0] https://news.ycombinator.com/item?id=27366397 https://news.ycombinator.com/item?id=27366397
zoomablemind 5y ago
I got greeted by the image of the frightened Octocat, saying "Oops!!! 500". Thought, that I broke something. ...Copilot must've been bored over the holidays, so probably went on to find a way and snacked on a few freshly updated repos, whichever were close by, then covered it all up as just a degraded perf incident... You've gotta feed da beast!
EugeneOZ 5y ago
Dear Github developers, I brought some love and hugs for you. And a hot coffee with chocolate. Not everyone hates you today, don’t be upset about toxic posts like this one.
- daptaq 5y ago
  > Not everyone hates you today, don’t be upset about toxic posts like this one. No, do get upset. You have contributed to the centralization and increasing dependence on a single entity, while piggybacking of the advances that distributed version control have brought about. You have blurred the difference between Git and your service, to the point that people don't know about the former, and routinely abuse it. You have normalized the fundamentally inconvenient "pull request" workflow as the default for a lot of software development (introducing the artificial steps of "forking" and proposing a "pull request"), to such a degree that people now complain when they are confronted with anything else. This is not a one-off issue, and isn't fixed when the site is operational again. You deserve all the criticism, and no not coffee chocolate.
  - EugeneOZ 5y ago
    One day you will wonder how much you were upset about not so critical things. The lifespan of our civilization is just a glimpse. Do you really think your toxic comment will make it brighter for a moment? Look at this image: https://www.nasa.gov/feature/goddard/2019/hubble-astronomers-assemble-wide-view-of-the-evolving-universe https://www.nasa.gov/feature/goddard/2019/hubble-astronomers...
    - raro11 5y ago
      Did you think yours would?
  - isbvhodnvemrwvn 5y ago
    Propose a better solution then. If it's inconvenient enough that people prefer centralized solutions, is it really better?
- 5y ago
veltas 5y ago
Reminder that git is distributed(TM)
- deleted 5y ago
  [deleted]
hk1337 5y ago
At least CodeSpaces is working https://i.imgur.com/x4IFTwY.png https://i.imgur.com/x4IFTwY.png *EDIT* Now CodeSpaces is gone
slmjkdbtl 5y ago
HN for the most time is a better status page for big sites like Github.
gime_tree_fiddy 5y ago
This seems to be a big one. Even the pulls are failing. I just discovered there isn't not Github mirror for OpenCV code.
axiom92 5y ago
Hmm explains why the following line is suddenly causing my jobs to crash: > nltk.download("punkt") Apparently NLTK uses Github for hosting their sentence tokenization models.
xfbs 5y ago
It's amazing how much stuff breaks when GitHub goes down. I'm doing some Rust coding right now, the rust-s3 crate tells me that to look up what feature I need to enable (tokio + rustls), I need to look into the Cargo.toml. Well, the repo won't load, and nor can I clone it. Well okay, fuck that I can use the default dependencies. But no wait I can't, I can't even do a cargo build because cargo uses a github repository as the source of truth for all crates. No more Rust for me today :(
- pasabagi 5y ago
  Cargo has an --offline option. It's actually pretty possible to use rust totally offline - the doccumentation can be built, for instance, then locally served with (iirc) cargo doc.
  - jiggawatts 5y ago
    Offline should be the default, not an option. Otherwise nobody will notice how fragile their workflow is until it is too late to fix it!
    - Aeolun 5y ago
      I recently realized I was the only person running the API locally when suddenly all our unmocked tests started failing only on my machine. All other tests, including the ones on the CI server, were connecting to the hosted integration environment for any unmocked call (and thus randomly modifying state there).
  - xfbs 5y ago
    I build and host documentation on every commit anyways in the CI. And yes, that is true, I eventually figured it out (had some issues with it at first) but it seems like GitHub is back up so all's well anyhow. I do however wish that there was some public mirror that cargo could fall back to, wouldn't that make a lot of sense?
    - still_grokking 5y ago
lol768 5y ago
Why does their status page have components only for issues/pull requests? What about browsing e.g. repository code, downloading releases etc? There should be a general "web" component marked as unavailable - I can't even hit the homepage.
daxfohl 5y ago
Curious to see the postmortem here. Who makes production changes on Thanksgiving weekend!?!
- deleted 5y ago
  [deleted]
- daxfohl 5y ago
  Granted, it's probably the best time of the year for github to break, precisely because nobody else is pushing production changes....
- bifrost 5y ago
  You don't want to know the answer to this question. I'm totally serious too.
- daxfohl 5y ago
  I don't see any postmortems or any root causes on their incident history page. Just "problem resolved". Do they not make these public?
- isbvhodnvemrwvn 5y ago
  Does GitHub have employees outside of the US?
  - daxfohl 5y ago
    Yes but, given Thanksgiving weekend accounts for a huge percentage of annual retail sales in the US, most companies are reluctant to let anyone touch anything even remotely connected to production that weekend. (My company has been in production freeze for three weeks now.)
arendtio 5y ago
I have a website hosted on Github pages which uses ServiceWorkers to be accessible when being offline (cached). Sadly, Github serves some error page currently and kills the ServiceWorker functionality that way...
albeebe1 5y ago
Here i am on a Saturday afternoon learning how to programatically interact with GitHub using go-git, banging my head on my desk because the code should work, but im getting cryptic errors. I'm searching stackoverflow, nobody seems to have encountered the errors i'm getting (that typically means i'm doing something wrong)...... oh GitHub is down. To GitHubs credit, they've been very reliable for me over the years, it didn't even cross my mind that they could be down.
- vanusa 5y ago
  It didn't even cross my mind that they would be down. As soon as you start banking on "the cloud" to get useful work done, the very first thing that should enter your mind is: "Any of these nifty services can just turn into a pumpkin at any time"
  - dlsa 5y ago
    The real kicker is that they're still other people's computers. There's definite benefits as well as costs. Ignoring either is not a good idea. Centralisation into just a few large providers is another bad idea we're heading towards or have already arrived at.
    - NaturalPhallacy 5y ago
      "The cloud is just other people's computers." is the most succinct way I've heard it.
  - mindcrime 5y ago
    Much like we have the "Fallacies of Distributed Computing"[1], we probably need (if somebody hasn't already created) a "Fallacies of The Cloud". Fallacy #1 - The cloud is reliable [1]: https://en.wikipedia.org/wiki/Fallacies_of_distributed_computing https://en.wikipedia.org/wiki/Fallacies_of_distributed_compu...
  - mr_toad 5y ago
    > "Any of these nifty services can just turn into a pumpkin at any time" Which is just as true for on premise servers.
chrisco255 5y ago
Can't access Vercel because I used Github for authentication. I guess I'm done with using centralized authentication services.
- incognito_mode 5y ago
  Try logging in via email. It'll send you a code to cross-check. Voila!
  - zinekeller 5y ago
    For developers operating a high-risk or widely-used code, most of them opt to using multi-factor authentication, which disables this feature.
    - incognito_mode 5y ago
      I mean at Vercel. I have 2FA at GH but I can log in to Vercel using my GH email address via the email option on the Vercel website, after confirming the code on-screen versus the code sent to my email address.
- dmitshur 5y ago
  In case you or others find it interesting, I've had luck removing an authentication SPOF¹ by using IndieAuth on my personal site². I wish it were an option on more of the sites where I need to sign in. ¹ https://twitter.com/dmitshur/status/1223304521767669761 https://twitter.com/dmitshur/status/1223304521767669761 ² https://github.com/shurcooL/home/commit/bb504a4ef0d7c552d363742ebefb9e5cd5a32e40 https://github.com/shurcooL/home/commit/bb504a4ef0d7c552d363...
brw 5y ago
At least this outage allowed me to discover this cute pixel art octocat loading icon on the activity feed. Never noticed it before because I believe it always loads near instantaneously, or maybe I just never paid enough attention to it. https://i.imgur.com/6Uwlwh7.gif https://i.imgur.com/6Uwlwh7.gif
eugenhotaj 5y ago
Let me guess, someone pushed a bad config.
saltmeister 5y ago
it's not
samuelroland 5y ago
it seems to be up again...
- culi 5y ago
  same for me but the status page doesn't say so
rglover 5y ago
https://cheatcode.co/opinions/control-your-user-data https://cheatcode.co/opinions/control-your-user-data
ronyfadel 5y ago
Explains why I had less app downloads today. (My app binaries are hosted on Github; not sure if it’s the way to go but hey it works).
- Stampo00 5y ago
  Not today it doesn't. ;-)
rlucas 5y ago
I also had intermittency on 2021-11-27 between 1-2 PM PT. Would work one minute, not the next, retry and it works. Feels round-robin-y kind of Heisenbug.
axiom92 5y ago
Looks like it's up now.
deleted 5y ago
[deleted]
lordnacho 5y ago
It's not that hard to push to multiple remotes. Perhaps more projects ought to have multiple GitHub/Bitbucket/GitLab/etc repos. Then it wouldn't matter if one of them went down now and again.
egberts1 5y ago
That’s why I have a THREE-tier GIT repository. - Github - My Git server - My local git No need to worry about failures. :-)
deleted 5y ago
[deleted]
hk1337 5y ago
Homebrew is also not working since it relies on GitHub a lot.
gime_tree_fiddy 5y ago
Is it possible for GitHub to mirror the releases in multiple different places(they likely do that, but I mean complete isolation where an outage like this doesn't break the downloads). Maybe like a proxy to object store, so it is a little more reliable(a setup such as this, should have less moving and custom parts). So in a moment like this, you can convert https://github.com/opencv/opencv/archive/4.5.3.zip https://github.com/opencv/opencv/archive/4.5.3.zip to https://archive.github.com/opencv/opencv/archive/4.5.3.zip https://archive.github.com/opencv/opencv/archive/4.5.3.zip. Maybe an implicit agreement of somewhat stale data by add the sub-domain "archive.". They'll try to maintain low sync times on a "best effort basis".)
- amelius 5y ago
  Anybody could do that, I suppose. (I mean for the public repos).
riksucks 5y ago
Seems like the webhooks that populate most of the pages don't work, and hence leaves you with an incomplete github page. For some, no page at all But it also seems like that apart from the webhooks, the APIs and the pages served are slower too.
robertwt7 5y ago
omg it is still down!
naikrovek 5y ago
> Yet somehow https://www.githubstatus.com https://www.githubstatus.com is ALL GREEN! smh this is because status pages, at least the green/yellow/red light bits, are usually updated manually by a human. because if you automate those things, the automation can fail. also, it's a weekend, a holiday weekend for some, so expecting updates to happen on the minute is a little unrealistic. ALSO, it may take a bit of time for their "what is and is not really down" process a bit of time to be followed. on the weekend. by people who maybe haven't done this before outside of a training event 9 months ago or something. > somehow it is not hard to imagine how, to me.
- chagaif 5y ago
  A company the size of Microsoft can not hide behind - "it's weekend we also need a break". I assume a company this size has teams working around the clock, the question is the priority. I guess GitHub doesn't have a high enough priority which is very sad. I understand it's humans but we're also humans and we managed to figure out exactly what's up and what's down and we even reported it very conveniently here on HN for the people at GitHub for free Sorry for the rant I spent a few very confused minutes trying to understand wtf I'm doing wrong, because naive me was thinking there's no way GitHub is down....
  - naikrovek 5y ago
    anything can go down, and has. the only thing I know of that hasn't is the internet as a whole. some parts go offline, but the internet itself, as a cohesive system, has never failed. I bet it will one day, what with legislative "kill switches" being a thing.
    - chagaif 5y ago
      Is it really that difficult to have a system run redundantly? (Not trying to be sarcastic here)
- saganus 5y ago
  Perhaps it would be useful to have a sort of different shade of green/yellow/red (perhaps striped?) indicating it's a value determined by automatic processes, not yet confirmed by a human. At least that way, as a user of the service, you can make the assumption that it could be broken, and act accordingly, or at least take certain precautions while you wait for the human-confirmed status value.
zaptheimpaler 5y ago
> github goes down once a year on a saturday > HN: REEEE THE CLOUD CANNOT BE TRUSTED BURN IT DOWN
blindmute 5y ago
Git is Distributed ™

442 points by pupdogg 5y ago

Yet somehow https://www.githubstatus.com https://www.githubstatus.com is ALL GREEN! smh

bredren 5y ago

There goes my docs PR for `drf-turbo`. Guess I'll work on my stuff.

dgellow 5y ago

Oh wow, EVERYTHING is red or orange in the status page. That’s unusual.

carbocation 5y ago

And no updates from https://twitter.com/githubstatus https://twitter.com/githubstatus

nickradford 5y ago
I mean... it literally just went down.
- yellow_lead 5y ago
  It's been like 30 mins at least...
  - nickradford 5y ago
    My coworker and I just reviewed and merged a PR, at 12:38 Pacific
- carbocation 5y ago
  Yes, if it's a manually run account, then the fact that it just went down a few minutes ago would be a good reason for it not to be updated.
yellow_lead 5y ago
https://twitter.com/githubstatus/status/1464696434645741574 https://twitter.com/githubstatus/status/1464696434645741574 Got one after you posted at least

milankragujevic 5y ago

I rebooted my modem two times before I realized (with VPN) that Github truly is down. And I need it right now. Guess I'll be setting up a local git instance very soon... edit: by local I meant in the intranet. not on my local machine.

maximilianroos 5y ago
> Guess I'll be setting up a local git instance very soon... The incorrectness of this highlights how useful DVCS can be — a git server going down doesn't affect working locally at all.
- markozivanovic 5y ago
  What if he needs to pull some changes he needs to continue with his work?
  - julianlam 5y ago
    Then a local git instance would preclude him from needing or receiving those changes anyway.
    - markozivanovic 5y ago
      Hmm, I think we understood 'local' as two different things. I understood it as a self hosted git server, like on premises in parent's company that he and his colleagues would use. Now that you mentioned it, I'm starting to think I understood it wrong.
      - milankragujevic 5y ago
        Nope, I may have misspoke, but I meant self-hosted git instance in the intranet.
  - Jweb_Guru 5y ago
    You can merge from anywhere, not just Github master. It's a little more work, but a coworker could share their repository without much difficulty. You can even apply patches sent over email, if you're so inclined.

yellow_lead 5y ago

https://downdetector.com/status/github/ https://downdetector.com/status/github/ Yep, can't even push

pupdogg 5y ago

For record keeping: https://imgur.com/a/b6JIQT3 https://imgur.com/a/b6JIQT3

svnpenn 5y ago
https://i.imgur.com/SJDJRyo.jpeg https://i.imgur.com/SJDJRyo.jpeg
- breakingcups 5y ago
  Showing (nearly) all-red for me, 18 minutes later: https://imgur.com/a/plZbFky https://imgur.com/a/plZbFky
  - svnpenn 5y ago
    https://i.imgur.com/3TyNW9A.png https://i.imgur.com/3TyNW9A.png

jmeyer2k 5y ago

Thought I might be the only one seeing this (account-related or something), but nope, just Github's misleading status page...

deleted 5y ago

[deleted]

spapas82 5y ago

Can't they at least fix their status page? https://www.githubstatus.com/ https://www.githubstatus.com/ It returns `All Systems Operational`. I mean what's the point of having a status page if it returns wrong info?

res0nat0r 5y ago
These things don't/can't get updated instantly. I was doing some work as of ~5 minutes ago and it was working fine, and is unavailable now. If it is a major outage it will likely be updated shortly.
- MrStonedOne 5y ago
  My status page for an open source multiplayer game updates the status page for each server on a minute by minute bases. They can do better, they just don't want to.
  - res0nat0r 5y ago
    Large orgs (like Github) don't want or use automated status updates. There is usually always some service having issues in complex systems and immediately updating a public status page does more harm than good, including false alarms which may not affect anything public facing.
    - Dylan16807 5y ago
      If you don't want to report sufficiently small issues, you can put that into the code, can't you? Besides that, how are you going to cause "more harm than good"?
      - derefr 5y ago
        More harm than good to the company's own long-term reputation. A status page is a kind of PR. Think of it like a policy for a flight attendant to come out into the cabin to tell everyone what's going on when the plane encounters turbulence. That policy is Public Relations -driven. You only do it if you expect that it's positive PR, compared to not doing it — i.e. if telling people what's going on is boosting your reputation compared to saying nothing at all. If a status page just makes your stakeholders think your service is crappy, such that you'd be better off with no status page at all... then why have a status page? It's not doing its job as a PR tool.

yellow_lead 5y ago

Maybe companies should alert on an increased amount of traffic to their status pages.

kelnos 5y ago
That might be a good idea as a last-resort measure, but if you're only finding out problems because customers are telling you about them (even indirectly through a signal like this), your monitoring and alerting is woefully inadequate.
- hnarn 5y ago
  It’s not last resort, you should do both because you’re alerting on different things.
- szundi 5y ago
  fun fact here that no matter how short the automated system is set up to alert, customers are faster - no one knows how is that possible
  - therein 5y ago
    Yup, that's another reason why it pays off having a direct line of communication to at least some of your top customers. Can't tell you how many times I've had a customer write to me, only to be receiving some automated alerts 30-40 seconds later.
    - lanstin 5y ago
      And the best customers for this are the pickiest annoying ones
  - strictfp 5y ago
    Generally I agree. Humans are much better at recognizing patterns than machines. But as a counterpoint I was once working in a project where our monitoring dashboard showed an anomaly in incoming traffic. It turned out that it was an ISP problem and that we were the first ones to notice according to them. So maybe part of the answer as to why customers are faster is that different monitoring systems are monitoring each other :)

thebradbain 5y ago

Right in the middle of replying with a lengthy comment on a code review, too :). Guess we'll find out if the server persisted it or not!

szundi 5y ago
When it's more that 5 minutes of typing, I always select all and press ctrl-c. And then submit. Saved my time several times.
- brundolf 5y ago
  Usually just clicking the back button, your browser will remember the form field contents

gray_-_wolf 5y ago

Github.com does not work properly (borderline on "at all") in a browser, and I cannot download a release of one project via curl. This sucks. And is definitely not just "degraded performance for GitHub Actions".

staticassertion 5y ago
Yeah I noticed because I'm trying to pull down a protobuf release binary.
cube00 5y ago
It's funny seeing the description say "degraded availability" but if you hover over the red icon you get "major outage"
- noir_lord 5y ago
  degraded covers everything from 1 in 100,000 requests is slow through to "the data centres all simultaneously caught fire". It's weasel words in their purest form. Right up there with "We apologise for any inconveniences you may have experienced". If you are apologising it's because you know you inconvenienced someone. I'd genuinely have more respect if instead they just said "We fucked up, we'll do better". It's at least honest.

revskill 5y ago

Rails is hard to scale.

activitypea 5y ago
onejoke.png

aliswe 5y ago

The error message for me, when trying to push, was "Make sure you have the correct permissions" and also something about repo not found ...

sidarape 5y ago

All red now.

ejanus 5y ago

Not only GitHub from my side. I can't access Telegram from browser.

13415 5y ago

Of course it's down. That is the one and only time I wanted to check something on it on Saturday evening.

tisryno 5y ago
This is exactly what's happened to me also, oh well, back to procrastinating

zapf 5y ago

Wish we could stop using such msft run nonsense, and build a p2p decentralised PR system. What's the challenges we need to address to make it happen?

dataflow 5y ago
> What's the challenges we need to address to make it happen? I imagine the ability to resist $billion offers when you're successful would be one.
sodality2 5y ago
> p2p decentralised PR system My first thought was https://radicle.xyz/ https://radicle.xyz/. What a cruel twist of fate. Their downloads page is hosted on Github Pages, which is down right now! https://radicle.xyz/downloads.html https://radicle.xyz/downloads.html In any other situation I'd recommend it since it really was the tool of choice for decentralized git repos with lots of cool features. (Despite the "Empowering developers with DeFi building blocks like NFTs, token streams, and more" mentioned on the home page which throws me off) Edit: It seems like the download page is up and working now.
pshc 5y ago
That would be great. But Github is a massive coordination point with network effects and free tiers so it's not so easy to compete. Some other challenges: - despite being decentralized, you still need moderation and access control - open source projects typically aren't great at frontend UI/UX It would be neat if I could publish git repos on IPFS and receive patches from people. It's just hard to compete with a centralized, CDN'd service where you can click and get a result in 100ms. edit: cribbing from sibling comment, it looks like Radicle has thought hard about decentralized git: https://radicle.xyz/blog/radicle-link.html https://radicle.xyz/blog/radicle-link.html
vimda 5y ago
> p2p decentralised pr system You mean Git? The Linux Kernel for e.g. does this just fine.
- 5y ago

jerbearito 5y ago

Thankfully command line / desktop app still work

minedwiz 5y ago
Thanks for pointing that out. I needed to clone a new repo. EDIT: nope, command line is down, too.
- jerbearito 5y ago
  Odd, sorry. I'm able to push and pull from one of my private repos. [Edit: no longer accurate. everything has stopped working]

Gsydvdndh12876 5y ago

Would be interesting to know PRE-MS and POST-MS acquisition statistics on downtime...

meibo 5y ago
Even if you were to find the data somewhere, GitHub grew a lot after MS and added a lot of features, including the now most popular CI, so it's not a great comparison.

deeblering4 5y ago

The tone of this post could be improved by describing the problem objectively, and there is no need to say “down again”

julianlam 5y ago
Not to mention the vitriol over a manually updated status page not updated the instant a service is down. I mean, the moment it went down I'm sure Github's SREs were paged. Give them a minute to process it first, jeez.
- MrStonedOne 5y ago
  I have a status page that is automatically and manually updated and it costs 10 bucks a month to run, updates within 2 minutes of a downtime. Github has no excuse. We can and will hold them accountable for their decision to hide information from their paying customers and any other case they prioritize their ego over their users.
- Dylan16807 5y ago
  Mockery, not vitriol. Nobody asked them to make it manual.
  - naikrovek 5y ago
    it SHOULD be manual. you're foolish if you automate your red/yellow/green status indicators, or even host any of that kind of thing on your own infrastructure in any way. the status page needs to be updated by a human that knows what the issues are, and not automation that will flag everything red the instant anything trivial in the automation fails. you want static pages and images served up, nothing automated or scripted.
    - Dylan16807 5y ago
      Manual reporting tends to undershoot a lot. I find that worse than the risk of the status page going red when things are fine. > you want static pages and images served up Totally separate issue. You can update static pages with any method.

midasuni 5y ago

They should run it in the cloud /s (My self hosted gitlab is working fine)

Longwelwind 5y ago
I would wager that your self-hosted GitLab has less uptime than GitHub (or the quality of the deployment). We typically underestimate the downtime of the software we deploy ourselves, probably because we're attached to it, or because we're busy fixing it, instead of having to wait. I'd take the occasional downtimes of cloud solutions, if it means not having to use some of the self-hosted software I had to to use at work.
- raro11 5y ago
  Nope. Been self-hosting for four years without any issues. We only do security upgrades or wait for x.1 upgrades. Never x.0 The hosted .com however I wouldn't recommend.
- js4ever 5y ago
  I'm managing a Gitlab instance for 10k users since 3 years, we had zero unplanned downtime in the last 3 years, I just had to apply the upgrade every 3 months and it's running smoothly ever since
  - mishac 5y ago
    If Github had planned downtime every 3 months people would scream bloody murder, so the point stands that github's uptime is almost surely better than someone who selfhosts.
    - deepakkarki 5y ago
      Ironically enough, Github seems to have unplanned downtimes every 3 months!
    - midasuni 5y ago
      Not really, it’s far better to have downtime at a time of your choosing than in the middle of a key time. It’s why we run certain services in my department ourselves rather than central IT, because central IT thinks that the best time for planned downtime is 6pm, which is hilarious as for our department 5pm-11pm is the most critical time, and the best time to make changes is between 9 and 12 when the services are least important (not that we have a full outage, but we can have a degraded outage - our internet service for half of our equipment can take upto 20 seconds to fail over, and not every end device is dual connected)

seirl 5y ago

If you urgently need to retrieve a piece of software, it's likely archived in Software Heritage: http://archive.softwareheritage.org/ http://archive.softwareheritage.org/

withinboredom 5y ago
I ended up here because automated cluster deployments failed trying to download releases from GitHub... I wonder if that software is served there and I can update the URLs before GitHub fixes their issues :)
davegauer 5y ago
This is fantastic! It also has some projects I thought had been lost to the mists of time when Bitbucket stopped hosting Mercurial repos.

cube00 5y ago

Watermelon status.

qwertox 5y ago

I just had a very odd thing happen to me on GitHub. I accidentally closed my browser so I reopened it with Undo Tab Close, and GitHub's tab title was labeled "Your account recovery is unable to load" for a very brief moment. Then the GitHub error site with the pink unicorn loaded. The URL which was supposed to load was https://github.com/hlissner/doom-emacs https://github.com/hlissner/doom-emacs which I had tried to load about 15 minutes earlier or so, but which would not load because the site is down.

sodality2 5y ago
It sounds like some HTML was cached by your browser and it then tried to load some info, which of course results in a 5xx error. Then the existing HTML realized that and showed the error.
- qwertox 5y ago
  Why would something related to account recovery be cached in such a way that it is the first thing which gets rendered? I really doubt that it is related to caching. Also, that tab was idle for a couple of minutes while I was trying to load GitHub in another tab multiple times before I accidentally closed the browser (just that window, I had other windows of the same browser instance open).
  - sodality2 5y ago
    Just the HTML of the existing page could have been cached, and it may check in the background to reload account info. Then, if this can't be loaded, it shows the error.

donutloop 5y ago

Dam!

activitypea 5y ago

Started learning Emacs tonight, and now I can't access Helm's documentation :\

brabel 5y ago
Good time to swith to the lighter (and cooler) Ivy [1]. This site is up, but the actual Ivy Docs Page [2] is also returning a 500! [1] https://writequit.org/denver-emacs/presentations/2017-04-11-ivy.html https://writequit.org/denver-emacs/presentations/2017-04-11-... [2] https://oremacs.com/swiper/ https://oremacs.com/swiper/
- activitypea 5y ago
  Thanks for the suggestion, but for now I can only bookmark it for later. My priority right now is to reach basic competency ASAP, so I'm running Spacemacs. I love the "batteries included" thing, but just installing it took like an hour of fiddling before I just gave up and ingored the error message. I don't think it'd be wise to fiddle around with the settings if even the defaults are going haywire.
  - brabel 5y ago
    Ah sure, it's good to stick with the default packages in your distribution then... I started from scracth and found that Ivy was more than sufficient for everything I need (and it's quite a lot lighter than Helm). BTW what was the error mesage? Good luck!
thumbellina 5y ago
Helm might have info documentation, available offline within Emacs. Try `M-x info-apropos` and type "helm". Info docs form a tree: press '^' to go up a node, 'n' and 'p' to go forward and back on same level, '[' and ']' to go forward/back a page like a book.

rvz 5y ago

Again? Oh dear. I think I lost track of how many times they went down since, suggesting to self-host, repeatedly. [0] So they seem to continue to be very unreliable, I expected them to be up without any issues for a month. Anyway, going all in on GitHub once again makes no sense. So at least have a (self-hosted) backup solution to this. [0] https://news.ycombinator.com/item?id=27366397 https://news.ycombinator.com/item?id=27366397

zoomablemind 5y ago

I got greeted by the image of the frightened Octocat, saying "Oops!!! 500". Thought, that I broke something. ...Copilot must've been bored over the holidays, so probably went on to find a way and snacked on a few freshly updated repos, whichever were close by, then covered it all up as just a degraded perf incident... You've gotta feed da beast!

EugeneOZ 5y ago

Dear Github developers, I brought some love and hugs for you. And a hot coffee with chocolate. Not everyone hates you today, don’t be upset about toxic posts like this one.

daptaq 5y ago
> Not everyone hates you today, don’t be upset about toxic posts like this one. No, do get upset. You have contributed to the centralization and increasing dependence on a single entity, while piggybacking of the advances that distributed version control have brought about. You have blurred the difference between Git and your service, to the point that people don't know about the former, and routinely abuse it. You have normalized the fundamentally inconvenient "pull request" workflow as the default for a lot of software development (introducing the artificial steps of "forking" and proposing a "pull request"), to such a degree that people now complain when they are confronted with anything else. This is not a one-off issue, and isn't fixed when the site is operational again. You deserve all the criticism, and no not coffee chocolate.
- EugeneOZ 5y ago
  One day you will wonder how much you were upset about not so critical things. The lifespan of our civilization is just a glimpse. Do you really think your toxic comment will make it brighter for a moment? Look at this image: https://www.nasa.gov/feature/goddard/2019/hubble-astronomers-assemble-wide-view-of-the-evolving-universe https://www.nasa.gov/feature/goddard/2019/hubble-astronomers...
  - raro11 5y ago
    Did you think yours would?
- isbvhodnvemrwvn 5y ago
  Propose a better solution then. If it's inconvenient enough that people prefer centralized solutions, is it really better?
5y ago

veltas 5y ago

Reminder that git is distributed(TM)

deleted 5y ago
[deleted]

hk1337 5y ago

At least CodeSpaces is working https://i.imgur.com/x4IFTwY.png https://i.imgur.com/x4IFTwY.png *EDIT* Now CodeSpaces is gone

slmjkdbtl 5y ago

HN for the most time is a better status page for big sites like Github.

gime_tree_fiddy 5y ago

This seems to be a big one. Even the pulls are failing. I just discovered there isn't not Github mirror for OpenCV code.

axiom92 5y ago

Hmm explains why the following line is suddenly causing my jobs to crash: > nltk.download("punkt") Apparently NLTK uses Github for hosting their sentence tokenization models.

xfbs 5y ago

It's amazing how much stuff breaks when GitHub goes down. I'm doing some Rust coding right now, the rust-s3 crate tells me that to look up what feature I need to enable (tokio + rustls), I need to look into the Cargo.toml. Well, the repo won't load, and nor can I clone it. Well okay, fuck that I can use the default dependencies. But no wait I can't, I can't even do a cargo build because cargo uses a github repository as the source of truth for all crates. No more Rust for me today :(

pasabagi 5y ago
Cargo has an --offline option. It's actually pretty possible to use rust totally offline - the doccumentation can be built, for instance, then locally served with (iirc) cargo doc.
- jiggawatts 5y ago
  Offline should be the default, not an option. Otherwise nobody will notice how fragile their workflow is until it is too late to fix it!
  - Aeolun 5y ago
    I recently realized I was the only person running the API locally when suddenly all our unmocked tests started failing only on my machine. All other tests, including the ones on the CI server, were connecting to the hosted integration environment for any unmocked call (and thus randomly modifying state there).
- xfbs 5y ago
  I build and host documentation on every commit anyways in the CI. And yes, that is true, I eventually figured it out (had some issues with it at first) but it seems like GitHub is back up so all's well anyhow. I do however wish that there was some public mirror that cargo could fall back to, wouldn't that make a lot of sense?
  - still_grokking 5y ago

lol768 5y ago

Why does their status page have components only for issues/pull requests? What about browsing e.g. repository code, downloading releases etc? There should be a general "web" component marked as unavailable - I can't even hit the homepage.

daxfohl 5y ago

Curious to see the postmortem here. Who makes production changes on Thanksgiving weekend!?!

deleted 5y ago
[deleted]
daxfohl 5y ago
Granted, it's probably the best time of the year for github to break, precisely because nobody else is pushing production changes....
bifrost 5y ago
You don't want to know the answer to this question. I'm totally serious too.
daxfohl 5y ago
I don't see any postmortems or any root causes on their incident history page. Just "problem resolved". Do they not make these public?
isbvhodnvemrwvn 5y ago
Does GitHub have employees outside of the US?
- daxfohl 5y ago
  Yes but, given Thanksgiving weekend accounts for a huge percentage of annual retail sales in the US, most companies are reluctant to let anyone touch anything even remotely connected to production that weekend. (My company has been in production freeze for three weeks now.)

arendtio 5y ago

I have a website hosted on Github pages which uses ServiceWorkers to be accessible when being offline (cached). Sadly, Github serves some error page currently and kills the ServiceWorker functionality that way...

albeebe1 5y ago

Here i am on a Saturday afternoon learning how to programatically interact with GitHub using go-git, banging my head on my desk because the code should work, but im getting cryptic errors. I'm searching stackoverflow, nobody seems to have encountered the errors i'm getting (that typically means i'm doing something wrong)...... oh GitHub is down. To GitHubs credit, they've been very reliable for me over the years, it didn't even cross my mind that they could be down.

vanusa 5y ago
It didn't even cross my mind that they would be down. As soon as you start banking on "the cloud" to get useful work done, the very first thing that should enter your mind is: "Any of these nifty services can just turn into a pumpkin at any time"
- dlsa 5y ago
  The real kicker is that they're still other people's computers. There's definite benefits as well as costs. Ignoring either is not a good idea. Centralisation into just a few large providers is another bad idea we're heading towards or have already arrived at.
  - NaturalPhallacy 5y ago
    "The cloud is just other people's computers." is the most succinct way I've heard it.
- mindcrime 5y ago
  Much like we have the "Fallacies of Distributed Computing"[1], we probably need (if somebody hasn't already created) a "Fallacies of The Cloud". Fallacy #1 - The cloud is reliable [1]: https://en.wikipedia.org/wiki/Fallacies_of_distributed_computing https://en.wikipedia.org/wiki/Fallacies_of_distributed_compu...
- mr_toad 5y ago
  > "Any of these nifty services can just turn into a pumpkin at any time" Which is just as true for on premise servers.

chrisco255 5y ago

Can't access Vercel because I used Github for authentication. I guess I'm done with using centralized authentication services.

incognito_mode 5y ago
Try logging in via email. It'll send you a code to cross-check. Voila!
- zinekeller 5y ago
  For developers operating a high-risk or widely-used code, most of them opt to using multi-factor authentication, which disables this feature.
  - incognito_mode 5y ago
    I mean at Vercel. I have 2FA at GH but I can log in to Vercel using my GH email address via the email option on the Vercel website, after confirming the code on-screen versus the code sent to my email address.
dmitshur 5y ago
In case you or others find it interesting, I've had luck removing an authentication SPOF¹ by using IndieAuth on my personal site². I wish it were an option on more of the sites where I need to sign in. ¹ https://twitter.com/dmitshur/status/1223304521767669761 https://twitter.com/dmitshur/status/1223304521767669761 ² https://github.com/shurcooL/home/commit/bb504a4ef0d7c552d363742ebefb9e5cd5a32e40 https://github.com/shurcooL/home/commit/bb504a4ef0d7c552d363...

brw 5y ago

At least this outage allowed me to discover this cute pixel art octocat loading icon on the activity feed. Never noticed it before because I believe it always loads near instantaneously, or maybe I just never paid enough attention to it. https://i.imgur.com/6Uwlwh7.gif https://i.imgur.com/6Uwlwh7.gif

eugenhotaj 5y ago

Let me guess, someone pushed a bad config.

saltmeister 5y ago

it's not

samuelroland 5y ago

it seems to be up again...

culi 5y ago
same for me but the status page doesn't say so

rglover 5y ago

https://cheatcode.co/opinions/control-your-user-data https://cheatcode.co/opinions/control-your-user-data

ronyfadel 5y ago

Explains why I had less app downloads today. (My app binaries are hosted on Github; not sure if it’s the way to go but hey it works).

Stampo00 5y ago
Not today it doesn't. ;-)

rlucas 5y ago

I also had intermittency on 2021-11-27 between 1-2 PM PT. Would work one minute, not the next, retry and it works. Feels round-robin-y kind of Heisenbug.

axiom92 5y ago

Looks like it's up now.

deleted 5y ago

[deleted]

lordnacho 5y ago

It's not that hard to push to multiple remotes. Perhaps more projects ought to have multiple GitHub/Bitbucket/GitLab/etc repos. Then it wouldn't matter if one of them went down now and again.

egberts1 5y ago

That’s why I have a THREE-tier GIT repository. - Github - My Git server - My local git No need to worry about failures. :-)

deleted 5y ago

[deleted]

hk1337 5y ago

Homebrew is also not working since it relies on GitHub a lot.

gime_tree_fiddy 5y ago

Is it possible for GitHub to mirror the releases in multiple different places(they likely do that, but I mean complete isolation where an outage like this doesn't break the downloads). Maybe like a proxy to object store, so it is a little more reliable(a setup such as this, should have less moving and custom parts). So in a moment like this, you can convert https://github.com/opencv/opencv/archive/4.5.3.zip https://github.com/opencv/opencv/archive/4.5.3.zip to https://archive.github.com/opencv/opencv/archive/4.5.3.zip https://archive.github.com/opencv/opencv/archive/4.5.3.zip. Maybe an implicit agreement of somewhat stale data by add the sub-domain "archive.". They'll try to maintain low sync times on a "best effort basis".)

amelius 5y ago
Anybody could do that, I suppose. (I mean for the public repos).

riksucks 5y ago

Seems like the webhooks that populate most of the pages don't work, and hence leaves you with an incomplete github page. For some, no page at all But it also seems like that apart from the webhooks, the APIs and the pages served are slower too.

robertwt7 5y ago

omg it is still down!

naikrovek 5y ago

> Yet somehow https://www.githubstatus.com https://www.githubstatus.com is ALL GREEN! smh this is because status pages, at least the green/yellow/red light bits, are usually updated manually by a human. because if you automate those things, the automation can fail. also, it's a weekend, a holiday weekend for some, so expecting updates to happen on the minute is a little unrealistic. ALSO, it may take a bit of time for their "what is and is not really down" process a bit of time to be followed. on the weekend. by people who maybe haven't done this before outside of a training event 9 months ago or something. > somehow it is not hard to imagine how, to me.

chagaif 5y ago
A company the size of Microsoft can not hide behind - "it's weekend we also need a break". I assume a company this size has teams working around the clock, the question is the priority. I guess GitHub doesn't have a high enough priority which is very sad. I understand it's humans but we're also humans and we managed to figure out exactly what's up and what's down and we even reported it very conveniently here on HN for the people at GitHub for free Sorry for the rant I spent a few very confused minutes trying to understand wtf I'm doing wrong, because naive me was thinking there's no way GitHub is down....
- naikrovek 5y ago
  anything can go down, and has. the only thing I know of that hasn't is the internet as a whole. some parts go offline, but the internet itself, as a cohesive system, has never failed. I bet it will one day, what with legislative "kill switches" being a thing.
  - chagaif 5y ago
    Is it really that difficult to have a system run redundantly? (Not trying to be sarcastic here)
saganus 5y ago
Perhaps it would be useful to have a sort of different shade of green/yellow/red (perhaps striped?) indicating it's a value determined by automatic processes, not yet confirmed by a human. At least that way, as a user of the service, you can make the assumption that it could be broken, and act accordingly, or at least take certain precautions while you wait for the human-confirmed status value.

zaptheimpaler 5y ago

> github goes down once a year on a saturday > HN: REEEE THE CLOUD CANNOT BE TRUSTED BURN IT DOWN

blindmute 5y ago

Git is Distributed ™