7 ms·
Ask HN: Is GitHub down?
Not loading for me at all, but status page shows green across the board.
- etimberg 3y agoSame for me
- computershit 3y agoYup. Totally down, happened right as I opened a PR.
- jonapro 3y agoYou broke it.
- adpirz 3y agoSame
- saintblasphemer 3y agoSame here.
- wcarss 3y agoI can't even load the status page.
- makeworld 3y agoStatus page loads for me, it just incorrectly says all green: https://www.githubstatus.com/ https://www.githubstatus.com/
- onionisafruit 3y agoMaybe it's right and we're all wrong
- ketchupdebugger 3y agoThat requires manual acknowledgement. Probably requires an approval from a VP or some high level exec to change that status.
- wcarss 3y agoInteresting, I'm used to using status.github.com, which got hit by whatever issue is hitting the main site.
- computronus 3y agoGitHub should monitor their status page traffic for spikes, which probably mean something is wrong somewhere, even if they themselves haven't noticed yet.
- ketchupdebugger 3y agothey just updated it, now its all red
- gwbas1c 3y agoI can't push from my desktop, and https://github.com/ https://github.com/ spins in the browser
- MorningInfidel 3y agoSame. Spins for a while then dreaded `ERR_CONNECTION_TIMED_OUT` chrome error page.
- aftaylor2 3y agoUnable to load Github website or push using git-remote-https as of the last several minutes.
- GrumpyNl 3y agoSame here, it cant send a verification sms.
- zamalek 3y agoUse the downtime to purchase a Yubikey/FIDO2.
- coder9874 3y agoYep it's down. Why do they even bother with the status page
- joshstrange 3y agoYep, looks like it's down. Can't pull/push and can't even get the web to load at all.
- ActualHacker 3y agoYep
- whitewingjek 3y agoYes, github status says all is operational but hacker news is faster. Go figure.
- munk-a 3y agoAt this point I don't trust self-owned status pages at all - those crowd-sourced ones where users report issues are much faster to respond to outages that may never even go reported by status pages.
- jetpackjoe 3y agoIts down for me
- klysm 3y agoSeems to be a fairly catastrophic failure. https://github.com/ https://github.com/ fails to load. https://www.githubstatus.com/ https://www.githubstatus.com/ shows all green as of this writing. Nothing on the twitter yet https://twitter.com/githubstatus https://twitter.com/githubstatus edit: The outage is now acknowledged on the status page https://www.githubstatus.com/ https://www.githubstatus.com/ edit: EU folks appear to have things working so it looks like a regional network fault
- edgyquant 3y agoEven the status page isn’t loading for me currently
- pc86 3y agoStatus page is red now, it probably only checks once every couple minutes.
- mkolassa 3y agoLooks like they finally updated the second status page to show the outage.
- ChadyWady 3y agoLooks like they've updated it now.
- urda 3y agoStatus page is fully red now.
- fluix 3y agoPages hosted on github pages also show the unicorn 503 page. However, https://pages.github.com/?(null) https://pages.github.com/?(null) loads.
- pjot 3y agoAck’d on twitter: https://twitter.com/githubstatus/status/1674475870931808256 https://twitter.com/githubstatus/status/1674475870931808256
- 3y ago
- stefanos82 3y agoSo down right now...I wonder why they still use https://www.githubstatus.com/ https://www.githubstatus.com/ that reports everything is alright when it's not!
- maschultz 3y agoMaybe the status page is down - it needs a status page to tell us if the status page is down
- SV_BubbleTime 3y agoIf it takes someone to manually change it from green to red, that does seem to defeat the purpose.
- evulhotdog 3y agoYep, and when money comes into play when you're supposed to meet SLAs, you certainly don't want it being automatic.
- klysm 3y agoPossibly, but sometimes with failures this bad you can't get to the page to update it.
- munk-a 3y agoThere was that hilarious multi-hour AWS failure a while back where the status page was updated via one of their internal services... and that service went down as part of the outage.
- jabart 3y agoNo it doesn't. The amount of false alarm alerts you can get with internet based monitoring is more than 0. You could have a BGP route break things for one ISP your monitoring happens to use. You could have a failover event happening where it takes 30 seconds for everything to converge. I have multiple monitors on my app at 1 minute intervals from different vendors and ALWAYS a user will email us within 5 seconds of an issue. It's not realistic for a company to have automatic status updates trigger things without a person manually reviewing them because too many things can go wrong on the automatic status update to cause panic.
- gazelle21 3y ago[dead]
- deleted 3y ago[deleted]
- ushakov 3y agoActions won't start for me
- awesomelvin 3y ago[dead]
- charlieyu1 3y agoLoads for me Unless it is cached Edit: I could even login
- lights0123 3y agoSame. I'm in Europe and it loads slowly but it gets there.
- charlieyu1 3y agoAnd it seems like loading HN is even slower
- iso1631 3y agoProbably because everyone in the US is piling on HN to see if github is down
- Maxion 3y agoSame, in EU and is just as normal.
- tednotfred 3y agoDown for me right now.
- turtleyacht 3y agoWon't load for me. https://githubstatus.com https://githubstatus.com shows all green, but it's not the case...
- jerrygenser 3y agoGithub status page doesn't even load for me ... "We're having a really bad day, the unicorns have taken over"
- JohnMakin 3y agoFirst noticed when trying to pull a helm chart - get a 503 backend error page.
- hn8305823 3y ago140.82.113.0/24 is visible in the global routing table: route-views>sh bgp 140.82.113.0 BGP routing table entry for 140.82.113.0/24, version 62582026 Paths: (19 available, best #4, table default) The route is verified by RPKI so it's not a route hijack. Edit: deleted traceroute
- llimllib 3y agoYour requests made it farther than mine - mine get to charter in nyc and die there 6 lag-26.nycmny837aw-bcr00.netops.charter.com (24.30.201.130) 158.033 ms lag-16.nycmny837aw-bcr00.netops.charter.com (66.109.6.74) 29.575 ms lag-416.nycmny837aw-bcr00.netops.charter.com (66.109.6.10) 30.077 ms 7 lag-1.pr2.nyc20.netops.charter.com (66.109.9.5) 81.351 ms 37.879 ms 27.877 ms 8 * * *
- alexeldeib 3y agoI’m in US east coast with a dev box in Helsinki. My dev box can still hit github.com, but I can’t at home.
- llimllib 3y agoWhat IP does it resolve to in Helsinki?
- Maxion 3y agoFrom Finland, but not Helsinki: 140.82.121.3
- alexeldeib 3y agoyep, same
- musha68k 3y agoCurious aside: That sounds like quite the roundtrip for day to day work. How do you cope with that, used to IntelliJ IDEs? ;D
- abe-101 3y agoSame here
- kashnote 3y agoAre these status pages updated manually? At the very least it should be able to detect that the home page doesn't even load and turn itself red.
- ChrisArchitect 3y ago[dupe] / merge https://news.ycombinator.com/item?id=36523843 https://news.ycombinator.com/item?id=36523843
- pieresqi 3y ago[dead]
- almost_usual 3y agoWorks for me?
- distortionfield 3y agoDown for me as well, Operation Timed Out errors on all attempted SSH connections
- numbsafari 3y agoIf they used AI to rebuild their system and migrated it Azure, I bet they would stop having all the problems.
- 88913527 3y agoI can't tell if this is sarcasm or not.
- numbsafari 3y agoI'm pretty sure if I e-mailed my sales rep right now they would tell me that Azure Dev Ops doesn't have these problems.
- jesterman 3y agoFrom GitHub - Incident On 2023-06-29: https://www.githubstatus.com/incidents/gqx5l06jjxhp?u=ry1xb4w4nsqr https://www.githubstatus.com/incidents/gqx5l06jjxhp?u=ry1xb4...
- duckkg5 3y agoStatus page updated showing all red https://www.githubstatus.com/ https://www.githubstatus.com/
- edgyquant 3y agoEverytime github goes down, and my push/pull is rejected, I immediately assume they’ve discovered I’m incompetent and fired me. And I’m the head of engineering at my company.
- antoineMoPa 3y ago> And I’m the head of engineering at my company. Haha! Happy to see impostor syndrome goes all the way to the top of the hierarchy.
- dreday 3y agoSomething that I've found that helps me with impostor syndrome is to read and talk about it. Check out this Ted talk from the co-founder of Atlassian. https://www.ted.com/talks/mike_cannon_brookes_how_you_can_use_impostor_syndrome_to_your_benefit?language=en https://www.ted.com/talks/mike_cannon_brookes_how_you_can_us...
- vrosas 3y agoYou’re not alone.
- Maxion 3y agoI think the same thing every time my credentials to our issue tracker expires and I have to log in again. I am the lead dev on two projects.
- CommitSyn 3y agoIs there a name for firing trauma? I'm like this ever since I was scapegoated.
- scrum-treats 3y agoCan you describe what happened that qualifies as "scapegoated"?
- pandacake 3y ago
- deleted 3y ago[deleted]
- mbreese 3y agohttps://www.githubstatus.com/ https://www.githubstatus.com/ Just flipped to red. See here: https://www.githubstatus.com/incidents/gqx5l06jjxhp https://www.githubstatus.com/incidents/gqx5l06jjxhp >Investigating - We are currently experiencing an outage of GitHub products and are investigating. >Jun 29, 2023 - 17:52 UTC
- jachee 3y agoAs the person in charge of one such page, I’d like to take the opportunity to remind folks that many— if not most—of these status pages are hand-updated, and most bosses absolutely hate anyone having to update them to anything but green.
- salawat 3y agoSounds like an anti-pattern or SLA dodge to me.
- berniedurfee 3y agoSometimes, but sometimes it’s just a precaution against automatic false alarms causing a huge panic.
- salawat 3y agoSee, I prefer panic. People don't pay enough attention to systems as it is. May just yeet together a bunch of parts and never bother to learn to actually troubleshoot or maintain, or reason through things.
- billy1kaplan 3y agoIt's down! https://twitter.com/githubstatus/status/1674475870931808256?cxt=HHwWgMDT_dm797wuAAAA https://twitter.com/githubstatus/status/1674475870931808256?...
- facu17y 3y agoseems to be a network issue, not a service issue
- lucb1e 3y agoBeing up on the localhost interface doesn't count!
- berniedurfee 3y agoWorks on my machine!
- lucb1e 3y agohttps://codeberg.org https://codeberg.org open source GitHub without Microsoft (it's a German non-profit). You can also host your own lightweight https://forgejo.org https://forgejo.org instance In case anyone questions whether centralizing literally everything onto GitHub is a good idea, at least as a mirror for things you depend on
- escape-big-tech 3y agogitea is also a great self hosted alternative!
- lucb1e 3y agoI think that's what Forgejo forked from (and Gitea, in turn, forked from Gogs). I am not involved so don't know the details, but yeah basically all of these will do. I ran my own in the Gitea era and was happy with it, 10x lighter and easier than gitlab, I expect Forgejo has a similar experience.
- Gordonjcp 3y agoNo it isn't, it's working absolutely fine and has been all afternoon.
- sophacles 3y agogithubstatus.com disagrees. Heres the specific incident: https://www.githubstatus.com/incidents/gqx5l06jjxhp https://www.githubstatus.com/incidents/gqx5l06jjxhp I think I'll believe them when they say it's down, no offense.
- onionisafruit 3y agoI heard from somebody at GitHub that this one will make a good incident report. No other details or estimates for recovery time.
- rk32 3y agoits all down currently
- circuit10 3y agoIt works for me at the moment
- renonce 3y agoNot down for me accessing from Hong Kong. I suspect this is a regional outage.
- ActualHacker 3y agoSince nobody can work, I'll just leave this here: "I must have put a decimal point in the wrong place or something. Shit! I always do that. I always mess up some mundane detail."
- datalus 3y agoAnd just when I was about to get into flow state...
- Xeamek 3y agolooking on the bright side, at least we'll get an interesting post-mortem to read in a day or two.
- lucb1e 3y agoTo all the "same" and "not for me" posters: the very least you could add is a location
- mydriasis 3y agoYes. Can't even pull ;(
- oblongatacur 3y agoI noticed Github's OIDC token changed about a half an hour ago. Security incident?
- klysm 3y agoInteresting observation, but I'd be surprised if that was related to a regional network fault
- Ysx 3y agoThey added a second token on Tuesday: https://github.blog/changelog/2023-06-27-github-actions-update-on-oidc-integration-with-aws/ https://github.blog/changelog/2023-06-27-github-actions-upda...
- berniedurfee 3y agoI never got that memo. Found out when it broke something.
- ralgozino 3y agoI must ask, how did you notice that?!
- gaoshan 3y agoMy team ran some code that crushed our Github actions very shortly before this outage. Nervous laughter around the office that it was us.
- ukrainiancrab 3y agoCannot use oauth2 in algoexpert :/
- hejcloud 3y agoI feel sympathy for all the engs at companies I've implemented CI/CD based on Gh Actions in recent years. It's not like I didn't tell them that Github showed to be somewhat unreliable in the recent years and in contrast to their claim "it's just the build pipeline, not the product" I think it is a horrible incident if you're not able to deploy to production and have barely any ad-hoc backup. I'm always evangelizing Argo or Flux and some self-hosted Gitlab or gitea, but seems like they all prefer to throw their money at Github as of now.
- berniedurfee 3y agoTradeoffs and tolerances need to be considered.
- sparc24 3y agoAzure Strikes Again!
- connorgutman 3y agoUp in Africa (Morocco).
- sdsd 3y agoI am laughing so hard right now. My best friend mocked me for using git.lain.faith to host my code, saying it wasn't reliable. Well, well, well. In the last year GitLain hasn't gone down once. I know he was still right in a way, who knows when git.lain.faith will just disappear. But still. I'm going to send some texts to bother him right now, hahaha.
- Macuyiko 3y agoUpdate We have identified the root cause of the outage and are working toward mitigation Posted 4 minutes ago. Jun 29, 2023 - 18:02 UTC
- deleted 3y ago[deleted]
- saintblasphemer 3y agoLooks like we are back online.
- escape-big-tech 3y agooften people point out how unreliable self-hosted services are, well, hosted services are just as unreliable if not more. this, ladies and gentlemen, is why you should always self host critical infrastructure
- iso1631 3y agoDepends on what you want. If you want uptime, then sure. If you want to be able to blame someone then no. If you are down for 1 hour a year on self hosting, but Office 364 is down 3 days a year, your CEO is going to be more understanding of the Office outage as all his golf buddies have the same problem, and he reads about it in the NYT. But in any case zero downtime is difficult, that's why you need two independent systems. I had a a 500 microsecond outage at the weekend when a circuit failed which caused an business affecting incident, not a big one fortunately, as it was only some singers, but it was still one that was unacceptable -- had it happened at a couple of other events in the last 12 months it would have been far more problematic. Work has started to ensure it doesn't happen next year.
- gwbas1c 3y agoYaay! I just pushed and my new commit showed up in CI!
- ny711 3y agoThis should be a weekly ASK HN; seems to happy pretty frequently at this point
- rvz 3y agoUnsurprising. There is at least one outage with GitHub every single month. [0] [0] https://news.ycombinator.com/item?id=35967921 https://news.ycombinator.com/item?id=35967921
- ughitsaaron 3y agoGood time to grab a beer!
- acim 3y agoI just noticed that artifacts download didn't work although the web site was up. There was some varnish proxy error.
- themusicgod1 3y agoIt shouldn't matter. Nobody should be using github post-2018.
- deleted 3y ago[deleted]
- deleted 3y ago[deleted]
- squalo 3y agoNot a single day passes without a MAJOR outage in a Microsoft owned service.