5 ms·
Gitlab.com is down
- freedomben 3y agoThe status page links to https://gitlab.com/gitlab-com/gl-infra/production/-/issues/15997 https://gitlab.com/gitlab-com/gl-infra/production/-/issues/1... for details, but that 503s because it's on gitlab.com
- dnsmichi 3y agoThanks for your feedback. GitLab team member here. The incident review is updated in https://gitlab.com/gitlab-com/gl-infra/production/-/issues/15999 https://gitlab.com/gitlab-com/gl-infra/production/-/issues/1... I have added your feedback into https://gitlab.com/gitlab-com/gl-infra/production/-/issues/15999#note_1463632220 https://gitlab.com/gitlab-com/gl-infra/production/-/issues/1...
- mdaniel 3y ago> Components: Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - GitLab SaaS Shared Runners, CI/CD - GitLab SaaS Private Runners, CI/CD - Windows Shared Runners (Beta), SAML SSO - GitLab SaaS, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, docs.gitlab.com, Canary > Locations: Google Compute Engine, Digital Ocean, Zendesk, AWS that's some _impressive_ blast radius and I actually struggle to think what would cause that much damage
- junon 3y agoMight be a domain issue or a routing misconfiguration. But people are mentioning 5xx errors so it could also be FELB/NLB misconfiguration.
- rewmie 3y ago> that's some _impressive_ blast radius Most of the entries look like batch job processing, either by gitlab or managing jobs in runners. Perhaps authentication, which is mandatory when managing batch jobs including in third party nodes. If auth goes down on any service, you'll see similar blast radius.
- trashburger 3y agoWell, consider the fact that "most of the entries" comprise 10% of GitLab and things like "Git Operations" comprise the remaining 90%. This is a total system failure.
- jtriangle 3y agoDNS, obviously.
- mdaniel 3y agoAccording to the title of the issue (2023-07-07: Blackbox probes for https://cdn.artifacts.gitlab-static.net/cdn-test https://cdn.artifacts.gitlab-static.net/cdn-test are failing) you win the game: $ host cdn.artifacts.gitlab-static.net Host cdn.artifacts.gitlab-static.net not found: 3(NXDOMAIN) They must have used that same healthcheck for every single load-balancer or something, though, for it to nuke AWS, DO, and GCP
- av10 3y ago[flagged]
- deleted 3y ago[deleted]
- cratermoon 3y agoThis also affects Gitlab Pages. If your site is hosted there, your visitors are getting 503s.
- darau1 3y agoMy site seems unaffected.
- cratermoon 3y agoIt may be affecting authorization, then. I have a site that's private and requires auth and I can't reach it.
- jcbcn 3y agoMy GitLab Pages site is working but I'm getting 503s when attempting to push to the repo and browsing the GitLab site
- ironSkillet 3y agoTheir status page is not very helpful because they just link to a tracker that is also returning a 503. Does anyone know what's going on under the hood?
- dnsmichi 3y agoGitLab team member here. Apologies for the problem. The incident review issue continues in https://gitlab.com/gitlab-com/gl-infra/production/-/issues/15999 https://gitlab.com/gitlab-com/gl-infra/production/-/issues/1...
- mholt 3y agoWelp, time to migrate everything to GitHub
- the_common_man 3y agoYou have to wait till GitHub is up
- JoshuaRogers 3y agoThat’s why I store all my code on Twitter.
- tomschwiha 3y agoI thought we should use the Blockchain.
- chakintosh 3y agoYou mean Threads*
- diggerdata 3y agoNah I print all my code.
- xeromal 3y agoLucky for you, I have a shitty, not-well-tested tool to do that. https://github.com/tylerjgarland/git2git https://github.com/tylerjgarland/git2git lol. I used it to migrate my gitlab to github on the last outage.
- kokanee 3y agoI'm seeing a mix of 503s, 522s, 500s, and once in a while a successful page load on the website. No word on the root cause at this point.
- trashburger 3y agoI hope they didn't delete the production database again.
- kunwon1 3y agoWhen that incident happened, Gitlab published an actual chat transcript of the incident response. It was a very interesting read. It had the real names of the engineers involved for the first 24 hours or so, then they anonymized it. At some later date they seem to have removed it entirely, which is a shame, as it was an educational read All that I can find left online is this [1], which is still informative, but not nearly as interesting as I remember the chat transcript being [1] https://about.gitlab.com/blog/2017/02/10/postmortem-of-database-outage-of-january-31/ https://about.gitlab.com/blog/2017/02/10/postmortem-of-datab...
- ironSkillet 3y agoI thought the parent post was a joke, and then I read your link...
- jonatron 3y agohttps://web.archive.org/web/20170527000404/https://docs.google.com/document/d/1GCK53YDcBWQveod9kfzW-VCxIABGiryG7_z_6jHdVik/pub https://web.archive.org/web/20170527000404/https://docs.goog...
- rbancroft 3y agoThat's pretty cool. I notice how MS has become so much more respected as they have embraced open source. Is the next frontier for companies to become open process where the best companies attract the best people because of their open processes?
- sim7c00 3y agoi think ultimately this will not be entirely possible. if a company posts transcripts with real names of people that can be damaging for the people (they moght misbehave simply due to company culture, pressure not visible through the open process etc.) which then would give a company a cheap way out (emplyee scapegoat). since this is a possibility, i think employees amd unions etc. would resist. despite if it was an honest afair it would be awesome, and that (possible but improbable) future looks super cool :)
- pplanel 3y agoGKE Related? https://status.cloud.google.com/ https://status.cloud.google.com/
- WrtCdEvrydy 3y agoJesus, this is worse than AWS... had to scroll a while to see the one service was broken.
- gitfan86 3y agoThat is just some extra logging, it has been happening to one of my clusters for weeks
- deleted 3y ago[deleted]
- classified 3y agoGood thing I'm off for the weekend.
- berkle4455 3y agoIf only there was a way to distribute version control and not rely on a central entity
- freedomben 3y agoI get that this is sarcasm, but Gitlab is a lot more than just version control. CI/CD, project management, even hosting (Gitlab pages). But even for just version control for most use cases you need some common remote for people to push and pull to. Even a bare git setup has the potential of going down and putting people in the exact same boat that they are in now (they can commit locally, but can't push).
- zzbn00 3y agoThere is the git send-email & co. I understand the extreme convenience of things like github and gitlab but as everything moves onto them it does seem we are becoming really dependent on a few sites.
- delfinom 3y agoHave you tried to run your own mail infrastructure in 2023? It's a nightmare from all the junk mail handling schemes of every individual email provider to just dealing with antiquated software for the mailing list and mailer daemons.
- qD67ClSDh3XOgB 3y agoI'm not sure why you think you would need to "run your own mail infrastructure" to use git send-email, but you don't.
- berkle4455 3y agoI know it is, which imo is the problem. You shouldn’t have to rely on an additional third-party for your company to be able to commit, checkout, build, and deploy code. All these startups who experience a 100% dev outage when github or gitlab go down are fundamentally broken.
- johnea 3y agoJust another reason to self host...
- ndsipa_pomu 3y agoYeah, I self host a couple of gitlab-ce instances in Docker (one personal, one for work) and can recommend doing so. However, that's just changing where the points of failure are and who gets to do the work when there's failures.
- rewmie 3y ago> However, that's just changing where the points of failure are and who gets to do the work when there's failures. ...and more importantly who gets to work so that there isn't a failure.
- pjot 3y agoConsidering Gitlab employees use Gitlab to make changes to Gitlab, how will they fix Gitlab?
- dnsmichi 3y agoGitLab team member here. Thanks for asking. Incidents can have different types, i.e. when an application bug or performance regression is discovered, this can involve reverting MRs and rolling back releases. The Platform, Delivery group has a top-level responsibility for ensuring continuous delivery of the GitLab application software to GitLab SaaS, https://about.gitlab.com/handbook/engineering/infrastructure/team/delivery/#top-level-responsibilities https://about.gitlab.com/handbook/engineering/infrastructure... Other incidents may involve hardware or infrastructure failures, or a combination of both, infrastructure failure that renders GitLab application services unavailable. This requires cross-functional collaboration from infrastructure, product, engineering, etc. teams in the incident. To get a better understanding here, it is helpful to review the incident management handbook https://about.gitlab.com/handbook/engineering/infrastructure/incident-management/ https://about.gitlab.com/handbook/engineering/infrastructure... Additional helpful information: - The GitLab.com SaaS production architecture is documented in https://about.gitlab.com/handbook/engineering/infrastructure/production/architecture/#gitlab-com-architecture https://about.gitlab.com/handbook/engineering/infrastructure... - The Monitoring of GitLab.com handbook provides insights into monitoring workflows, incident management, SLAs, etc. https://about.gitlab.com/handbook/engineering/monitoring/ https://about.gitlab.com/handbook/engineering/monitoring/ - Runbooks https://about.gitlab.com/handbook/engineering/infrastructure/incident-management/#runbooks https://about.gitlab.com/handbook/engineering/infrastructure... For the current incident discussed in this HN thread, the review issue can be followed in https://gitlab.com/gitlab-com/gl-infra/production/-/issues/15999 https://gitlab.com/gitlab-com/gl-infra/production/-/issues/1... to learn more.
- panzi 3y agoIt's git, so everyone has a copy of the repo. You can continue collaborating with send-email or some random SSH server. That's the beauty of a DSCM.
- 3y ago
- pid-1 3y agoWell, those incidents are becoming rather frequent.
- reganhoss 3y ago[flagged]
- reganhoss 3y ago[flagged]
- deleted 3y ago[deleted]
- nickpeterson 3y ago[flagged]
- cloud9actual 3y ago[flagged]
- mdaniel 3y agoin case folks found the relevant submission via https://news.ycombinator.com/from?site=status.gitlab.com https://news.ycombinator.com/from?site=status.gitlab.com this is the url-less submission: https://news.ycombinator.com/item?id=36634089 https://news.ycombinator.com/item?id=36634089
- revskill 3y agoJust a normal day with microservice.
- rewmie 3y agoMonoliths crashing and taking everything down with them isn't news, right?
- revskill 3y agoNot quite, it's 100x harder to find out where is the cause from with microservice.
- openthc 3y agoIME it's one microservice doing something unexpected -- then down(|up)-stream starts to act unexpected -- and it cascades. It's way more fun to debug than some single-monster code base with a complete stack-trace from the single-process. That basically tells you exactly where to look and how it got there! What fun is that, where's the adventure!?
- delfinom 3y agoGitLab isn't really a microservice architecture.
- dnsmichi 3y agoGitLab team member here. Next to all-in-one packages provided with the Omnibus package on Linux https://docs.gitlab.com/omnibus/ https://docs.gitlab.com/omnibus/ and in containers https://docs.gitlab.com/ee/install/docker.html https://docs.gitlab.com/ee/install/docker.html you can also install GitLab into Kubernetes clusters using the Operator https://docs.gitlab.com/operator/ https://docs.gitlab.com/operator/ or Helm Chart. https://docs.gitlab.com/charts/ https://docs.gitlab.com/charts/ Most of GitLab.com SaaS is deployed on Kubernetes using the GitLab cloud-native Helm chart (with a few exceptions). More details in https://about.gitlab.com/handbook/engineering/infrastructure/production/architecture/#gitlab-com-architecture https://about.gitlab.com/handbook/engineering/infrastructure...
- openthc 3y agoStatus page is hosted by status.io behind cloudfront.net(amazon.com)
- jnwatson 3y agoAt least their status page works.
- I_am_tiberius 3y agoI can't say anything bad about gitlab. Incidents happen and I really like that they are transparent about it.
- programshall 3y agoLooks like their infrastructure team is hiring site reliability engineers. https://about.gitlab.com/jobs/all-jobs/ https://about.gitlab.com/jobs/all-jobs/
- TheLoafOfBread 3y agoWhat a difference. GitLab is down -> They will tell on their status page that they have problems. Unlike other cloud providers which when are obviously down, they won't change anything on status page.
- openthc 3y agoGitLab appears to be back online (for me anyway). Status page still shows red tho.
- freedomben 3y agoYep the web UI is back up and the remote is accepting pushes/pulls now. I'm guessing the major bleeding has been stopped and they have a massive cleanup job to do over the next few days or weeks.
- deathbypenguin 3y agoI'm still getting... remote: ERROR: The git server, Gitaly, is not available at this time. What a drag... I guess I'll just commit locally... bah! Friday anyway!
- jjgreen 3y agoApproaching wine o'clock in my time zone ...
- deathbypenguin 3y agoI know! I can feel it coming as well... just riding that last hour at the office.
- jjgreen 3y agoCI jobs are still failing ... Fetching changes... Initialized empty Git repository in /builds/jjg/cptutils/.git/ Created fresh repository. fatal: unable to access 'https://git-us-east1-d.ci gateway.int.gprd.gitlab.net:8989/jjg/cptutils.git/': Failed to connect to git-us-east1-d.ci-gateway.int.gprd.gitlab.net port 8989 after 132297 ms: Couldn't connect to server