4 ms·
The status page links to https://gitlab.com/gitlab-com/gl-infra/production/-/issues/15997 https://gitlab.com/gitlab-com/gl-infra/production/-/issues/1... for de
by freedomben 3y ago
The status page links to https://gitlab.com/gitlab-com/gl-infra/production/-/issues/15997 https://gitlab.com/gitlab-com/gl-infra/production/-/issues/1... for details, but that 503s because it's on gitlab.com
- dnsmichi 3y agoThanks for your feedback. GitLab team member here. The incident review is updated in https://gitlab.com/gitlab-com/gl-infra/production/-/issues/15999 https://gitlab.com/gitlab-com/gl-infra/production/-/issues/1... I have added your feedback into https://gitlab.com/gitlab-com/gl-infra/production/-/issues/15999#note_1463632220 https://gitlab.com/gitlab-com/gl-infra/production/-/issues/1...
- mdaniel 3y ago> Components: Website, API, Git Operations, Container Registry, GitLab Pages, CI/CD - GitLab SaaS Shared Runners, CI/CD - GitLab SaaS Private Runners, CI/CD - Windows Shared Runners (Beta), SAML SSO - GitLab SaaS, Background Processing, GitLab Customers Portal, Support Services, packages.gitlab.com, version.gitlab.com, forum.gitlab.com, docs.gitlab.com, Canary > Locations: Google Compute Engine, Digital Ocean, Zendesk, AWS that's some _impressive_ blast radius and I actually struggle to think what would cause that much damage
- junon 3y agoMight be a domain issue or a routing misconfiguration. But people are mentioning 5xx errors so it could also be FELB/NLB misconfiguration.
- rewmie 3y ago> that's some _impressive_ blast radius Most of the entries look like batch job processing, either by gitlab or managing jobs in runners. Perhaps authentication, which is mandatory when managing batch jobs including in third party nodes. If auth goes down on any service, you'll see similar blast radius.
- trashburger 3y agoWell, consider the fact that "most of the entries" comprise 10% of GitLab and things like "Git Operations" comprise the remaining 90%. This is a total system failure.
- jtriangle 3y agoDNS, obviously.
- mdaniel 3y agoAccording to the title of the issue (2023-07-07: Blackbox probes for https://cdn.artifacts.gitlab-static.net/cdn-test https://cdn.artifacts.gitlab-static.net/cdn-test are failing) you win the game: $ host cdn.artifacts.gitlab-static.net Host cdn.artifacts.gitlab-static.net not found: 3(NXDOMAIN) They must have used that same healthcheck for every single load-balancer or something, though, for it to nuke AWS, DO, and GCP