3 ms·

Considering Gitlab employees use Gitlab to make changes to Gitlab, how will they fix Gitlab?

by pjot 3y ago

dnsmichi 3y ago
GitLab team member here. Thanks for asking. Incidents can have different types, i.e. when an application bug or performance regression is discovered, this can involve reverting MRs and rolling back releases. The Platform, Delivery group has a top-level responsibility for ensuring continuous delivery of the GitLab application software to GitLab SaaS, https://about.gitlab.com/handbook/engineering/infrastructure/team/delivery/#top-level-responsibilities https://about.gitlab.com/handbook/engineering/infrastructure... Other incidents may involve hardware or infrastructure failures, or a combination of both, infrastructure failure that renders GitLab application services unavailable. This requires cross-functional collaboration from infrastructure, product, engineering, etc. teams in the incident. To get a better understanding here, it is helpful to review the incident management handbook https://about.gitlab.com/handbook/engineering/infrastructure/incident-management/ https://about.gitlab.com/handbook/engineering/infrastructure... Additional helpful information: - The GitLab.com SaaS production architecture is documented in https://about.gitlab.com/handbook/engineering/infrastructure/production/architecture/#gitlab-com-architecture https://about.gitlab.com/handbook/engineering/infrastructure... - The Monitoring of GitLab.com handbook provides insights into monitoring workflows, incident management, SLAs, etc. https://about.gitlab.com/handbook/engineering/monitoring/ https://about.gitlab.com/handbook/engineering/monitoring/ - Runbooks https://about.gitlab.com/handbook/engineering/infrastructure/incident-management/#runbooks https://about.gitlab.com/handbook/engineering/infrastructure... For the current incident discussed in this HN thread, the review issue can be followed in https://gitlab.com/gitlab-com/gl-infra/production/-/issues/15999 https://gitlab.com/gitlab-com/gl-infra/production/-/issues/1... to learn more.
panzi 3y ago
It's git, so everyone has a copy of the repo. You can continue collaborating with send-email or some random SSH server. That's the beauty of a DSCM.
- dnsmichi 3y ago
  GitLab team member here. If you plan on a GitLab mirror, including the Git repository and groups/projects, the direct transfer migration can be helpful https://about.gitlab.com/blog/2023/01/18/try-out-new-way-to-migrate-projects/ https://about.gitlab.com/blog/2023/01/18/try-out-new-way-to-... GitLab Geo can be an alternative for larger, high-availability setup requirements. https://docs.gitlab.com/ee/administration/reference_architectures/#high-availability-ha https://docs.gitlab.com/ee/administration/reference_architec...
- theta_d 3y ago
  Yeah, but they also use the Pipeline features to build and deploy changes.
  - hakre 3y ago
    But they are clever, you can have the runner local. (They start to ruin that, but lets say, the idea is still visible.)
    - mdaniel 3y ago
      Only for `script: "echo hello world"` style jobs; anything more real makes `gitlab-runner exec` a total farce
    - slowmovintarget 3y ago
      If by local you mean local to a cloud instance that runs automated builds on every merge for specific branches, then yes. Granted, those build instance pull from the central git repo, and are triggered from GitLab... so also, no.
      - hakre 3y ago
        local like in tree.
    - IshKebab 3y ago
      Yeah you can't really. It doesn't support a ton of syntax and features. And they also said they were going to remove that ability too. Tbf to them after the inevitable response they did say they will come up with a proper solution, though that was some time ago. Obviously the real solution is to use Bazel and not to use Gitlab CI as a crap build system.
jpraczyk 3y ago
I think it gets deployed from https://ops.gitlab.net/ https://ops.gitlab.net/
grepfru_it 3y ago
At GitHub, we (eventually) ran a completely separate GitHub instance and would deploy there first. We had tooling which allowed us to deploy manually to production from that backup instance in the event we made a breaking change.

Considering Gitlab employees use Gitlab to make changes to Gitlab, how will they fix Gitlab?

by pjot 3y ago

Considering Gitlab employees use Gitlab to make changes to Gitlab, how will they fix Gitlab?

GitLab team member here. Thanks for asking. Incidents can have different types, i.e. when an application bug or performance regression is discovered, this can involve reverting MRs and rolling back releases. The Platform, Delivery group has a top-level responsibility for ensuring continuous delivery of the GitLab application software to GitLab SaaS, https://about.gitlab.com/handbook/engineering/infrastructure/team/delivery/#top-level-responsibilities https://about.gitlab.com/handbook/engineering/infrastructure... Other incidents may involve hardware or infrastructure failures, or a combination of both, infrastructure failure that renders GitLab application services unavailable. This requires cross-functional collaboration from infrastructure, product, engineering, etc. teams in the incident. To get a better understanding here, it is helpful to review the incident management handbook https://about.gitlab.com/handbook/engineering/infrastructure/incident-management/ https://about.gitlab.com/handbook/engineering/infrastructure... Additional helpful information: - The GitLab.com SaaS production architecture is documented in https://about.gitlab.com/handbook/engineering/infrastructure/production/architecture/#gitlab-com-architecture https://about.gitlab.com/handbook/engineering/infrastructure... - The Monitoring of GitLab.com handbook provides insights into monitoring workflows, incident management, SLAs, etc. https://about.gitlab.com/handbook/engineering/monitoring/ https://about.gitlab.com/handbook/engineering/monitoring/ - Runbooks https://about.gitlab.com/handbook/engineering/infrastructure/incident-management/#runbooks https://about.gitlab.com/handbook/engineering/infrastructure... For the current incident discussed in this HN thread, the review issue can be followed in https://gitlab.com/gitlab-com/gl-infra/production/-/issues/15999 https://gitlab.com/gitlab-com/gl-infra/production/-/issues/1... to learn more.

panzi 3y ago

It's git, so everyone has a copy of the repo. You can continue collaborating with send-email or some random SSH server. That's the beauty of a DSCM.

dnsmichi 3y ago
GitLab team member here. If you plan on a GitLab mirror, including the Git repository and groups/projects, the direct transfer migration can be helpful https://about.gitlab.com/blog/2023/01/18/try-out-new-way-to-migrate-projects/ https://about.gitlab.com/blog/2023/01/18/try-out-new-way-to-... GitLab Geo can be an alternative for larger, high-availability setup requirements. https://docs.gitlab.com/ee/administration/reference_architectures/#high-availability-ha https://docs.gitlab.com/ee/administration/reference_architec...
theta_d 3y ago
Yeah, but they also use the Pipeline features to build and deploy changes.
- hakre 3y ago
  But they are clever, you can have the runner local. (They start to ruin that, but lets say, the idea is still visible.)
  - mdaniel 3y ago
    Only for `script: "echo hello world"` style jobs; anything more real makes `gitlab-runner exec` a total farce
  - slowmovintarget 3y ago
    If by local you mean local to a cloud instance that runs automated builds on every merge for specific branches, then yes. Granted, those build instance pull from the central git repo, and are triggered from GitLab... so also, no.
    - hakre 3y ago
      local like in tree.
  - IshKebab 3y ago
    Yeah you can't really. It doesn't support a ton of syntax and features. And they also said they were going to remove that ability too. Tbf to them after the inevitable response they did say they will come up with a proper solution, though that was some time ago. Obviously the real solution is to use Bazel and not to use Gitlab CI as a crap build system.

jpraczyk 3y ago

I think it gets deployed from https://ops.gitlab.net/ https://ops.gitlab.net/

grepfru_it 3y ago

At GitHub, we (eventually) ran a completely separate GitHub instance and would deploy there first. We had tooling which allowed us to deploy manually to production from that backup instance in the event we made a breaking change.