4 ms·

CIs + runners + embedded registry is a really nice add-on that I couldn't do without

by Faaak 5y ago

KronisLV 5y ago
Agreed! However the amount of maintenance that you need to do because of the vulnerabilities is disappointing, especially due to how contrived the upgrade paths are: https://docs.gitlab.com/ee/update/#upgrade-paths https://docs.gitlab.com/ee/update/#upgrade-paths Thus, it might be a better idea to look into a less popular stack that's less likely to be targeted as much due to not being such a juicy target. For example: Code: Gitea/Gogs/GitBucket CI: Drone/Jenkins (okay there are probably better options than Jenkins, to be honest) Registry: Nexus/Artifactory (not just for containers, they support most formats and have better control over cleanup of old data so you don't have to schedule GitLab cleanup yourself) Of course, at the end of the day all of those still have an attacks surface, so i'm really leaning more and more into the camp of exposing nothing publicly since it's a losing battle.
- tenken 5y ago
  What maintenance?! I just bump the docker-compose.yml version numbers and Stop/Start the service. It's very painless... My cellphone has more frequent updates than Gitlab does.
  - KronisLV 5y ago
    If you do that with minor versions, you should generally be fine. When you need to upgrade across major versions, you'll most likely be met with the following in case you haven't followed the updates closely: > It seems you are upgrading from major version X to major version Y. > It is required to upgrade to the latest Y.0.x version first before proceeding. > Please follow the upgrade documentation at https://docs.gitlab.com/ee/update/index.html#upgrading-to-a-new-major-version https://docs.gitlab.com/ee/update/index.html#upgrading-to-a-... In addition to that, you should NEVER just bump versions without having backups (which you've hopefully considered), so there is probably another step in there, either validating that your latest automatic backups work, or even just manually copying the current GitLab data directory into another folder, in the case of an Omnibus install, or doing the same manually for all components in the more distributed installation type. Disclaimer: this has little do to with GitLab in particular but is something you should consider with any and all software packages that you upgrade, especially the kind with non-trivial dependencies and data storage mechanisms, like PostgreSQL. Of course, you can always dump the DB but it's easier to back up everything else as well by taking the instances offline and making data copies of all container volumes/bind mounts.
    - tenken 5y ago
      * I never skip (minor) versions. * I have automated backups created every 2 days stored via S3, I've done a full restore twice in 5+ years of uptime. * I run Gitlab at home and at work. None of the points touch on a maintenance burden ... Just saying. Skipping versions while updating any software is just being a lazy sysadmin and praying it works. Typically skipping to major versions during upgrades always comes with breaking changes so operator beware.
      - KronisLV 5y ago
        > Skipping versions while updating any software is just being a lazy sysadmin and praying it works. Typically skipping to major versions during upgrades always comes with breaking changes so operator beware. It's nice that GitLab actually prevent you from doing that and give you messages that it's unsupported and direct you to their documentation, which describes the supported upgrade paths... However, at the same time one cannot help but to wonder about why you can't go from version #1 to version #999 in one go. Most of the software at my dayjob (at least the one that i've written) absolutely can do that - since the DB migrations are fully automated, even if i had to create a lot of pushback against other devs going: "Well, it would be easier just to tell the clients who are running this software to just do X manually when going from version Y to Z." But GitLab's updates are largely automated (the Chef scripts and PostgreSQL migrations etc.), it's just that for some reason they either don't include all of them or require that you visit certain key points throughout the process, which cannot be skipped (e.g. certain milestone releases, as described in their docs). Of course, i acknowledge that it's extremely hard to sustain backwards compatibility and i've seen numerous projects start out that way and the devs give up on the idea at first sign of difficulty, since it's not like they care much for that and it doesn't always lead to clear value add - it's a nice to have and they won't earn any less of a salary for making some ops' person's life harder down the line. I also have automated backups with BackupPC, however i expect software to remain reasonably secure and stable without having to update that often - props to GitLab for disclosing the important releases, but i'm migrating over to Gitea for my personal needs as we speak, even if having someone else manage a GitLab install at work is still like having a superpower (with GitLab CI, GitLab Registry etc.). I actually wrote an article about how really frequent updates cause problems and lots of churn: https://blog.kronis.dev/articles/never-update-anything https://blog.kronis.dev/articles/never-update-anything (though the title is a bit tongue in cheek, as explained by the disclaimer at the top of the article).
- dnsmichi 5y ago
  > how contrived the upgrade paths are: https://docs.gitlab.com/ee/update/#upgrade-paths https://docs.gitlab.com/ee/update/#upgrade-paths Thanks for your feedback, agreed. I've created an issue https://gitlab.com/gitlab-org/gitlab/-/issues/353862 https://gitlab.com/gitlab-org/gitlab/-/issues/353862 - please add additional thoughts and suggestions there as well. Thanks!
  - KronisLV 5y ago
    Well, i don't think that there's actually that much that can be done here, since that page does contain adequate documentation and a linear example migration path: 8.11.Z -> 8.12.0 -> 8.17.7 -> 9.5.10 -> 10.8.7 -> 11.11.8 -> 12.0.12 -> 12.1.17 -> 12.10.14 -> 13.0.14 -> 13.1.11 -> 13.8.8 -> 13.12.15 -> 14.0.12 -> latest 14.Y.Z It's just that the process itself is troublesome, in that you can'd just go from let's say 11.11.8 to 14.6.5 in one go and let all of the thousands of changes and migrations be applied automatically with no problems, as some software out there attempts to (with varying degrees of success). Of course, it's probably not viable due to the significant changes that the actual application undergoes and therefore one just needs to bite the bullet and deal with either constant small updates or few and longer updates for private instances. But hey, thanks for creating the issue and best of luck!
forty 5y ago
Side question: is the gitlab docker registry really that useful? The fact that any non-protected runner can push to it makes it useless to store images to be used in other CI pipeline (unless I missed something, which I would be really glad)
- mhio 5y ago
  The registry is fine for our use case, which is to manage all the dev artefacts on a private repo before CI might push a release to a production registry. > any non-protected runner can push to it My understanding is the job token inherits its permission set from the user causing the job to run. If the user has `write_registry` to a project (developer up), then the job does. Do you see more access than that? The access can be limited per project to specific projects by setting a scope [0] but your description sounds like it might access within the project that is the issue. 0: https://gitlab.sam3.io/help/ci/jobs/ci_job_token#configure-the-job-token-scope-limit https://gitlab.sam3.io/help/ci/jobs/ci_job_token#configure-t...
  - forty 5y ago
    The whole point of having protected runners is to do things that developers are not allowed to do. If any developer can push images to the registry without any review/approval, and those images are used in other CI pipeline that's a problem for us. Having a separate production registry is good indeed, but for images to be used for CI itself, having something self-contained within gitlab would have been nice.

CIs + runners + embedded registry is a really nice add-on that I couldn't do without

by Faaak 5y ago

CIs + runners + embedded registry is a really nice add-on that I couldn't do without

Agreed! However the amount of maintenance that you need to do because of the vulnerabilities is disappointing, especially due to how contrived the upgrade paths are: https://docs.gitlab.com/ee/update/#upgrade-paths https://docs.gitlab.com/ee/update/#upgrade-paths Thus, it might be a better idea to look into a less popular stack that's less likely to be targeted as much due to not being such a juicy target. For example: Code: Gitea/Gogs/GitBucket CI: Drone/Jenkins (okay there are probably better options than Jenkins, to be honest) Registry: Nexus/Artifactory (not just for containers, they support most formats and have better control over cleanup of old data so you don't have to schedule GitLab cleanup yourself) Of course, at the end of the day all of those still have an attacks surface, so i'm really leaning more and more into the camp of exposing nothing publicly since it's a losing battle.

tenken 5y ago
What maintenance?! I just bump the docker-compose.yml version numbers and Stop/Start the service. It's very painless... My cellphone has more frequent updates than Gitlab does.
- KronisLV 5y ago
  If you do that with minor versions, you should generally be fine. When you need to upgrade across major versions, you'll most likely be met with the following in case you haven't followed the updates closely: > It seems you are upgrading from major version X to major version Y. > It is required to upgrade to the latest Y.0.x version first before proceeding. > Please follow the upgrade documentation at https://docs.gitlab.com/ee/update/index.html#upgrading-to-a-new-major-version https://docs.gitlab.com/ee/update/index.html#upgrading-to-a-... In addition to that, you should NEVER just bump versions without having backups (which you've hopefully considered), so there is probably another step in there, either validating that your latest automatic backups work, or even just manually copying the current GitLab data directory into another folder, in the case of an Omnibus install, or doing the same manually for all components in the more distributed installation type. Disclaimer: this has little do to with GitLab in particular but is something you should consider with any and all software packages that you upgrade, especially the kind with non-trivial dependencies and data storage mechanisms, like PostgreSQL. Of course, you can always dump the DB but it's easier to back up everything else as well by taking the instances offline and making data copies of all container volumes/bind mounts.
  - tenken 5y ago
    * I never skip (minor) versions. * I have automated backups created every 2 days stored via S3, I've done a full restore twice in 5+ years of uptime. * I run Gitlab at home and at work. None of the points touch on a maintenance burden ... Just saying. Skipping versions while updating any software is just being a lazy sysadmin and praying it works. Typically skipping to major versions during upgrades always comes with breaking changes so operator beware.
    - KronisLV 5y ago
      > Skipping versions while updating any software is just being a lazy sysadmin and praying it works. Typically skipping to major versions during upgrades always comes with breaking changes so operator beware. It's nice that GitLab actually prevent you from doing that and give you messages that it's unsupported and direct you to their documentation, which describes the supported upgrade paths... However, at the same time one cannot help but to wonder about why you can't go from version #1 to version #999 in one go. Most of the software at my dayjob (at least the one that i've written) absolutely can do that - since the DB migrations are fully automated, even if i had to create a lot of pushback against other devs going: "Well, it would be easier just to tell the clients who are running this software to just do X manually when going from version Y to Z." But GitLab's updates are largely automated (the Chef scripts and PostgreSQL migrations etc.), it's just that for some reason they either don't include all of them or require that you visit certain key points throughout the process, which cannot be skipped (e.g. certain milestone releases, as described in their docs). Of course, i acknowledge that it's extremely hard to sustain backwards compatibility and i've seen numerous projects start out that way and the devs give up on the idea at first sign of difficulty, since it's not like they care much for that and it doesn't always lead to clear value add - it's a nice to have and they won't earn any less of a salary for making some ops' person's life harder down the line. I also have automated backups with BackupPC, however i expect software to remain reasonably secure and stable without having to update that often - props to GitLab for disclosing the important releases, but i'm migrating over to Gitea for my personal needs as we speak, even if having someone else manage a GitLab install at work is still like having a superpower (with GitLab CI, GitLab Registry etc.). I actually wrote an article about how really frequent updates cause problems and lots of churn: https://blog.kronis.dev/articles/never-update-anything https://blog.kronis.dev/articles/never-update-anything (though the title is a bit tongue in cheek, as explained by the disclaimer at the top of the article).
dnsmichi 5y ago
> how contrived the upgrade paths are: https://docs.gitlab.com/ee/update/#upgrade-paths https://docs.gitlab.com/ee/update/#upgrade-paths Thanks for your feedback, agreed. I've created an issue https://gitlab.com/gitlab-org/gitlab/-/issues/353862 https://gitlab.com/gitlab-org/gitlab/-/issues/353862 - please add additional thoughts and suggestions there as well. Thanks!
- KronisLV 5y ago
  Well, i don't think that there's actually that much that can be done here, since that page does contain adequate documentation and a linear example migration path: 8.11.Z -> 8.12.0 -> 8.17.7 -> 9.5.10 -> 10.8.7 -> 11.11.8 -> 12.0.12 -> 12.1.17 -> 12.10.14 -> 13.0.14 -> 13.1.11 -> 13.8.8 -> 13.12.15 -> 14.0.12 -> latest 14.Y.Z It's just that the process itself is troublesome, in that you can'd just go from let's say 11.11.8 to 14.6.5 in one go and let all of the thousands of changes and migrations be applied automatically with no problems, as some software out there attempts to (with varying degrees of success). Of course, it's probably not viable due to the significant changes that the actual application undergoes and therefore one just needs to bite the bullet and deal with either constant small updates or few and longer updates for private instances. But hey, thanks for creating the issue and best of luck!

forty 5y ago

Side question: is the gitlab docker registry really that useful? The fact that any non-protected runner can push to it makes it useless to store images to be used in other CI pipeline (unless I missed something, which I would be really glad)

mhio 5y ago
The registry is fine for our use case, which is to manage all the dev artefacts on a private repo before CI might push a release to a production registry. > any non-protected runner can push to it My understanding is the job token inherits its permission set from the user causing the job to run. If the user has `write_registry` to a project (developer up), then the job does. Do you see more access than that? The access can be limited per project to specific projects by setting a scope [0] but your description sounds like it might access within the project that is the issue. 0: https://gitlab.sam3.io/help/ci/jobs/ci_job_token#configure-the-job-token-scope-limit https://gitlab.sam3.io/help/ci/jobs/ci_job_token#configure-t...
- forty 5y ago
  The whole point of having protected runners is to do things that developers are not allowed to do. If any developer can push images to the registry without any review/approval, and those images are used in other CI pipeline that's a problem for us. Having a separate production registry is good indeed, but for images to be used for CI itself, having something self-contained within gitlab would have been nice.