3 ms·

Ah ok sounds like a good setup, I could never get past the slow image pull times for systems like that though? Maybe I missed a setting. Containerd especially w

Ah ok sounds like a good setup, I could never get past the slow image pull times for systems like that though? Maybe I missed a setting. Containerd especially was quite slow at pulling images. Image of 2GB~ could take a couple of minutes to come down, order of magnitude more than straight s3 download. Once you’ve got your images fine, but if your build nodes are elastic you can easily hit cold/new nodes and pay the penalty.

KronisLV 5y ago
> Once you’ve got your images fine, but if your build nodes are elastic you can easily hit cold/new nodes and pay the penalty. If you have your own Nexus or Artifactory, or even just a registry that's configured as a pull through cache (https://docs.docker.com/registry/recipes/mirror/ https://docs.docker.com/registry/recipes/mirror/), then it shouldn't be a problem. If those two servers are in an internal network, even better, otherwise just get a VPS with a good port speed. Honestly, even 100 Mbps should be enough for 2 GB, taking less than 30 seconds, with 1 Gbps that time becomes 2 seconds: https://techinternets.com/copy_calc https://techinternets.com/copy_calc The difference here would be that you're using your own software on your own servers and therefore can deal with the load that you generate yourself with the full allotment of resources that you have, vs having to rely on public registries that are used by tens of thousands of other developers/processes at the same time. That's not that different from S3 either (apart from maybe more capacity being available to you on private buckets, depending on the vendor), since you can also use something like MinIO or Zenko on your own servers as well instead of relying on the cloud.
dnsmichi 5y ago
The dependency proxy in GitLab can help with caching the Docker images. https://docs.gitlab.com/ee/user/packages/dependency_proxy/ https://docs.gitlab.com/ee/user/packages/dependency_proxy/ When Docker Hub introduced rate limits last year, one way to mitigate its impact was to make the dependency proxy available for everyone. https://about.gitlab.com/blog/2020/10/30/mitigating-the-impact-of-docker-hub-pull-requests-limits/ https://about.gitlab.com/blog/2020/10/30/mitigating-the-impa... https://about.gitlab.com/blog/2020/10/30/minor-breaking-change-dependency-proxy/ https://about.gitlab.com/blog/2020/10/30/minor-breaking-chan... Another way can be maintaining your own group which maintains all base images for your environment, and stores them in the GitLab package registry. Using your own images can help enforce security policies, to avoid that containers introduce vulnerabilities (e.g. when always pulling :latest tag, or sticking to old tags). Reminds me of Infrastructure as Code security scanning, which can help detect things like this. Played with it when released in GitLab 14.5, examples in https://gitlab.com/gitlab-de/playground/infrastructure-as-code-scanning https://gitlab.com/gitlab-de/playground/infrastructure-as-co... + https://gitlab.com/gitlab-de/playground/infrastructure-as-code-scanning/-/blob/main/Dockerfile#L2 https://gitlab.com/gitlab-de/playground/infrastructure-as-co... Depending on the languages and environments involved, builder images might be needed to reduce complexity with installing build tools. Like, C++ with gcc and controlling the exact version being used (major versions may introduce ABI breaking changes, I've seen it with armhf and gcc6 a while ago). Builder images also reduce the possibly for users to make mistakes in CI/CD before_script/script sections. An optimized pipeline may just include a remote CI/CD template with the magic happening in central maintained projects, using the builder images. Another thought on builder images - multi arch images with buildx. I've read https://medium.com/@tomwillfixit/migrating-a-dockerized-gitlab-chat-bot-to-arm-in-an-afternoon-9324dc43480d https://medium.com/@tomwillfixit/migrating-a-dockerized-gitl... yesterday, need to learn more :)

KronisLV 5y ago

> Once you’ve got your images fine, but if your build nodes are elastic you can easily hit cold/new nodes and pay the penalty. If you have your own Nexus or Artifactory, or even just a registry that's configured as a pull through cache (https://docs.docker.com/registry/recipes/mirror/ https://docs.docker.com/registry/recipes/mirror/), then it shouldn't be a problem. If those two servers are in an internal network, even better, otherwise just get a VPS with a good port speed. Honestly, even 100 Mbps should be enough for 2 GB, taking less than 30 seconds, with 1 Gbps that time becomes 2 seconds: https://techinternets.com/copy_calc https://techinternets.com/copy_calc The difference here would be that you're using your own software on your own servers and therefore can deal with the load that you generate yourself with the full allotment of resources that you have, vs having to rely on public registries that are used by tens of thousands of other developers/processes at the same time. That's not that different from S3 either (apart from maybe more capacity being available to you on private buckets, depending on the vendor), since you can also use something like MinIO or Zenko on your own servers as well instead of relying on the cloud.

dnsmichi 5y ago

The dependency proxy in GitLab can help with caching the Docker images. https://docs.gitlab.com/ee/user/packages/dependency_proxy/ https://docs.gitlab.com/ee/user/packages/dependency_proxy/ When Docker Hub introduced rate limits last year, one way to mitigate its impact was to make the dependency proxy available for everyone. https://about.gitlab.com/blog/2020/10/30/mitigating-the-impact-of-docker-hub-pull-requests-limits/ https://about.gitlab.com/blog/2020/10/30/mitigating-the-impa... https://about.gitlab.com/blog/2020/10/30/minor-breaking-change-dependency-proxy/ https://about.gitlab.com/blog/2020/10/30/minor-breaking-chan... Another way can be maintaining your own group which maintains all base images for your environment, and stores them in the GitLab package registry. Using your own images can help enforce security policies, to avoid that containers introduce vulnerabilities (e.g. when always pulling :latest tag, or sticking to old tags). Reminds me of Infrastructure as Code security scanning, which can help detect things like this. Played with it when released in GitLab 14.5, examples in https://gitlab.com/gitlab-de/playground/infrastructure-as-code-scanning https://gitlab.com/gitlab-de/playground/infrastructure-as-co... + https://gitlab.com/gitlab-de/playground/infrastructure-as-code-scanning/-/blob/main/Dockerfile#L2 https://gitlab.com/gitlab-de/playground/infrastructure-as-co... Depending on the languages and environments involved, builder images might be needed to reduce complexity with installing build tools. Like, C++ with gcc and controlling the exact version being used (major versions may introduce ABI breaking changes, I've seen it with armhf and gcc6 a while ago). Builder images also reduce the possibly for users to make mistakes in CI/CD before_script/script sections. An optimized pipeline may just include a remote CI/CD template with the magic happening in central maintained projects, using the builder images. Another thought on builder images - multi arch images with buildx. I've read https://medium.com/@tomwillfixit/migrating-a-dockerized-gitlab-chat-bot-to-arm-in-an-afternoon-9324dc43480d https://medium.com/@tomwillfixit/migrating-a-dockerized-gitl... yesterday, need to learn more :)