4 ms·

> Also if you use your own runner, cloning the repo to the runner will also be included in your bandwidth limit. You make it sounds like it's horrible. I'd off

> Also if you use your own runner, cloning the repo to the runner will also be included in your bandwidth limit. You make it sounds like it's horrible. I'd offer a different take in that sure, it can be a huge negative if done recklessly on GitLab's part. But if done correctly, it can actually be a good thing. I think CI pipelines doing a full clone from scratch on every build (or npm installs or other bootstraps) is extremely wasteful in terms of resources, so I'd be glad if that reduces it significantly.

jacooper 4y ago
GitLab CI default clone configuration already AFAIK only includes the last 20 commits, so its already slimmed down.
- karamanolev 4y ago
  If you start from scratch, regardless of history, you're fetching at least the size of the repo. In many cases, that's hundreds of MBs for every build and I'm a huge fan of building for every commit. I find that a colossal waste. I have a very old fashioned and "not recommended" setup with Jenkins that only pulls new commits, because it works out of a persistent working directory. Works wonders for a Django/ES6 project and sips bandwidth. I wish more of the modern containerized, start-from-scratch and so on setups would work in a similar way wrt the bandwidth they use.
Dayshine 4y ago
GitLab's runner agent software doesn't support a local git cache. GitLab.com Saas agents have no sticky-ness, so basically always clone.
- capableweb 4y ago
  Yeah, that does sound wasteful, just like what parent said. They should add some sort of cache support if that's the case.
- dnsmichi 4y ago
  Thanks for the suggestion of adding a Git cache for a GitLab Runner. A similar mechanism already exists that allows control of the Git strategy with fetch on a local working copy which is faster, and clone if not yet existing. All benefits and limitations are documented [0]. To help prevent unneeded traffic when a new pipeline job is executed, GitLab Runner uses a shallow Git clone by default on GitLab.com SaaS that only pulls a limited set of Git commits from the current head instead of a full clone. [1] There is a feature proposal [2] to add support for partial clone and sparse-checkout strategies that have been added in more recent Git versions. Recommend commenting/subscribing. Please note that for GitLab.com SaaS Shared Runners, the traffic limits do not apply with mostly internal cloud traffic. [3] [0] https://docs.gitlab.com/ee/ci/runners/configure_runners.html#git-strategy https://docs.gitlab.com/ee/ci/runners/configure_runners.html... [1] https://docs.gitlab.com/ee/ci/large_repositories/#shallow-cloning https://docs.gitlab.com/ee/ci/large_repositories/#shallow-cl... [2] https://gitlab.com/gitlab-org/gitlab-runner/-/issues/26631 https://gitlab.com/gitlab-org/gitlab-runner/-/issues/26631 [3] https://about.gitlab.com/pricing/#what-counts-towards-my-transfer-limit https://about.gitlab.com/pricing/#what-counts-towards-my-tra...

jacooper 4y ago

GitLab CI default clone configuration already AFAIK only includes the last 20 commits, so its already slimmed down.

karamanolev 4y ago
If you start from scratch, regardless of history, you're fetching at least the size of the repo. In many cases, that's hundreds of MBs for every build and I'm a huge fan of building for every commit. I find that a colossal waste. I have a very old fashioned and "not recommended" setup with Jenkins that only pulls new commits, because it works out of a persistent working directory. Works wonders for a Django/ES6 project and sips bandwidth. I wish more of the modern containerized, start-from-scratch and so on setups would work in a similar way wrt the bandwidth they use.

Dayshine 4y ago

GitLab's runner agent software doesn't support a local git cache. GitLab.com Saas agents have no sticky-ness, so basically always clone.

capableweb 4y ago
Yeah, that does sound wasteful, just like what parent said. They should add some sort of cache support if that's the case.
dnsmichi 4y ago
Thanks for the suggestion of adding a Git cache for a GitLab Runner. A similar mechanism already exists that allows control of the Git strategy with fetch on a local working copy which is faster, and clone if not yet existing. All benefits and limitations are documented [0]. To help prevent unneeded traffic when a new pipeline job is executed, GitLab Runner uses a shallow Git clone by default on GitLab.com SaaS that only pulls a limited set of Git commits from the current head instead of a full clone. [1] There is a feature proposal [2] to add support for partial clone and sparse-checkout strategies that have been added in more recent Git versions. Recommend commenting/subscribing. Please note that for GitLab.com SaaS Shared Runners, the traffic limits do not apply with mostly internal cloud traffic. [3] [0] https://docs.gitlab.com/ee/ci/runners/configure_runners.html#git-strategy https://docs.gitlab.com/ee/ci/runners/configure_runners.html... [1] https://docs.gitlab.com/ee/ci/large_repositories/#shallow-cloning https://docs.gitlab.com/ee/ci/large_repositories/#shallow-cl... [2] https://gitlab.com/gitlab-org/gitlab-runner/-/issues/26631 https://gitlab.com/gitlab-org/gitlab-runner/-/issues/26631 [3] https://about.gitlab.com/pricing/#what-counts-towards-my-transfer-limit https://about.gitlab.com/pricing/#what-counts-towards-my-tra...