4 ms·

I don't get why they have to clone their repo frequently in the first place - seems to me as a brute force usage of a version control system prone to high cost

by bald42 6y ago

I don't get why they have to clone their repo frequently in the first place - seems to me as a brute force usage of a version control system prone to high cost in the first place.

mschuster91 6y ago
Ephemeral CI runners. I have the same problem at work - 4GB repository that is redownloaded on every single pipeline run. Another reason (which is why we went for ephemeral runners in the first place...) is that if you have stuff that mounts a directory from the repository directory as a volume in a Docker container (e.g. for processing data), you may end up with the Docker container frying permissions in the repo folder (e.g. 0:0 owned files). Now, you can put a cleanup step as part of the CI (=docker run --rm -v $(pwd):/mnt sh -c 'chown -R $runner_uid:$runner_gid)... but unfortunately, Gitlab does not allow a "finally" step that always gets run, so in case the processing fails, the build gets aborted, the server hosting the runner crashes, ... anything happens, the permissions will be fried, and a sysadmin will need to manually intervene. An ephemeral runner using docker:dind however? It simply gets removed.
ForHackernews 6y ago
I don't know about a big org like Pinterest, but it's pretty common for "clone the repo" to be the first step of a CI/CD pipeline when using something like CircleCI or GitlabCI. It's an easy (if inefficient) way to always get the latest changes and if you have disposable build-runners then it all gets thrown away at the end of the pipeline.
- DougBTX 6y ago
  It is interesting that we trust our tools so little. A git hash is a pretty robust way to know whether the code in the repo is what it is supposed to be, so a "git fetch" rather than a fresh "git clone" should be safe, but we can't trust the build steps to not trash the build-runner so the entire thing needs to be thrown away. Edit: for context, I wrote this comment while waiting for `npm ci` to run. Its first step is to delete the node_modules folder, as otherwise it can't be trusted to update correctly.
  - ForHackernews 6y ago
    > we can't trust the build steps to not trash the build-runner so the entire thing needs to be thrown away. I think it's partly this, and partly that everything is shared infrastructure now. I don't want to pay to keep a machine up 24/7 just to use it to run a build for 10 minutes half a dozen times per day. So instead I lease time on shared hardware with ephemeral "containers" or "virtual machines" or whatever.
rightbyte 6y ago
It is a nice and fool proof way to get a clean working environment to just download everything from nothing. And you want different working folder for different jobs anyway so they don't mess with eachother or build of state between jobs due to scripting messups.
user5994461 6y ago
Jenkins has a setting to keep the checkout directory (default) or to clear the directory between builds. At last job, the default was letting broken changes pass the build, they break some step of the setup/run process that's not run on a partial build. New joiners came in and they couldn't build because the build was broken. Had to fix it by setting up two jobs, one running from scratch (30 minutes) and one incremental (10 minutes). The build from scratch was catching a broken change or two every week.
mytailorisrich 6y ago
In order to start with a clean slate and to guarantee state and absence of artefacts from previous builds/pulls it is common practice to start off with a clean directory.

by bald42 6y ago

I don't get why they have to clone their repo frequently in the first place - seems to me as a brute force usage of a version control system prone to high cost in the first place.

mschuster91 6y ago

Ephemeral CI runners. I have the same problem at work - 4GB repository that is redownloaded on every single pipeline run. Another reason (which is why we went for ephemeral runners in the first place...) is that if you have stuff that mounts a directory from the repository directory as a volume in a Docker container (e.g. for processing data), you may end up with the Docker container frying permissions in the repo folder (e.g. 0:0 owned files). Now, you can put a cleanup step as part of the CI (=docker run --rm -v $(pwd):/mnt sh -c 'chown -R $runner_uid:$runner_gid)... but unfortunately, Gitlab does not allow a "finally" step that always gets run, so in case the processing fails, the build gets aborted, the server hosting the runner crashes, ... anything happens, the permissions will be fried, and a sysadmin will need to manually intervene. An ephemeral runner using docker:dind however? It simply gets removed.

ForHackernews 6y ago

I don't know about a big org like Pinterest, but it's pretty common for "clone the repo" to be the first step of a CI/CD pipeline when using something like CircleCI or GitlabCI. It's an easy (if inefficient) way to always get the latest changes and if you have disposable build-runners then it all gets thrown away at the end of the pipeline.

DougBTX 6y ago
It is interesting that we trust our tools so little. A git hash is a pretty robust way to know whether the code in the repo is what it is supposed to be, so a "git fetch" rather than a fresh "git clone" should be safe, but we can't trust the build steps to not trash the build-runner so the entire thing needs to be thrown away. Edit: for context, I wrote this comment while waiting for `npm ci` to run. Its first step is to delete the node_modules folder, as otherwise it can't be trusted to update correctly.
- ForHackernews 6y ago
  > we can't trust the build steps to not trash the build-runner so the entire thing needs to be thrown away. I think it's partly this, and partly that everything is shared infrastructure now. I don't want to pay to keep a machine up 24/7 just to use it to run a build for 10 minutes half a dozen times per day. So instead I lease time on shared hardware with ephemeral "containers" or "virtual machines" or whatever.

rightbyte 6y ago

It is a nice and fool proof way to get a clean working environment to just download everything from nothing. And you want different working folder for different jobs anyway so they don't mess with eachother or build of state between jobs due to scripting messups.

user5994461 6y ago

Jenkins has a setting to keep the checkout directory (default) or to clear the directory between builds. At last job, the default was letting broken changes pass the build, they break some step of the setup/run process that's not run on a partial build. New joiners came in and they couldn't build because the build was broken. Had to fix it by setting up two jobs, one running from scratch (30 minutes) and one incremental (10 minutes). The build from scratch was catching a broken change or two every week.

mytailorisrich 6y ago

In order to start with a clean slate and to guarantee state and absence of artefacts from previous builds/pulls it is common practice to start off with a clean directory.