6 ms·
Okay, I want to build a container image using gitlab CI, which runs builds in docker. How would you like me to build an image without using docker in docker, or
by yjftsjthsd-h 6y ago
Okay, I want to build a container image using gitlab CI, which runs builds in docker. How would you like me to build an image without using docker in docker, or buildah in docker?
- phillu 6y agoKaniko is a choice: https://github.com/GoogleContainerTools/kaniko https://github.com/GoogleContainerTools/kaniko Userland docker builds
- tikkabhuna 6y agoWe use kaniko[1] in Gitlab CI and it’s working great for us. It’s annoying the kaniko image requires us to specify the entrypoint. There’s some peculiarities with Dive [2], but otherwise it’s been a very easy migration. [1] https://github.com/GoogleContainerTools/kaniko https://github.com/GoogleContainerTools/kaniko [2] https://github.com/wagoodman/dive/issues/318 https://github.com/wagoodman/dive/issues/318
- merb 6y agokaniko is really really slow: - https://github.com/GoogleContainerTools/kaniko/issues/1392 - https://github.com/GoogleContainerTools/kaniko/issues/970 - https://github.com/GoogleContainerTools/kaniko/issues/875 I've observed basically the same stuff.
- ryan29 6y agoI tried kaniko and didn't like it. Having build and push as part of the same job is frustrating and I view it as a sign that a CI system is built with the expectation of having everything happen post commit by shoveling money into a CI auto-scaler. I know there's `--no-push`, but that's a poor substitute for independent `build`, `tag`, `push` build steps IMO. Do you have any way of running / debugging locally with GitLab CI plus kaniko? Can you run your build pipeline locally on your workstation against uncommitted code? IMO I can build a way better local workflow that allows me to run builds _before_ committing with Drone (`drone exec`). I can toggle between a locally bound Docker daemon or a DIND environment that's going to be virtually identical to the DIND environment on a build runner. The `push` step doesn't run locally plus the secrets needed to push are only accessible from an official runner. I can run it on Windows or Linux (and likely Mac) too. I've been trying to find a self-hosted CI system that's really good at building Docker (or OCI) images and I don't think any exist. They all have short-comings. Having build, tag, push act like an atomic build step is one of the areas where I think most fail. So many claim to enable repeatable builds but none actually do AFAIK. Whoever writes the Dockerfile needs to know a lot about how images are built to have the slightest chance at creating a repeatable build. A great example is having `apt-get update` in a Dockerfile. That command _always_ returns zero, so by itself it makes builds non-repeatable. Sometimes I have a tough time reconciling the development industry because things just don't make sense to me. I remember people complaining about Gradle start times so much they came up with the Gradle daemon. Now no one bats an eye at CI based build systems where you have to commit your code, wait for a runner to get provisioned, wait for Docker or the OCI runtime to spin up, and wait for your project to actually build on some anemic VM. People used to complain about seconds because the wait was "too slow" for good local iteration, but now waiting for minutes is a "good" build system. Seriously WTF? I guess I got on a bit of a rant...
- jrumbut 6y ago> I remember people complaining about Gradle start times so much they came up with the Gradle daemon. Now no one bats an eye at CI based build systems where you have to commit your code, wait for a runner to get provisioned, wait for Docker or the OCI runtime to spin up, and wait for your project to actually build on some anemic VM. I want this framed or sewn onto a pillow or something. It's amazing what we can build, it's baffling what we have built.
- sofixa 6y ago> Having build and push as part of the same job is frustrating and I view it as a sign that a CI system is built with the expectation of having everything happen post commit by shoveling money into a CI auto-scaler No, the idea is to build and push images you can test directly afterwards in the same conditions. With cache and such, build times shouldn't be too long Gitlab CI just runs shell commands, it's pretty trivial to pull the same image its using in the job, and run the same commands locally. If you have long CI times, that can hinder development productivity and should be improved as much as possible, or a local replica of the CI needs to be created.
- Nullabillity 6y ago> Gitlab CI just runs shell commands, it's pretty trivial to pull the same image its using in the job, and run the same commands locally. In fact, they already provide a utility for doing this: gitlab-runner exec[0]. [0]: https://docs.gitlab.com/runner/commands/#gitlab-runner-exec https://docs.gitlab.com/runner/commands/#gitlab-runner-exec
- choeger 6y agoIt is really working for you? Presumably you setup kaniko to build the images first that then run the second part of your pipeline. To run the images, you need to tag them with something that you use inside the gitlab-ci.yml, right? Now what happens when two people push code that make changes to the containers at the same time?
- ryan29 6y agoGitLab CI exposes the pipeline ID and it would work for that.
- choeger 6y agoYour problem is the assumption that a gitlab CI process requires a docker image. If you do the following, you are fine: * Switch to a shell runner * Put the CI dockerfile into your repo * Provide an entry script for CI that builds the container on-demand (and manages caching/cleanup) and then runs the tests/whatever inside that container The point here is that docker/podman provide you with everything you need as long as you have full control. By using gitlab's default CI, you relinquish this control.
- stingraycharles 6y agoBut how would you run integration tests between multiple docker containers? We launch our services in these docker containers and do some integration / e2e tests, and this very much requires Gitlab CI to launch docker containers while inside a Docker container. After using dind for some time we chose to just mount /var/run/docker.sock and keep using the host machine’s docker instance (mostly for the cache), but all in all dind was working fairly well. To be honest, to say “you shouldn’t be doing that” is missing the point; one should be able to do anything they want. In my opinion, the root cause here is docker’s client/server model, which is fixed by other container runtimes such as podman and rkt (which unfortunately is deprecated). One should be able to just launch containers as if they were just another process.
- choeger 6y agoTo the contrary. You should never expect to be able to do what you want with a system provided by someone else. Gitlab has its design decisions in their default setup and you need to take control to cater your use case. This herd mentality of "everyone is doing it this way" is really fundamentally problematic.
- stingraycharles 6y agoNowhere in my post am I saying "everyone is doing it this way". You started out with asserting "you're too far into Docker", we bring up valid use cases for docker-in-docker, and then you saying "This herd mentality [..] is really fundamentally problematic" is really not adding a lot to the discussion.
- CraigJPerry 6y agoA container image is just an archive file with some metadata, could you take the same approach as Google Jib?
- Spidler 6y agoWhy not simply use that then? We're using podman in containers inside gitlab ci. We're _also_ running tests of containers inside containers in containers using gitlab-ci. The main workaround we've applied is using crun as the runtime rather than runc.
- jb_gericke 6y agoKaniko :) no DIND weirdness or privileged containers necessary
- nrk7doh 6y agoOur org just adopted Makisu[1] which works beautifully in our GitLab CI pipelines. After a week of setup and fiddling with the settings, we were able to migrate all our build jobs to Makisu and haven’t looked back. Build times are great too, especially if you set it up with a Redis cache. [1]: https://github.com/uber/makisu https://github.com/uber/makisu
- Narretz 6y agoHave you used / considered kaniko before? If so, what are the advantages of Makisu?
- crummy 6y agoLots of folks recommending kaniko, which we use. But it seems quite slow compared to docker build. Is this not an issue for others?
- ownagefool 6y agohttps://github.com/GoogleContainerTools/kaniko https://github.com/GoogleContainerTools/kaniko can do it but I just use podman / buildah with storage driver as vfs, and then neither requires root / high privileges. I'm not sure why it doesn't do this by default. Performance I guess.