7 ms·

Maybe you are just too far into docker. I noticed that a lot of default workflows (needlessly) depended on docker running with privileges. One big reason for th

Maybe you are just too far into docker. I noticed that a lot of default workflows (needlessly) depended on docker running with privileges. One big reason for that seem to be Mac users that only know docker from inside a VM. However, if you think about what you're really needing for CI you will easily see that docker-in-docker gains you nothing. You can as well use plain docker (or podman). The same holds for privileges. No CI operation should need privileges, if only for the reason that it should never alter the CI system itself. I encourage you to not take the standard workflows as a given and really think about what you need and I bet you either end up with a use case that can be covered by rootless podman or something that requires real VMs anyways.

zomglings 6y ago
Why do you say that docker-in-docker buys him nothing? It's not obvious at all and you go into no detail whatsoever to back up your opinion. In my experience, that is not true at all. Docker-in-docker allows me to deliver smaller images that can fit into a CI flow as language plugins instead of shipping a beastly 5G docker image with every possible language runtime I need to support for my CI tool.
- core-questions 6y ago
  > beastly 5G docker image my beastly 12GB image that even includes Matlab wants a word with you
  - joana035 6y ago
    >> beastly 5G docker image > my beastly 12GB image that even includes Matlab wants a word with you Perhaps in the next 10 years we will be rediscovering packages. :P If you are in the business of charging complex prices per bits over the network, then docker seems to be quite a good investment and making it as popular as possible is a good strategy to print money. /s
    - core-questions 6y ago
      Well, to be fair, it is packages - I'm just using Docker (for this section of our stack) as a different sort of VM, essentially. It runs a service manager and a VNC X session, for chrissakes ;)
    - disgruntledphd2 6y ago
      > If you are in the business of charging complex prices per bits over the network, then docker seems to be quite a good investment and making it as popular as possible is a good strategy to print money. /s True, that. To be fair, at least it allows me to avoid lots of the brokenness of Python packaging.
      - joana035 6y ago
        Is always good to report packaging bugs so then people can fix them, do you have examples of python packages that can be improved?
        disgruntledphd2 6y ago
        See my previous comments. tl;dr pip silently breaks my environments, mostly connected to upgrading numpy and other scientific/data science libraries.
  - deleted 6y ago
    [deleted]
  - maccard 6y ago
    Our images are 35GB, and I've spent much of the last two weeks breaking up files so we don't hit the 8GB per file limit, and my next week will be trying to avoid hitting the per-layer limit.
- joana035 6y ago
  It is because to build the image using docker requires the docker client to talk with a dockerd daemon, so one has to configure the client to access the dockerd which allow untrusted code to run as root in the host. Docker-in-docker is a workaround to make docker work in CI. Basically a security nightmare and bad design that podman doesn't have.
  - wwright 6y ago
    Kubernetes as a layer of indirection is another solution.
    - joana035 6y ago
      Kubernetes is a consistent management api for linux (so then you don't need to interact with iptables, mount and all that "hard stuff"). I don't think kubernetes is a solution for context of building an image (a rootfs tree into a .tar.gz file). Unless you are using kaniko which extends the kubernetes api to add the capability of creating images, but that is handled by kaniko itself via the same api.
      - wwright 6y ago
        I was probably unclear. Kubernetes is a good solution for managing containers (obviously). I use it for CI and it works very, very well, though the CI tools still have more features they could add with the integration.
  - hinkley 6y ago
    Any build script can do serious damage to the environment it runs in. Before docker, you'd have to create a new VM from time to time because the build agent had rotted away or died in an altercation with a bad build. Docker in Docker in CI is like a lock on a door. It keeps honest people from being naughty, and is fairly efficient about it. I don't think the question is "should I run CI in docker in docker," it's whose CI should I run in docker in docker. Me in my coworkers can share docker images. Customers or freeloaders cannot. So if that's in your problem domain, then you're right, it's a bad idea. But it isn't for most people.
yjftsjthsd-h 6y ago
Okay, I want to build a container image using gitlab CI, which runs builds in docker. How would you like me to build an image without using docker in docker, or buildah in docker?
- phillu 6y ago
  Kaniko is a choice: https://github.com/GoogleContainerTools/kaniko https://github.com/GoogleContainerTools/kaniko Userland docker builds
- tikkabhuna 6y ago
  We use kaniko[1] in Gitlab CI and it’s working great for us. It’s annoying the kaniko image requires us to specify the entrypoint. There’s some peculiarities with Dive [2], but otherwise it’s been a very easy migration. [1] https://github.com/GoogleContainerTools/kaniko https://github.com/GoogleContainerTools/kaniko [2] https://github.com/wagoodman/dive/issues/318 https://github.com/wagoodman/dive/issues/318
  - merb 6y ago
    kaniko is really really slow: - https://github.com/GoogleContainerTools/kaniko/issues/1392 - https://github.com/GoogleContainerTools/kaniko/issues/970 - https://github.com/GoogleContainerTools/kaniko/issues/875 I've observed basically the same stuff.
  - ryan29 6y ago
    I tried kaniko and didn't like it. Having build and push as part of the same job is frustrating and I view it as a sign that a CI system is built with the expectation of having everything happen post commit by shoveling money into a CI auto-scaler. I know there's `--no-push`, but that's a poor substitute for independent `build`, `tag`, `push` build steps IMO. Do you have any way of running / debugging locally with GitLab CI plus kaniko? Can you run your build pipeline locally on your workstation against uncommitted code? IMO I can build a way better local workflow that allows me to run builds _before_ committing with Drone (`drone exec`). I can toggle between a locally bound Docker daemon or a DIND environment that's going to be virtually identical to the DIND environment on a build runner. The `push` step doesn't run locally plus the secrets needed to push are only accessible from an official runner. I can run it on Windows or Linux (and likely Mac) too. I've been trying to find a self-hosted CI system that's really good at building Docker (or OCI) images and I don't think any exist. They all have short-comings. Having build, tag, push act like an atomic build step is one of the areas where I think most fail. So many claim to enable repeatable builds but none actually do AFAIK. Whoever writes the Dockerfile needs to know a lot about how images are built to have the slightest chance at creating a repeatable build. A great example is having `apt-get update` in a Dockerfile. That command _always_ returns zero, so by itself it makes builds non-repeatable. Sometimes I have a tough time reconciling the development industry because things just don't make sense to me. I remember people complaining about Gradle start times so much they came up with the Gradle daemon. Now no one bats an eye at CI based build systems where you have to commit your code, wait for a runner to get provisioned, wait for Docker or the OCI runtime to spin up, and wait for your project to actually build on some anemic VM. People used to complain about seconds because the wait was "too slow" for good local iteration, but now waiting for minutes is a "good" build system. Seriously WTF? I guess I got on a bit of a rant...
fiddlerwoaroof 6y ago
I think the ideal is using Nix to manage development dependencies and to handle building minimal docker images for deployment.
- jokethrowaway 6y ago
  Why not just use nix at that point? You can at least retain the advantage of having truly immutable and reproducible builds. I use docker for most of my clients' work but for in house stuff I just use nix.
  - fiddlerwoaroof 6y ago
    Well, just nix works pretty well unless you want to deploy to K8s or Fargate or similar.
  - soraminazuki 6y ago
    Docker and Nix are mostly orthogonal technologies though. Nix is an excellent build tool while Docker is useful as an universal software distribution format. They're really useful together because Nix is actually good at creating compact Docker images. But of course, if you don't have to worry much about distribution, it would be much nicer to stick with Nix as you've mentioned.
coldtea 6y ago
>One big reason for that seem to be Mac users that only know docker from inside a VM That has not been the case for a good while now... Docker has been running directly on a hypervisor on the Mac.
- babarock 6y ago
  Honest question, what's the difference between "from inside a VM" and "directly on a hypervisor"? I always thought it meant the same thing.
  - tomnipotent 6y ago
    Docker for Mac is still running a VM via LinuxKit/HyperKit, but there's a big difference in overhead between LinuxKit and running VMWare/VirtualBox.
  - coldtea 6y ago
    Well, in the case of Mac, basically performance, as you still need linux libs, kernel API and everything. But it's much lighter-weight than something like Virtualbox, VMWare, etc. In Linux iirc, a hypervisor can share such resources with the host system (since they are both Linuxes).

zomglings 6y ago

Why do you say that docker-in-docker buys him nothing? It's not obvious at all and you go into no detail whatsoever to back up your opinion. In my experience, that is not true at all. Docker-in-docker allows me to deliver smaller images that can fit into a CI flow as language plugins instead of shipping a beastly 5G docker image with every possible language runtime I need to support for my CI tool.

core-questions 6y ago
> beastly 5G docker image my beastly 12GB image that even includes Matlab wants a word with you
- joana035 6y ago
  >> beastly 5G docker image > my beastly 12GB image that even includes Matlab wants a word with you Perhaps in the next 10 years we will be rediscovering packages. :P If you are in the business of charging complex prices per bits over the network, then docker seems to be quite a good investment and making it as popular as possible is a good strategy to print money. /s
  - core-questions 6y ago
    Well, to be fair, it is packages - I'm just using Docker (for this section of our stack) as a different sort of VM, essentially. It runs a service manager and a VNC X session, for chrissakes ;)
  - disgruntledphd2 6y ago
    > If you are in the business of charging complex prices per bits over the network, then docker seems to be quite a good investment and making it as popular as possible is a good strategy to print money. /s True, that. To be fair, at least it allows me to avoid lots of the brokenness of Python packaging.
    - joana035 6y ago
      Is always good to report packaging bugs so then people can fix them, do you have examples of python packages that can be improved?
      - disgruntledphd2 6y ago
        See my previous comments. tl;dr pip silently breaks my environments, mostly connected to upgrading numpy and other scientific/data science libraries.
- deleted 6y ago
  [deleted]
- maccard 6y ago
  Our images are 35GB, and I've spent much of the last two weeks breaking up files so we don't hit the 8GB per file limit, and my next week will be trying to avoid hitting the per-layer limit.
joana035 6y ago
It is because to build the image using docker requires the docker client to talk with a dockerd daemon, so one has to configure the client to access the dockerd which allow untrusted code to run as root in the host. Docker-in-docker is a workaround to make docker work in CI. Basically a security nightmare and bad design that podman doesn't have.
- wwright 6y ago
  Kubernetes as a layer of indirection is another solution.
  - joana035 6y ago
    Kubernetes is a consistent management api for linux (so then you don't need to interact with iptables, mount and all that "hard stuff"). I don't think kubernetes is a solution for context of building an image (a rootfs tree into a .tar.gz file). Unless you are using kaniko which extends the kubernetes api to add the capability of creating images, but that is handled by kaniko itself via the same api.
    - wwright 6y ago
      I was probably unclear. Kubernetes is a good solution for managing containers (obviously). I use it for CI and it works very, very well, though the CI tools still have more features they could add with the integration.
- hinkley 6y ago
  Any build script can do serious damage to the environment it runs in. Before docker, you'd have to create a new VM from time to time because the build agent had rotted away or died in an altercation with a bad build. Docker in Docker in CI is like a lock on a door. It keeps honest people from being naughty, and is fairly efficient about it. I don't think the question is "should I run CI in docker in docker," it's whose CI should I run in docker in docker. Me in my coworkers can share docker images. Customers or freeloaders cannot. So if that's in your problem domain, then you're right, it's a bad idea. But it isn't for most people.

yjftsjthsd-h 6y ago

Okay, I want to build a container image using gitlab CI, which runs builds in docker. How would you like me to build an image without using docker in docker, or buildah in docker?

phillu 6y ago
Kaniko is a choice: https://github.com/GoogleContainerTools/kaniko https://github.com/GoogleContainerTools/kaniko Userland docker builds
tikkabhuna 6y ago
We use kaniko[1] in Gitlab CI and it’s working great for us. It’s annoying the kaniko image requires us to specify the entrypoint. There’s some peculiarities with Dive [2], but otherwise it’s been a very easy migration. [1] https://github.com/GoogleContainerTools/kaniko https://github.com/GoogleContainerTools/kaniko [2] https://github.com/wagoodman/dive/issues/318 https://github.com/wagoodman/dive/issues/318
- merb 6y ago
  kaniko is really really slow: - https://github.com/GoogleContainerTools/kaniko/issues/1392 - https://github.com/GoogleContainerTools/kaniko/issues/970 - https://github.com/GoogleContainerTools/kaniko/issues/875 I've observed basically the same stuff.
- ryan29 6y ago
  I tried kaniko and didn't like it. Having build and push as part of the same job is frustrating and I view it as a sign that a CI system is built with the expectation of having everything happen post commit by shoveling money into a CI auto-scaler. I know there's `--no-push`, but that's a poor substitute for independent `build`, `tag`, `push` build steps IMO. Do you have any way of running / debugging locally with GitLab CI plus kaniko? Can you run your build pipeline locally on your workstation against uncommitted code? IMO I can build a way better local workflow that allows me to run builds _before_ committing with Drone (`drone exec`). I can toggle between a locally bound Docker daemon or a DIND environment that's going to be virtually identical to the DIND environment on a build runner. The `push` step doesn't run locally plus the secrets needed to push are only accessible from an official runner. I can run it on Windows or Linux (and likely Mac) too. I've been trying to find a self-hosted CI system that's really good at building Docker (or OCI) images and I don't think any exist. They all have short-comings. Having build, tag, push act like an atomic build step is one of the areas where I think most fail. So many claim to enable repeatable builds but none actually do AFAIK. Whoever writes the Dockerfile needs to know a lot about how images are built to have the slightest chance at creating a repeatable build. A great example is having `apt-get update` in a Dockerfile. That command _always_ returns zero, so by itself it makes builds non-repeatable. Sometimes I have a tough time reconciling the development industry because things just don't make sense to me. I remember people complaining about Gradle start times so much they came up with the Gradle daemon. Now no one bats an eye at CI based build systems where you have to commit your code, wait for a runner to get provisioned, wait for Docker or the OCI runtime to spin up, and wait for your project to actually build on some anemic VM. People used to complain about seconds because the wait was "too slow" for good local iteration, but now waiting for minutes is a "good" build system. Seriously WTF? I guess I got on a bit of a rant...

fiddlerwoaroof 6y ago

I think the ideal is using Nix to manage development dependencies and to handle building minimal docker images for deployment.

jokethrowaway 6y ago
Why not just use nix at that point? You can at least retain the advantage of having truly immutable and reproducible builds. I use docker for most of my clients' work but for in house stuff I just use nix.
- fiddlerwoaroof 6y ago
  Well, just nix works pretty well unless you want to deploy to K8s or Fargate or similar.
- soraminazuki 6y ago
  Docker and Nix are mostly orthogonal technologies though. Nix is an excellent build tool while Docker is useful as an universal software distribution format. They're really useful together because Nix is actually good at creating compact Docker images. But of course, if you don't have to worry much about distribution, it would be much nicer to stick with Nix as you've mentioned.

coldtea 6y ago

>One big reason for that seem to be Mac users that only know docker from inside a VM That has not been the case for a good while now... Docker has been running directly on a hypervisor on the Mac.

babarock 6y ago
Honest question, what's the difference between "from inside a VM" and "directly on a hypervisor"? I always thought it meant the same thing.
- tomnipotent 6y ago
  Docker for Mac is still running a VM via LinuxKit/HyperKit, but there's a big difference in overhead between LinuxKit and running VMWare/VirtualBox.
- coldtea 6y ago
  Well, in the case of Mac, basically performance, as you still need linux libs, kernel API and everything. But it's much lighter-weight than something like Virtualbox, VMWare, etc. In Linux iirc, a hypervisor can share such resources with the host system (since they are both Linuxes).