7 ms·

Kaniko: Build container images in Kubernetes

261 points by dlor 8y ago

linkmotif 8y ago
This is cool! Thanks for posting. I can see how this is useful if building images is part of your CI process. I’ve been using https://github.com/dminkovsky/kube-cloud-build https://github.com/dminkovsky/kube-cloud-build to build images on Google Cloud Container Builder. It handles generating Cloud Container Builder build requests based on the images specified in my Kubernetes manifests, which was a big deal for me since writing build requests by hand was a total pain.
- amq 8y ago
  If you have CI, you normally shouldn't need something like this.
  - rckrd 8y ago
    The idea is to not have to maintain separate CI infrastructure in addition to your Kubernetes cluster. Disclaimer: I worked on this kaniko at Google
    - CSDude 8y ago
      Thanks for your work really appreciated, but is there any way to cache some layers?
      - dlor 8y ago
        There should be! We haven't thought too much about it yet, but there's no reason it wouldn't work. We can't reuse the Docker daemon cache because that would imply access to the docker daemon. Discloser - I work on kaniko and other container things at Google.
jacques_chester 8y ago
This work (and related efforts like Img and Buildah) is a big deal. Right now docker images and Dockerfiles are joined at the hip to the Docker daemon. It works great for local development, but for hosted systems that run on containers, it's a dire mess. I have personally slammed head-first into Docker-in-Docker quagmires on Kubernetes and Concourse. Not knowing the particular arcane rites and having neither sufficient eye of newt nor sufficient patience to get it to work, I like everyone else in the universe gave up. Not an acceptable state of affairs, given the many problems of Dockerfiles in themselves. Dockerfiles force an ugly choice. You can have ease of development or you can have fast, safe production images. But you can't really have both. Kaniko is another step in the direction of divorcing docker images as a means of distributing bits from Dockerfiles as a means of describing docker images from Docker daemons as a means for assembling the images. All three are different and should no longer be conflated. Disclosure: I work for Pivotal, we have a lot of stuff that does stuff with containers.
- derefr 8y ago
  > Not knowing the particular arcane rites and having neither sufficient eye of newt nor sufficient patience to get it to work, I like everyone else in the universe gave up. One thing I feel like more people need to know: Docker container-images are really not that hard to build "manually", without using Docker. Just because Docker itself builds images by repeatedly invoking `docker run` and then snapshotting the new layers, people think that's what their build tools need to do as well. No! You just need to have the files you want, and know the config you want, and the ability to build a tar file. Here's a look inside an average one-layer Docker image: $ mkdir busybox_image; cd busybox_image $ docker pull busybox:latest $ docker save busybox:latest | tar x $ tree . ├── 8ac48589692a53a9b8c2d1ceaa6b402665aa7fe667ba51ccc03002300856d8c7.json ├── f4752d3dbb207ca444ab74169ca5e21c5a47085c4aba49e367315bd4ca3a91ba │ ├── VERSION │ ├── json │ └── layer.tar ├── manifest.json └── repositories 1 directory, 6 files • `repositories` contains the tag refs that will be imported when you `docker load` this archive; • `manifest.json` contains the declarations needed for the daemon to unpack the layers into its storage backend (just a listing of the layer.tar files, basically); • the SHA-named config file specifies how to reconstruct a container from this archive, if you dumped it from a container (and I believe it's optional when constructing a "fresh" archive for `docker load`ing); Each SHA-named layer directory contains: • a `layer.tar` file, which is what you'd expect, e.g.: -rwxr-xr-x 0 0 0 1037528 16 May 2017 bin/bash • a `json` file, specifying (the patch of!) the container config that that layer creates. (If you're composing a docker image from scratch, you just need the one layer, so you don't have to worry about the patching semantics.) That's pretty much it. Make a directory that looks like that, tar it up, and `docker load` will accept it and turn it into something you can `docker push` to a registry. No need to have the privileges required to run docker containers (i.e. unshare(3)) in your environment. (And `docker load` and `docker push` work fine without a working Docker execution backend, IIRC.)
gingerlime 8y ago
We’re using docker for development, but we still have to take the leap into production. The whole build/push/pull part is rather confusing somehow. I tried docker hub or docker cloud build as it’s now called(?), but the build itself takes forever... what are people using these days?? Also for development machines, how do you sync things between developers. I can commit a docker file change, but unless I explicitly tell docker compose to rebuild my images and containers, it will happily stick to the old version. I have to keep nagging our (3) developers to do this from time to time... what am I doing wrong?? Sorry if these are dumb questions but we’re still stuck with the basics it seems.
- lobster_johnson 8y ago
  If you're still struggling with the build workflow, it's probably not yet the right time to take that leap. It's not rocket science, of course. You build an image somewhere (your local machine, a CI server, anywhere), push to a registry, and when you want run the image, you pull from the registry and run it. ("docker run" will, by default, automatically pull when you ask it to run something.) I don't quite understand what your Compose problem is. Is the Compose file referencing images published to, say, Docker Hub? If so, the image obviously has to be built and published beforehand. However, it's also possible to run Compose against local checkouts, then run "docker-compose up --build", e.g.: version: '3.2' services: mainApp: build: context: . service1: build: context: ../service1 service2: build: context: ../service2 and so on. There's a whole ecosystem of tools built around Docker for building, testing, deploying and orchestrating Docker applications. Kubernetes is one. If you're having issues with the Docker basics, however, I wouldn't consider any of these systems quite yet, although you should consider automating your building and testing with a CI (continuous integration) system, rather than making your devs build and test on their local machines. As with anything, to actually use Docker in production you'll need an ops person/team that knows how to run it. That could be something as simple as a manual "docker run" or a manual "docker-compose", to something much more complex such as Kubernetes. This is the complicated part.
  - gingerlime 8y ago
ofrzeta 8y ago
With the availability of the free Red Hat tools for building container images (buildah...) and this, it will be interesting to see what remains of Docker (Inc).
- CSDude 8y ago
  Although Docker images are not hard to build, (it is just a layers of tars with proper jsons) it is very nice to see such tools rise. Although I have a nice Kubernetes cluster, or any orchestrator, due to security reasons, I have to come up with a new VM with Docker installed and build it there, which really sucks. It is sad to see Docker did not implement this years ago although people wanted it a lot. They were busy deprecating the Swarm Whatever^TM for the 3rd time and not listening as usual.
- zapita 8y ago
  It's pretty clear that Docker has been focused on moving downstream. They want to add value by assembling open-source components into a complete platform that they can control and sell. They don't want to be the ones developing all the components themselves - at this level of maturity and sophistication in the container market, they just don't have the manpower to do that. A major benefit of that strategy is that they can use the best component available, regardless of who developed it. I bet they're feeling spread very thin on the open-source side, and would love to redirect some of their resources away from developing a gazillion open-source gadgets on their own, and towards their commercial products (which historically have been not as good in my experience). Evidence that Docker is doing this: - They only advertise three things with the name Docker: "Docker for Mac" (a free product that is not open-source), "Docker EE" (an enterprise product), and "Docker Hub" (a cloud service). Those are all downstream products, like RHEL or Openshift. - The whole "Moby" thing is basically their upstream brand, aka "the things not called Docker". - They spun out tons of smaller projects like buildkit, linuxkit, containerd, runc, and seem eager to get others to use them and contribute, even competitors. - They embraced Kubernetes as part of their downstream product, even though they famously did not invent it, and they certainly don't control it. So I think people saying "these free open-source tools are killing Docker" are missing the point. The real competition for Docker is Openshift vs Docker EE, everything else is implementation details. If you listen to the sales pitch of these two companies right now, it's an absolute tug of war. Docker focuses on independence and innovation ("we know where containers are going, and we don't force RHEL down your throat"). Red Hat focuses on maturity and upstream control ("We've been by your side for 20 years, are you going to trust us or some Silicon Valley hipster? Also we employ more Kubernetes contributors than anyone else"). That's the real battle, in my experience on the open-source side you'll find mostly engineers from all side collaborating peacefully and building whatever they need to get their job done.
medyadaily 8y ago
Sweet, This is truly great, I was hoping for a service like this for a long time. being able to build images without root privilege!
- amouat 8y ago
  Kaniko does run in the container as root, but the container doesn't need to be granted any extra privileges when run (you don't need the equivalent of Docker's --privileged flag).
zapita 8y ago
Does this (or could this) use Buildkit? It seems that Docker themselves are encouraging the development of an ecosystem of third-party container build tools, with buildkit as an interoperability layer. I heard good things about buildkit but haven't tried it yet. If Kaniko authors are reading this: have you considered buildkit and, if not, would you be open to contributions based on it? My understanding is that the official 'docker build' itself is based on Buildkit. https://github.com/moby/buildkit https://github.com/moby/buildkit
- dlor 8y ago
  Kaniko doesn't use buildkit - buildkit still uses containerd/runC under the hood so it can't run inside a container easily. We are looking at interoperability with buildkit (and the large set of of other tooling like this) through the CBI: https://github.com/containerbuilding/cbi https://github.com/containerbuilding/cbi which aims to be a neutral interface on top of things like buildkit, buildah, docker and kaniko that build images. Discloser: I work on kaniko and other container things at Google.
  - zapita 8y ago
    Very interesting, and it looks like the people working on CBI are also active on Buildkit, which is a good sign! Thank you for the pointer.
    - AkihiroSuda 8y ago
      @zapita FYI: Kaniko plugin for CBI is now available. https://github.com/containerbuilding/cbi/pull/35 https://github.com/containerbuilding/cbi/pull/35
      - zapita 8y ago
        Thanks Akihiro for the follow-up. And while I'm at it, thank you for all this excellent code that we get to enjoy for free! It's really fantastic work.
bkeroack 8y ago
Interesting! We built and use a service called Furan: https://github.com/dollarshaveclub/furan https://github.com/dollarshaveclub/furan That said, Furan isn't suitable for untrusted Dockerfiles (or multi-tenant environments) exactly due to the security implications of access to the Docker engine socket. The issue I see with Kaniko is drift from upstream Moby/Docker syntax. One of the strengths with Furan is that you have the guarantee that the Docker build you perform locally is exactly what happens by the service. When you can't make this guarantee you get into weird situations where "the build works for me locally" but there's some issue when doing a remote build. That's also why we've resisted putting special build magic into Furan (like injecting metadata into the build context, for example).
tyrankh 8y ago
This is great work. Github link for the lazy: https://github.com/GoogleCloudPlatform/kaniko https://github.com/GoogleCloudPlatform/kaniko
ksajadi 8y ago
This is a great tool. Wish it could work with build workflow tools like Habitus (http://www.habitus.io http://www.habitus.io)
spockz 8y ago
My understanding is that it is best practice to run your docker builds and images as s non root user. OpenShift will complain if you do for example. Now this kaniko image runs the build as root contrary to the recommendation and the post explicitly mentions this difference with Orca. Why is it okay now for kaniko to run as root user?
- humbleMouse 8y ago
  This was my first thought as well.

261 points by dlor 8y ago

This is cool! Thanks for posting. I can see how this is useful if building images is part of your CI process. I’ve been using https://github.com/dminkovsky/kube-cloud-build https://github.com/dminkovsky/kube-cloud-build to build images on Google Cloud Container Builder. It handles generating Cloud Container Builder build requests based on the images specified in my Kubernetes manifests, which was a big deal for me since writing build requests by hand was a total pain.

amq 8y ago
If you have CI, you normally shouldn't need something like this.
- rckrd 8y ago
  The idea is to not have to maintain separate CI infrastructure in addition to your Kubernetes cluster. Disclaimer: I worked on this kaniko at Google
  - CSDude 8y ago
    Thanks for your work really appreciated, but is there any way to cache some layers?
    - dlor 8y ago
      There should be! We haven't thought too much about it yet, but there's no reason it wouldn't work. We can't reuse the Docker daemon cache because that would imply access to the docker daemon. Discloser - I work on kaniko and other container things at Google.

jacques_chester 8y ago

This work (and related efforts like Img and Buildah) is a big deal. Right now docker images and Dockerfiles are joined at the hip to the Docker daemon. It works great for local development, but for hosted systems that run on containers, it's a dire mess. I have personally slammed head-first into Docker-in-Docker quagmires on Kubernetes and Concourse. Not knowing the particular arcane rites and having neither sufficient eye of newt nor sufficient patience to get it to work, I like everyone else in the universe gave up. Not an acceptable state of affairs, given the many problems of Dockerfiles in themselves. Dockerfiles force an ugly choice. You can have ease of development or you can have fast, safe production images. But you can't really have both. Kaniko is another step in the direction of divorcing docker images as a means of distributing bits from Dockerfiles as a means of describing docker images from Docker daemons as a means for assembling the images. All three are different and should no longer be conflated. Disclosure: I work for Pivotal, we have a lot of stuff that does stuff with containers.

derefr 8y ago
> Not knowing the particular arcane rites and having neither sufficient eye of newt nor sufficient patience to get it to work, I like everyone else in the universe gave up. One thing I feel like more people need to know: Docker container-images are really not that hard to build "manually", without using Docker. Just because Docker itself builds images by repeatedly invoking `docker run` and then snapshotting the new layers, people think that's what their build tools need to do as well. No! You just need to have the files you want, and know the config you want, and the ability to build a tar file. Here's a look inside an average one-layer Docker image: $ mkdir busybox_image; cd busybox_image $ docker pull busybox:latest $ docker save busybox:latest | tar x $ tree . ├── 8ac48589692a53a9b8c2d1ceaa6b402665aa7fe667ba51ccc03002300856d8c7.json ├── f4752d3dbb207ca444ab74169ca5e21c5a47085c4aba49e367315bd4ca3a91ba │ ├── VERSION │ ├── json │ └── layer.tar ├── manifest.json └── repositories 1 directory, 6 files • `repositories` contains the tag refs that will be imported when you `docker load` this archive; • `manifest.json` contains the declarations needed for the daemon to unpack the layers into its storage backend (just a listing of the layer.tar files, basically); • the SHA-named config file specifies how to reconstruct a container from this archive, if you dumped it from a container (and I believe it's optional when constructing a "fresh" archive for `docker load`ing); Each SHA-named layer directory contains: • a `layer.tar` file, which is what you'd expect, e.g.: -rwxr-xr-x 0 0 0 1037528 16 May 2017 bin/bash • a `json` file, specifying (the patch of!) the container config that that layer creates. (If you're composing a docker image from scratch, you just need the one layer, so you don't have to worry about the patching semantics.) That's pretty much it. Make a directory that looks like that, tar it up, and `docker load` will accept it and turn it into something you can `docker push` to a registry. No need to have the privileges required to run docker containers (i.e. unshare(3)) in your environment. (And `docker load` and `docker push` work fine without a working Docker execution backend, IIRC.)

gingerlime 8y ago

We’re using docker for development, but we still have to take the leap into production. The whole build/push/pull part is rather confusing somehow. I tried docker hub or docker cloud build as it’s now called(?), but the build itself takes forever... what are people using these days?? Also for development machines, how do you sync things between developers. I can commit a docker file change, but unless I explicitly tell docker compose to rebuild my images and containers, it will happily stick to the old version. I have to keep nagging our (3) developers to do this from time to time... what am I doing wrong?? Sorry if these are dumb questions but we’re still stuck with the basics it seems.

lobster_johnson 8y ago
If you're still struggling with the build workflow, it's probably not yet the right time to take that leap. It's not rocket science, of course. You build an image somewhere (your local machine, a CI server, anywhere), push to a registry, and when you want run the image, you pull from the registry and run it. ("docker run" will, by default, automatically pull when you ask it to run something.) I don't quite understand what your Compose problem is. Is the Compose file referencing images published to, say, Docker Hub? If so, the image obviously has to be built and published beforehand. However, it's also possible to run Compose against local checkouts, then run "docker-compose up --build", e.g.: version: '3.2' services: mainApp: build: context: . service1: build: context: ../service1 service2: build: context: ../service2 and so on. There's a whole ecosystem of tools built around Docker for building, testing, deploying and orchestrating Docker applications. Kubernetes is one. If you're having issues with the Docker basics, however, I wouldn't consider any of these systems quite yet, although you should consider automating your building and testing with a CI (continuous integration) system, rather than making your devs build and test on their local machines. As with anything, to actually use Docker in production you'll need an ops person/team that knows how to run it. That could be something as simple as a manual "docker run" or a manual "docker-compose", to something much more complex such as Kubernetes. This is the complicated part.
- gingerlime 8y ago

ofrzeta 8y ago

With the availability of the free Red Hat tools for building container images (buildah...) and this, it will be interesting to see what remains of Docker (Inc).

CSDude 8y ago
Although Docker images are not hard to build, (it is just a layers of tars with proper jsons) it is very nice to see such tools rise. Although I have a nice Kubernetes cluster, or any orchestrator, due to security reasons, I have to come up with a new VM with Docker installed and build it there, which really sucks. It is sad to see Docker did not implement this years ago although people wanted it a lot. They were busy deprecating the Swarm Whatever^TM for the 3rd time and not listening as usual.
zapita 8y ago
It's pretty clear that Docker has been focused on moving downstream. They want to add value by assembling open-source components into a complete platform that they can control and sell. They don't want to be the ones developing all the components themselves - at this level of maturity and sophistication in the container market, they just don't have the manpower to do that. A major benefit of that strategy is that they can use the best component available, regardless of who developed it. I bet they're feeling spread very thin on the open-source side, and would love to redirect some of their resources away from developing a gazillion open-source gadgets on their own, and towards their commercial products (which historically have been not as good in my experience). Evidence that Docker is doing this: - They only advertise three things with the name Docker: "Docker for Mac" (a free product that is not open-source), "Docker EE" (an enterprise product), and "Docker Hub" (a cloud service). Those are all downstream products, like RHEL or Openshift. - The whole "Moby" thing is basically their upstream brand, aka "the things not called Docker". - They spun out tons of smaller projects like buildkit, linuxkit, containerd, runc, and seem eager to get others to use them and contribute, even competitors. - They embraced Kubernetes as part of their downstream product, even though they famously did not invent it, and they certainly don't control it. So I think people saying "these free open-source tools are killing Docker" are missing the point. The real competition for Docker is Openshift vs Docker EE, everything else is implementation details. If you listen to the sales pitch of these two companies right now, it's an absolute tug of war. Docker focuses on independence and innovation ("we know where containers are going, and we don't force RHEL down your throat"). Red Hat focuses on maturity and upstream control ("We've been by your side for 20 years, are you going to trust us or some Silicon Valley hipster? Also we employ more Kubernetes contributors than anyone else"). That's the real battle, in my experience on the open-source side you'll find mostly engineers from all side collaborating peacefully and building whatever they need to get their job done.

medyadaily 8y ago

Sweet, This is truly great, I was hoping for a service like this for a long time. being able to build images without root privilege!

amouat 8y ago
Kaniko does run in the container as root, but the container doesn't need to be granted any extra privileges when run (you don't need the equivalent of Docker's --privileged flag).

zapita 8y ago

Does this (or could this) use Buildkit? It seems that Docker themselves are encouraging the development of an ecosystem of third-party container build tools, with buildkit as an interoperability layer. I heard good things about buildkit but haven't tried it yet. If Kaniko authors are reading this: have you considered buildkit and, if not, would you be open to contributions based on it? My understanding is that the official 'docker build' itself is based on Buildkit. https://github.com/moby/buildkit https://github.com/moby/buildkit

dlor 8y ago
Kaniko doesn't use buildkit - buildkit still uses containerd/runC under the hood so it can't run inside a container easily. We are looking at interoperability with buildkit (and the large set of of other tooling like this) through the CBI: https://github.com/containerbuilding/cbi https://github.com/containerbuilding/cbi which aims to be a neutral interface on top of things like buildkit, buildah, docker and kaniko that build images. Discloser: I work on kaniko and other container things at Google.
- zapita 8y ago
  Very interesting, and it looks like the people working on CBI are also active on Buildkit, which is a good sign! Thank you for the pointer.
  - AkihiroSuda 8y ago
    @zapita FYI: Kaniko plugin for CBI is now available. https://github.com/containerbuilding/cbi/pull/35 https://github.com/containerbuilding/cbi/pull/35
    - zapita 8y ago
      Thanks Akihiro for the follow-up. And while I'm at it, thank you for all this excellent code that we get to enjoy for free! It's really fantastic work.

bkeroack 8y ago

Interesting! We built and use a service called Furan: https://github.com/dollarshaveclub/furan https://github.com/dollarshaveclub/furan That said, Furan isn't suitable for untrusted Dockerfiles (or multi-tenant environments) exactly due to the security implications of access to the Docker engine socket. The issue I see with Kaniko is drift from upstream Moby/Docker syntax. One of the strengths with Furan is that you have the guarantee that the Docker build you perform locally is exactly what happens by the service. When you can't make this guarantee you get into weird situations where "the build works for me locally" but there's some issue when doing a remote build. That's also why we've resisted putting special build magic into Furan (like injecting metadata into the build context, for example).

tyrankh 8y ago

This is great work. Github link for the lazy: https://github.com/GoogleCloudPlatform/kaniko https://github.com/GoogleCloudPlatform/kaniko

ksajadi 8y ago

This is a great tool. Wish it could work with build workflow tools like Habitus (http://www.habitus.io http://www.habitus.io)

spockz 8y ago

My understanding is that it is best practice to run your docker builds and images as s non root user. OpenShift will complain if you do for example. Now this kaniko image runs the build as root contrary to the recommendation and the post explicitly mentions this difference with Orca. Why is it okay now for kaniko to run as root user?

humbleMouse 8y ago
This was my first thought as well.