5 ms·

> # install chromedriver > RUN apk update > RUN apk add chromium chromium-chromedriver These kind of poor examples lead to a huge amount of waste when using

by orf 7y ago

> # install chromedriver > RUN apk update > RUN apk add chromium chromium-chromedriver These kind of poor examples lead to a huge amount of waste when using Docker because people learning it are not taught about how layers interact, leading to ridiculous things like 'COPY' followed by 'RUN chown'. Layers are a _core_ part of how Docker works, why not make this example "correct" by doing: > RUN apk --no-cache add chromium chromium-chromedriver Then just a comment like this: "it's important to group layers if possible to reduce your image size. By using `--no-cache` apk will update at the same time as installing".

deleted 7y ago
[deleted]
sneak 7y ago
This article also has an image that shows containers running “on top of” docker; that’s not how any of this works. Docker is just the daemon that sets up the namespaces and runs the containers. It’s not some magical runtime, any more than “iptables” is. It just talks to the kernel. I think this post is attempting to oversimplify, and is doing a bad job of teaching in the process.
- kortex 7y ago
  Running Docker containers is definitely on top of the Docker runtime. Maybe not from a PID parent point of view, but absolutely from an abstraction point of view. Try doing anything hardware-network-topology relevant stuff without `network=host`, like running a ROS cluster with direct ethernet connections between hosts and cameras. It's definitely "on top" in a way bare-metal isn't. Same way that a vm is "on top of" kvm even though the cpu supports hardware virtualization. Nvidia-docker is another abstraction. It's a pass-through. It's not the same as bare-metal. Certainly feels the same 98% of the time, but in that 2% it is very much apparent there is a layer in between.
3fe9a03ccd14ca5 7y ago
The article is a “what is docker?” and This is a perfectly fine way to use docker. There’s always room for optimizing layers in the future. Always start with the basics rather than overwhelming people with details.
- orf 7y ago
  The point I was making is why the following snippet doesn't make your image smaller _is the basics_: > RUN echo something > some_file > RUN rm some_file Neglecting to even mention it leads to issues down the line because people don't get the fundamentals. There are a lot more confusing things than this when working with Docker.
  - 3fe9a03ccd14ca5 7y ago
    Small images is one component of optimizing a container. I disagree that a small image is fundamental. In fact I’d say it’s largely a small optimization detail that has very little real world impact. Really nobody really cares about container sizes except for their own efficiency. We run thousands of containers. Many of them tens of GB. They could all be optimized to a small size but it really doesn’t matter and is not worth the engineering time most of the time.
    - orf 7y ago
      It's not necessary about the size, and nor is it always a premature/useless optimization. It's about knowing how layers interact which is important. And not hard to teach either. And yes, if you're adding 5+ GB to each image build through something silly like a 'RUN chown' it's worth fixing. Maybe not complex refactoring, sure, but we are talking about the basics here.
      - hansvm 7y ago
        I think their point is that for most use cases, large images don't have a sizeable real world impact. They're more expensive to move around, and you can only fit so many large, distinct layers on a single machine, but a moderately sized organization will barely notice the cost.
        adgud 7y ago
        They do have a sizeable real world impact. Moderately sized organizations usually build and store the images in a public cloud, which in turn do not provide any straightforward registry cleaning capabilities (at least I didn't find any in AWS or GCP). This leads to hundreds and thousands of gigabytes being wasted and money being flushed down the drain. Not to mention that network traffic in public clouds can quickly rack up the bill as well. This Docker and Kubernetes hype and the fact that few people understand and care to implement these technologies correctly fills Amazon's and Google's pockets with money that could be spent elsewhere.
fnord77 7y ago
I wish Dockerfiles allowed you to group commands so that they would end up in the same layer, so you don't have to do something ridiculous like RUN cmd1 && cmd2 && cmd3 && cmd4 maybe with brackets: # 1 layer { RUN cmd1 COPY someshit RUN cmd2 -lots -of -long flags ... }
- IanCal 7y ago
  You might be interested in this: https://github.com/uber/makisu https://github.com/uber/makisu I've not used it but it's on my list to try out, the explicit caching I think ties in with what you're after: * Requires no elevated privileges or containerd/Docker daemon, making the build process portable. * Uses a distributed layer cache to improve performance across a build cluster. * Provides control over generated layers with a new optional keyword #!COMMIT, reducing the number of layers in images. * Is Docker compatible. Note, the Dockerfile parser in Makisu is opinionated in some scenarios. More details can be found here.
  - IanCal 7y ago
    I can't edit any more, but I'd really be interested in people's experiences with it.
- fctorial 7y ago
  Why do you want to avoid multiple layers. Performance?
  - lbotos 7y ago
    Size of the resulting container is my understanding.
  - orf 7y ago
    Build performance, image size and deployment speed mostly.
  - kortex 7y ago
    Each layer is a write only operation. So if you `RUN some_build.sh` and then `RUN rm junk`, you basically do nothing to reduce the final image size. So you see a lot of like `RUN some_build.sh && rm -rf junk`. There's some overhead with each layer as well.
karakanb 7y ago
Hey there, OP here. Thanks a lot for the feedback. I believe simplicity at this phase of learning is much more valuable than efficiency, that's why I haven't tried to produce the most efficient images as examples. Since your suggestion also simplifies the Dockerfile itself, I have replaced the previous one with your suggestion, thanks!
- robbyoconnor 7y ago
  You do realize that bad habits tend to not die, right? Don't teach bad practices because you're targeting newbies...teach them the best practices from the start.
  - karakanb 7y ago
    Agreed, I didn't mean to teach bad practices to anyone, I am just trying to put simplicity in front of efficiency so that people can grasp the technology better and benefit from that.
londons_explore 7y ago
That is more the fault of docker than the user. Docker should be smart enough to spot files that exist transiently only for a few layers in a final image, and then maybe decide to never store those intermediate layers (always regenerating them on demand, or storing them only in some less-permanent form)

by orf 7y ago

deleted 7y ago

[deleted]

sneak 7y ago

This article also has an image that shows containers running “on top of” docker; that’s not how any of this works. Docker is just the daemon that sets up the namespaces and runs the containers. It’s not some magical runtime, any more than “iptables” is. It just talks to the kernel. I think this post is attempting to oversimplify, and is doing a bad job of teaching in the process.

kortex 7y ago
Running Docker containers is definitely on top of the Docker runtime. Maybe not from a PID parent point of view, but absolutely from an abstraction point of view. Try doing anything hardware-network-topology relevant stuff without `network=host`, like running a ROS cluster with direct ethernet connections between hosts and cameras. It's definitely "on top" in a way bare-metal isn't. Same way that a vm is "on top of" kvm even though the cpu supports hardware virtualization. Nvidia-docker is another abstraction. It's a pass-through. It's not the same as bare-metal. Certainly feels the same 98% of the time, but in that 2% it is very much apparent there is a layer in between.

3fe9a03ccd14ca5 7y ago

The article is a “what is docker?” and This is a perfectly fine way to use docker. There’s always room for optimizing layers in the future. Always start with the basics rather than overwhelming people with details.

orf 7y ago
The point I was making is why the following snippet doesn't make your image smaller _is the basics_: > RUN echo something > some_file > RUN rm some_file Neglecting to even mention it leads to issues down the line because people don't get the fundamentals. There are a lot more confusing things than this when working with Docker.
- 3fe9a03ccd14ca5 7y ago
  Small images is one component of optimizing a container. I disagree that a small image is fundamental. In fact I’d say it’s largely a small optimization detail that has very little real world impact. Really nobody really cares about container sizes except for their own efficiency. We run thousands of containers. Many of them tens of GB. They could all be optimized to a small size but it really doesn’t matter and is not worth the engineering time most of the time.
  - orf 7y ago
    It's not necessary about the size, and nor is it always a premature/useless optimization. It's about knowing how layers interact which is important. And not hard to teach either. And yes, if you're adding 5+ GB to each image build through something silly like a 'RUN chown' it's worth fixing. Maybe not complex refactoring, sure, but we are talking about the basics here.
    - hansvm 7y ago
      I think their point is that for most use cases, large images don't have a sizeable real world impact. They're more expensive to move around, and you can only fit so many large, distinct layers on a single machine, but a moderately sized organization will barely notice the cost.
      - adgud 7y ago
        They do have a sizeable real world impact. Moderately sized organizations usually build and store the images in a public cloud, which in turn do not provide any straightforward registry cleaning capabilities (at least I didn't find any in AWS or GCP). This leads to hundreds and thousands of gigabytes being wasted and money being flushed down the drain. Not to mention that network traffic in public clouds can quickly rack up the bill as well. This Docker and Kubernetes hype and the fact that few people understand and care to implement these technologies correctly fills Amazon's and Google's pockets with money that could be spent elsewhere.

fnord77 7y ago

I wish Dockerfiles allowed you to group commands so that they would end up in the same layer, so you don't have to do something ridiculous like RUN cmd1 && cmd2 && cmd3 && cmd4 maybe with brackets: # 1 layer { RUN cmd1 COPY someshit RUN cmd2 -lots -of -long flags ... }

IanCal 7y ago
You might be interested in this: https://github.com/uber/makisu https://github.com/uber/makisu I've not used it but it's on my list to try out, the explicit caching I think ties in with what you're after: * Requires no elevated privileges or containerd/Docker daemon, making the build process portable. * Uses a distributed layer cache to improve performance across a build cluster. * Provides control over generated layers with a new optional keyword #!COMMIT, reducing the number of layers in images. * Is Docker compatible. Note, the Dockerfile parser in Makisu is opinionated in some scenarios. More details can be found here.
- IanCal 7y ago
  I can't edit any more, but I'd really be interested in people's experiences with it.
fctorial 7y ago
Why do you want to avoid multiple layers. Performance?
- lbotos 7y ago
  Size of the resulting container is my understanding.
- orf 7y ago
  Build performance, image size and deployment speed mostly.
- kortex 7y ago
  Each layer is a write only operation. So if you `RUN some_build.sh` and then `RUN rm junk`, you basically do nothing to reduce the final image size. So you see a lot of like `RUN some_build.sh && rm -rf junk`. There's some overhead with each layer as well.

karakanb 7y ago

Hey there, OP here. Thanks a lot for the feedback. I believe simplicity at this phase of learning is much more valuable than efficiency, that's why I haven't tried to produce the most efficient images as examples. Since your suggestion also simplifies the Dockerfile itself, I have replaced the previous one with your suggestion, thanks!

robbyoconnor 7y ago
You do realize that bad habits tend to not die, right? Don't teach bad practices because you're targeting newbies...teach them the best practices from the start.
- karakanb 7y ago
  Agreed, I didn't mean to teach bad practices to anyone, I am just trying to put simplicity in front of efficiency so that people can grasp the technology better and benefit from that.

londons_explore 7y ago

That is more the fault of docker than the user. Docker should be smart enough to spot files that exist transiently only for a few layers in a final image, and then maybe decide to never store those intermediate layers (always regenerating them on demand, or storing them only in some less-permanent form)