7 ms·

Anti-patterns when building container images

159 points by henrik_w 5y ago

BiteCode_dev 5y ago
Years ago, freshly out of school, I ended up in a job at a french small shop that required me to learn a lot of things. They had a brillant sysadmin, that I bothered to explain them to me. Sometimes, he'll come to my desk, looking at my screen, typing the solution while explaining the problem to me with simple words. Standing up, typing with one hand, and working on what was supposed to be my specialty, he was doing a better job than I was, despite me interrupting his workflow. I left the job, and levelled up. One day, I read an article about docker, and the name of the dev caught my eyes: it was the friendly former sysadmin colleague, Jérôme Petazzoni, who went on becomming docker's biggest evangelist. Thank you Jérôme, for making sure I started my carrer by not being the smartest guy in the room. I kept it as a habit. Or I'm just dumb :)
- klibertp 5y ago
  > for making sure I started my carrer by not being the smartest guy in the room That's indeed invaluable. That should be the first advice to anyone starting out: if you find yourself being the smartest in a room, go find a new room. Right now.
  - arch-ninja 5y ago
    I left the company I was part-timing at because after college I realized I was the smartest one in the room; my boss thought some of the code I wrote was "brilliant" and for this reason did not let me improve/fix any of it, which led to the company shipping a broken product to some very annoyed customers. If you are ever managing someone who is "your best employee" and they want to do something you think is useless, please ask them why and what the expected results will be. Also when you get that data, listen a little? If priorities clash do a balancing act: "If you can do X in Y time, do it. But if day D comes along, switch over to my priority".
  - throwaway894345 5y ago
    I graduated college and started working at a local tractor manufacturer. In hindsight, this company had no idea how to make software. For instance, they required that all "software engineers" were from an accredited engineering program--at the time, none of the universities in the area had an accredited software engineering program, so they would hire electrical engineering majors who had no desire to write software but were looking for a way to get their foot in the door. Most of the EE majors had a single semester of C and that was their entire software qualification; however, I double majored in EE and CS and my passion was software. That said, when I graduated I didn't understand any of this--I just figured there was this successful company and there was so much I didn't know about software, so I thought I would learn loads there--and I did learn loads, but it was mostly about what not to do. One of my first projects was to take over an embedded C++/Qt project that had been bid out to some contractors in the Ukraine. The software was super buggy and the code was immensely convoluted. Test coverage was part of the contract, so it had tests, but even still, no one could change anything without breaking everything. It took me weeks of plotting the code paths to figure out that there were many paths through the code that were only exercised by tests--literally they were just trying to boost their code coverage metrics by adding code that was unreachable at production. I guess it didn't occur to me that professional software engineers would be so bad at writing code, and that a large and well-known software company would hire them. Thankfully, after eventually gaining my entirely-nontechnical manager's trust, we were able to bring in a consultancy who helped me demonstrate that it would be no more than half the cost to rewrite the code rather than try to fix all of the existing bugs even if we never found other bugs or introduced a single bug of our own. We did the rewrite and it was successful, but I was beginning to realize that in some overly simplistic sense, I was the "smartest" one in the room as it pertained to software development anyway. Ten years later, when I talk to my peers who stayed, many of them have been writing embedded C programs and various MatLab models and they still don't understand pointers or dynamic memory or git or anything outside of their very local niche. Which would be fine if they were happy, but now many want to move to another company but they lack the skills and confidence to get anything more than an entry level job at many other software companies--so much of their compensation is derived from their knowledge of their employer's business and processes rather than their technical acumen, so they would likely have to take a significant pay cut to move to another employer. Anyway, this is already a very long post despite oversimplifying many things, but I hope this illustrates what it looks like to be the smartest person in the room and why it's not good.
kator 5y ago
When I read people complaining about "Small Images" like mentioned here where the author seems to warn against it and even proposes making sure enough "tooling" is in the container. I'm always a bit curious why more people don't talk about using nsenter in the rare situation they need to inspect stuff in a running container? For a production microservice container, why have a bunch of bloat in the container that increases the surface area to manage while also only actually being used in edge cases?
- foxfluff 5y ago
  How do you use nsenter efficiently for running more than one program? You might need dozens of tools (and all their dependencies) to diagnose and fix issues.
  - yjftsjthsd-h 5y ago
    If I understand your question correctly, you just make sure that the first command is nsenter bash
- Lhiw 5y ago
  I aim for "small images" because downloading a full install of Ubuntu is rediculous on anything but gigabit fibre. But I don't aim for minimal images. Using alpine and installing what ever you like without worrying about all the bullshit to make caches stay empty still results in images only a few megs, don't need to do much better than that. Though I tend to agree, if you're talking about a compiled language. Static binary and there is little need for anything other than `from scratch`.
- hvgk 5y ago
  I shall look into nsenter. I have never actually used it. It may cover this use case. As for the status quo, I spend a lot of time extracting dumps from things in containers and debugging weird issues which is really really difficult without the ability to install any tools on the container at will. When done, we blast the container and let kubernetes reschedule a new one, disposing of all tools and state. On bloat, mine are sitting on top of a debian container layer, mostly because it’s a hell of a lot easier dealing with some dynamic libraries than fecking around with static linking stuff and copying dependencies into a minimalist container. Edit: There is overlap between these two things which nsenter may not be able to solve looking at what I am reading.
ithkuil 5y ago
You can run a small sidecar with a shell and if you enable shared process namespace you can access the filesystem of the main "distroless" container via /proc (e.g. /proc/123/root)
- BossingAround 5y ago
  I think the difficulty arises in the fact that you can't spawn the sidecar without terminating a pod, as far as I know. So you either create a sidecar for every pod, "just in case", or you have to restart your pod for debugging purposes (and modify your YAML files). That is, if we're talking Kubernetes of course.
  - ithkuil 5y ago
    There is an alpha feature: https://kubernetes.io/docs/concepts/workloads/pods/ephemeral-containers/ https://kubernetes.io/docs/concepts/workloads/pods/ephemeral... Otherwise you have to prepare in advance and add a small sidecar to all pods that need it
deleted 5y ago
[deleted]
machinerychorus 5y ago
This is a great article, I just have one tiny nitpick: please add the publication date! Information like this is very dependent on the context of when it was written, and something that is "best practice" or "an anti-pattern" will not always be that way. Adding a publication date and/or a "last updated" date to the top of the article is very helpful for readers to align the information in their minds. EDIT: I realized just after posting this comment that the publication date is in the url. Time for that second cup of coffee :)
- remram 5y ago
  It's also in the HTML but commented out. Hopefully the author is fixing their markup and it's coming back?
staticassertion 5y ago
The small image problem seems like it may be best solved through volumes. Consider that busybox is just under 1MB when dynamically linked and installed. That's roughly 50% larger than a distroless container. Further, distroless doesn't just target size for size's sake - it's also to ensure that an attacker in the container doesn't have tooling that they like to use. Instead if your containers are mounting a volume, say `/var/run/debug-tools/`, you could copy the tools into that mount before entering your containers. This probably has a good dev UX in that you can enter containers and have your tools, it amortizes the cost across your containers on a given host, and it isn't container specific ie: you perform one copy and now all containers on that host have it.
returningfory2 5y ago
RE small images: advice like this always seems to ignore that Docker images are cached incrementally. If you build, say, a Go binary and then put it in a popular stock Debian or Alpine Docker image, pushing the final image to a registry will only transfer the diff over the network. The size of the diff is basically the compressed Go binary and doesn’t depend on the base image you’re using! Then when you pull the Docker image onto a machine it’s likely the base image is already in cache so again it’s only the diff that is transferred. It’s sometimes said this Alpine stuff is a premature optimization, but in most cases it’s not even an optimization.
- varikin 5y ago
  > It’s sometimes said this Alpine stuff is a premature optimization, but in most cases it’s not even an optimization. Not always. If you have thousands of containers running, the network cost* for changing the base image is massive because each instance needs possibly all new layers, not just the last layer or 2. Having a smaller base image is very practical at this point. * And for network cost, there is both the literal cost with egress from dockerhub, or another registry, but also the load on your network infra.
- p_l 5y ago
  Actually using that caching in a way that benefits you is, well, very hard. Especially since a lot of the time you have a lot of stuff included in order to start the final program, and every step above the base image will result in different layer that will be separate and probably not cacheable Also, Alpine images are used for being, well, small in general - there's not much to exploit on such image, nor is there going to be a lot of unplanned-for data, and especially it will speed up deployment when you upload a new image. It takes special, dedicated CI/CD work to have common base images to exploit sharing of layers properly.
- justin_oaks 5y ago
  While that's true, you or your company have to standardize on a base image to get benefit of shared image layers. In my job, I rarely have an exact match on the base layer because I'll use images from Docker Hub. Although many of those images use the same small set of distros (Alpine, Debian, or Ubuntu), any given image is likely from a different base version of that distro, thus there is no common base image. There may be differences in the base image even if you use the same tag since the same image may change over time. Thus if you pull debian:bullseye today, and then pull again tomorrow, you may have two different images. Generally I don't bother trying to get a common base image and I'm happy enough just attempt to lower the overall size of my images.
musicale 5y ago
TL;DR: using Docker is an anti-pattern > Big images > Requiring rebuilds for every single change > Forcing things to run in containers > Using overly complex tools > Building with Dockerfiles
- justin_oaks 5y ago
  While I won't address each item in your list. I'll say that I love Dockerfiles. They make it clear what steps occurred to arrive at the resulting image. For any image that is built from an open-source repository, I can find the Dockerfile, look at it and say "Oh, that's how they did it!" In the article it says that Dockerfiles might not be the best way to build your image. I think some of that may have come from before the days of multi-stage builds. With multi-stage builds you can use a lot of tools (like the Go compiler) at build time without including those tools in your final build. Can we do better than Dockerfiles as they are today? I'm sure we can, but can I appreciate what I have while knowing it could be better.
  - musicale 5y ago
    The listed points (>) are quoted from the article.
- BossingAround 5y ago
  Docker containers are easy to build in a way that's pretty bad. I don't think that necessarily implicates Docker containers as a bad tech. In the same way, you can write an insert language here application that's horribly unmaintainable, and that tells you nothing of the language. I don't like Docker as a company, but I can't imagine living my tech life without containers + Kubernetes, and Dockerfiles are really easy to write and read.
  - musicale 5y ago
    Exactly - for most people, "using docker" is equivalent to invoking these anti-patterns, usually because it is the path of least resistance. Unfortunately avoiding the anti-patterns seems to be much harder than adopting them.
IshKebab 5y ago
Good list. I would add that using Docker in the first place is an anti-pattern. If you're using software you wrote yourself that can be made self-contained, e.g. if it's written in Go or Rust, then you probably don't even need Docker. People always say "but how can you isolate the network?" and use Kurbernetes and .. and sure Docker is the easiest way to do that. But in my experience people rarely use Docker for that feature anyway. 99% of Docker use is because most Linux software is written in such a way that it is an absolute pain in the arse to distribute so shipping a whole OS is the bodgy solution that everyone accepts.
throwaway894345 5y ago
> Or, if you use Docker, you can use docker cp to copy some static tools (e.g. busybox) to a running container and check what’s going on. I tried this recently with `kubectl cp` for my scratch image and it turns out that command will fail if the target container lacks a `tar` program in its PATH. I guess under the covers it's just doing a `kubectl exec <pod> -- /bin/tar ...` on the file rather than asking the kubelet to untar the file into the target container's filesystem.
bob332 5y ago
This website ui is an anti pattern

159 points by henrik_w 5y ago

Years ago, freshly out of school, I ended up in a job at a french small shop that required me to learn a lot of things. They had a brillant sysadmin, that I bothered to explain them to me. Sometimes, he'll come to my desk, looking at my screen, typing the solution while explaining the problem to me with simple words. Standing up, typing with one hand, and working on what was supposed to be my specialty, he was doing a better job than I was, despite me interrupting his workflow. I left the job, and levelled up. One day, I read an article about docker, and the name of the dev caught my eyes: it was the friendly former sysadmin colleague, Jérôme Petazzoni, who went on becomming docker's biggest evangelist. Thank you Jérôme, for making sure I started my carrer by not being the smartest guy in the room. I kept it as a habit. Or I'm just dumb :)

klibertp 5y ago
> for making sure I started my carrer by not being the smartest guy in the room That's indeed invaluable. That should be the first advice to anyone starting out: if you find yourself being the smartest in a room, go find a new room. Right now.
- arch-ninja 5y ago
  I left the company I was part-timing at because after college I realized I was the smartest one in the room; my boss thought some of the code I wrote was "brilliant" and for this reason did not let me improve/fix any of it, which led to the company shipping a broken product to some very annoyed customers. If you are ever managing someone who is "your best employee" and they want to do something you think is useless, please ask them why and what the expected results will be. Also when you get that data, listen a little? If priorities clash do a balancing act: "If you can do X in Y time, do it. But if day D comes along, switch over to my priority".
- throwaway894345 5y ago
  I graduated college and started working at a local tractor manufacturer. In hindsight, this company had no idea how to make software. For instance, they required that all "software engineers" were from an accredited engineering program--at the time, none of the universities in the area had an accredited software engineering program, so they would hire electrical engineering majors who had no desire to write software but were looking for a way to get their foot in the door. Most of the EE majors had a single semester of C and that was their entire software qualification; however, I double majored in EE and CS and my passion was software. That said, when I graduated I didn't understand any of this--I just figured there was this successful company and there was so much I didn't know about software, so I thought I would learn loads there--and I did learn loads, but it was mostly about what not to do. One of my first projects was to take over an embedded C++/Qt project that had been bid out to some contractors in the Ukraine. The software was super buggy and the code was immensely convoluted. Test coverage was part of the contract, so it had tests, but even still, no one could change anything without breaking everything. It took me weeks of plotting the code paths to figure out that there were many paths through the code that were only exercised by tests--literally they were just trying to boost their code coverage metrics by adding code that was unreachable at production. I guess it didn't occur to me that professional software engineers would be so bad at writing code, and that a large and well-known software company would hire them. Thankfully, after eventually gaining my entirely-nontechnical manager's trust, we were able to bring in a consultancy who helped me demonstrate that it would be no more than half the cost to rewrite the code rather than try to fix all of the existing bugs even if we never found other bugs or introduced a single bug of our own. We did the rewrite and it was successful, but I was beginning to realize that in some overly simplistic sense, I was the "smartest" one in the room as it pertained to software development anyway. Ten years later, when I talk to my peers who stayed, many of them have been writing embedded C programs and various MatLab models and they still don't understand pointers or dynamic memory or git or anything outside of their very local niche. Which would be fine if they were happy, but now many want to move to another company but they lack the skills and confidence to get anything more than an entry level job at many other software companies--so much of their compensation is derived from their knowledge of their employer's business and processes rather than their technical acumen, so they would likely have to take a significant pay cut to move to another employer. Anyway, this is already a very long post despite oversimplifying many things, but I hope this illustrates what it looks like to be the smartest person in the room and why it's not good.

kator 5y ago

When I read people complaining about "Small Images" like mentioned here where the author seems to warn against it and even proposes making sure enough "tooling" is in the container. I'm always a bit curious why more people don't talk about using nsenter in the rare situation they need to inspect stuff in a running container? For a production microservice container, why have a bunch of bloat in the container that increases the surface area to manage while also only actually being used in edge cases?

foxfluff 5y ago
How do you use nsenter efficiently for running more than one program? You might need dozens of tools (and all their dependencies) to diagnose and fix issues.
- yjftsjthsd-h 5y ago
  If I understand your question correctly, you just make sure that the first command is nsenter bash
Lhiw 5y ago
I aim for "small images" because downloading a full install of Ubuntu is rediculous on anything but gigabit fibre. But I don't aim for minimal images. Using alpine and installing what ever you like without worrying about all the bullshit to make caches stay empty still results in images only a few megs, don't need to do much better than that. Though I tend to agree, if you're talking about a compiled language. Static binary and there is little need for anything other than `from scratch`.
hvgk 5y ago
I shall look into nsenter. I have never actually used it. It may cover this use case. As for the status quo, I spend a lot of time extracting dumps from things in containers and debugging weird issues which is really really difficult without the ability to install any tools on the container at will. When done, we blast the container and let kubernetes reschedule a new one, disposing of all tools and state. On bloat, mine are sitting on top of a debian container layer, mostly because it’s a hell of a lot easier dealing with some dynamic libraries than fecking around with static linking stuff and copying dependencies into a minimalist container. Edit: There is overlap between these two things which nsenter may not be able to solve looking at what I am reading.

ithkuil 5y ago

You can run a small sidecar with a shell and if you enable shared process namespace you can access the filesystem of the main "distroless" container via /proc (e.g. /proc/123/root)

BossingAround 5y ago
I think the difficulty arises in the fact that you can't spawn the sidecar without terminating a pod, as far as I know. So you either create a sidecar for every pod, "just in case", or you have to restart your pod for debugging purposes (and modify your YAML files). That is, if we're talking Kubernetes of course.
- ithkuil 5y ago
  There is an alpha feature: https://kubernetes.io/docs/concepts/workloads/pods/ephemeral-containers/ https://kubernetes.io/docs/concepts/workloads/pods/ephemeral... Otherwise you have to prepare in advance and add a small sidecar to all pods that need it

deleted 5y ago

[deleted]

machinerychorus 5y ago

This is a great article, I just have one tiny nitpick: please add the publication date! Information like this is very dependent on the context of when it was written, and something that is "best practice" or "an anti-pattern" will not always be that way. Adding a publication date and/or a "last updated" date to the top of the article is very helpful for readers to align the information in their minds. EDIT: I realized just after posting this comment that the publication date is in the url. Time for that second cup of coffee :)

remram 5y ago
It's also in the HTML but commented out. Hopefully the author is fixing their markup and it's coming back?

staticassertion 5y ago

The small image problem seems like it may be best solved through volumes. Consider that busybox is just under 1MB when dynamically linked and installed. That's roughly 50% larger than a distroless container. Further, distroless doesn't just target size for size's sake - it's also to ensure that an attacker in the container doesn't have tooling that they like to use. Instead if your containers are mounting a volume, say `/var/run/debug-tools/`, you could copy the tools into that mount before entering your containers. This probably has a good dev UX in that you can enter containers and have your tools, it amortizes the cost across your containers on a given host, and it isn't container specific ie: you perform one copy and now all containers on that host have it.

returningfory2 5y ago

RE small images: advice like this always seems to ignore that Docker images are cached incrementally. If you build, say, a Go binary and then put it in a popular stock Debian or Alpine Docker image, pushing the final image to a registry will only transfer the diff over the network. The size of the diff is basically the compressed Go binary and doesn’t depend on the base image you’re using! Then when you pull the Docker image onto a machine it’s likely the base image is already in cache so again it’s only the diff that is transferred. It’s sometimes said this Alpine stuff is a premature optimization, but in most cases it’s not even an optimization.

varikin 5y ago
> It’s sometimes said this Alpine stuff is a premature optimization, but in most cases it’s not even an optimization. Not always. If you have thousands of containers running, the network cost* for changing the base image is massive because each instance needs possibly all new layers, not just the last layer or 2. Having a smaller base image is very practical at this point. * And for network cost, there is both the literal cost with egress from dockerhub, or another registry, but also the load on your network infra.
p_l 5y ago
Actually using that caching in a way that benefits you is, well, very hard. Especially since a lot of the time you have a lot of stuff included in order to start the final program, and every step above the base image will result in different layer that will be separate and probably not cacheable Also, Alpine images are used for being, well, small in general - there's not much to exploit on such image, nor is there going to be a lot of unplanned-for data, and especially it will speed up deployment when you upload a new image. It takes special, dedicated CI/CD work to have common base images to exploit sharing of layers properly.
justin_oaks 5y ago
While that's true, you or your company have to standardize on a base image to get benefit of shared image layers. In my job, I rarely have an exact match on the base layer because I'll use images from Docker Hub. Although many of those images use the same small set of distros (Alpine, Debian, or Ubuntu), any given image is likely from a different base version of that distro, thus there is no common base image. There may be differences in the base image even if you use the same tag since the same image may change over time. Thus if you pull debian:bullseye today, and then pull again tomorrow, you may have two different images. Generally I don't bother trying to get a common base image and I'm happy enough just attempt to lower the overall size of my images.

musicale 5y ago

TL;DR: using Docker is an anti-pattern > Big images > Requiring rebuilds for every single change > Forcing things to run in containers > Using overly complex tools > Building with Dockerfiles

justin_oaks 5y ago
While I won't address each item in your list. I'll say that I love Dockerfiles. They make it clear what steps occurred to arrive at the resulting image. For any image that is built from an open-source repository, I can find the Dockerfile, look at it and say "Oh, that's how they did it!" In the article it says that Dockerfiles might not be the best way to build your image. I think some of that may have come from before the days of multi-stage builds. With multi-stage builds you can use a lot of tools (like the Go compiler) at build time without including those tools in your final build. Can we do better than Dockerfiles as they are today? I'm sure we can, but can I appreciate what I have while knowing it could be better.
- musicale 5y ago
  The listed points (>) are quoted from the article.
BossingAround 5y ago
Docker containers are easy to build in a way that's pretty bad. I don't think that necessarily implicates Docker containers as a bad tech. In the same way, you can write an insert language here application that's horribly unmaintainable, and that tells you nothing of the language. I don't like Docker as a company, but I can't imagine living my tech life without containers + Kubernetes, and Dockerfiles are really easy to write and read.
- musicale 5y ago
  Exactly - for most people, "using docker" is equivalent to invoking these anti-patterns, usually because it is the path of least resistance. Unfortunately avoiding the anti-patterns seems to be much harder than adopting them.

IshKebab 5y ago

Good list. I would add that using Docker in the first place is an anti-pattern. If you're using software you wrote yourself that can be made self-contained, e.g. if it's written in Go or Rust, then you probably don't even need Docker. People always say "but how can you isolate the network?" and use Kurbernetes and .. and sure Docker is the easiest way to do that. But in my experience people rarely use Docker for that feature anyway. 99% of Docker use is because most Linux software is written in such a way that it is an absolute pain in the arse to distribute so shipping a whole OS is the bodgy solution that everyone accepts.

throwaway894345 5y ago

> Or, if you use Docker, you can use docker cp to copy some static tools (e.g. busybox) to a running container and check what’s going on. I tried this recently with `kubectl cp` for my scratch image and it turns out that command will fail if the target container lacks a `tar` program in its PATH. I guess under the covers it's just doing a `kubectl exec <pod> -- /bin/tar ...` on the file rather than asking the kubelet to untar the file into the target container's filesystem.

bob332 5y ago

This website ui is an anti pattern