5 ms·

I think the importance of small Docker images is generally oversold. I regularly deploy multi-GB images on Google Cloud, and startup even on a fresh node only t

I think the importance of small Docker images is generally oversold. I regularly deploy multi-GB images on Google Cloud, and startup even on a fresh node only takes 60 seconds or so. If the node's already hot (i.e. has cached the image), starting a container takes no more than a few seconds. I think what's more important is layering the Dockerfile in the right order. You should be putting most of your large, infrequently changing assets in the lowest layers. Then put smaller, more frequently changing assets in the top layer. If you have a 4GB image, but only change the top 10MB layer, then it only requires caching 10MB of new data when you update the container. But if you change a lower layer, then it requires re-building and re-caching everything above it.

itamarst 7y ago
Multi-GB images still seem... big. Though to be fair in many situations those 60 seconds are meaningless, so might not be worth the time to optimize. Often images that large are the result of including build toolchains, which you can omit with multi-stage builds (e.g. for Python https://pythonspeed.com/articles/multi-stage-docker-python/ https://pythonspeed.com/articles/multi-stage-docker-python/.
tybit 7y ago
I agree it’s oversold for most use cases, clouds do have pretty decent networking after all. Having said that as an Australian with sub par internet at home, I do appreciate not making images gratuitously large. Just because SDK images won’t ever be deployed to prod doesn’t mean you shouldn’t strip out unnecessary junk.
- igor47 7y ago
  Similarly, just yesterday I had to go home from the cafe I was working in because I had to push a docker image. The wifi just wasn't fast enough for that, though plenty fast for googling documentation
cosmotic 7y ago
Adding a minute to some deployment process can be fairly significant. The docker push and pull is probably 70% of my deployment process and I already have ~100MB or less images. I don't think the concern is 'how long does deployment take' but 'how fast can we iterate?'; Building and loading the images on a local dev machine to test a 2 second change would take much longer with larger images. Getting feedback of a PR merge from a CI build agent would take minutes longer. I don't think the importance has been oversold.
- dcolkitt 7y ago
  > I don't think the concern is 'how long does deployment take' but 'how fast can we iterate? And for iteration, the only thing that matters is the size of the top layers that are being iterated on. Not the overall image size itself. You can put `RUN apt-get [kitchen sink]` at the beginning of the Dockerfile and it pretty much won't matter. When you change anything in the project repository, that bigass giant base layer doesn't get re-pulled because it doesn't change. To validate that a layer, Docker daemon just compares the hashes. So, for unmodified layers, docker pull is constant time with image size. The Docker daemon only downloads from the registry starting with the bottom-most modified layer relative to its cache. If anything, throwing the kitchen sink in the base layer is better for fast iteration. When you're being parsimonious about third-party libraries and packages, you'll frequently have to rebuild the base image. If you `RUN apt-get [everything]`, then you'll hardly ever never need to rebuild/re-push/re-pull that layer, because you'll always have whatever you need already available.
  - echevil 7y ago
    It depends on the size of project and what you want to optimize. The time spent on push and pull of docker images used to be a real deal breaker once upon a time when I tried to improve the deployment speed of a project by switching from heroku build to docker based solution
  - cosmotic 7y ago
    In a fresh pod/vm it still needs to pull all the layers, even the big, old one.
paulddraper 7y ago
> startup even on a fresh node only takes 60 seconds or so There are very complex web applications that download and start in < 10s. It's nice when things are fast not slow.
kube-system 7y ago
In our environment, we update way more frequently for security updates of dependencies than we do for application updates. These are usually updates to base images. A couple of gigs is not bad one time. But it's multiplicative, even without updating our application at all, we're transferring: [avg size] * [# of images] * [# of security updates] * [# of nodes] Given that we are usually able to get small images just by using slim base images, it's a no-brainier for us. No, we don't pull our hair out trying to save a meg, but we're not inheriting Ubuntu as a base image for a 1MB microservice. By using slimmer images we have fewer security updates to apply as well. Since we usually give security updates some human attention, that's fewer man-hours we have to spend on this too, which is way more expensive than bandwidth.
- deleted 7y ago
  [deleted]
PopeDotNinja 7y ago
Having needed to build a 25GB container for production, I'm all for containers being as small as possible :) For the curious, we had a business case that required in cluster file storage with an AWS S3 compatible API which was better than the alternative options at the moment. I learned a lot about how to keep the container image to "only" 25 GB. I had to download files into the container, start Minio (the object store) at build time, upload the files to Minio to generate some needed metadata, and then delete the downloaded copy of the file. All of this on a build server that had 80gb of storage. I tell myself I have embedded storage at scale :)
- GordonS 7y ago
  I've never seen a container image anywhere near that big, so still curious :) What exactly did the 25GB include?
  - PopeDotNinja 7y ago
    It came preloaded with a bunch of different versions of software package for various operating systems and architectures. This was for an on premise deploy of Kubernetes where downloading files from outside the cluster was not an option. We were in a rush and this was the best idea anyone had at the time.
oppositelock 7y ago
Small images are useful in many scenarios. Developers iterating locally don't have to wait as long to produce them, when you deploy them to many machines, you don't have to worry about as much bandwidth due to the thundering herd problem, and you're paying less for storage for your docker registry, this all adds up. I've got thousands of docker images in my registry currently, which is SaaS Artifactory, and the transfer and storage costs are significant. A tiny image for running your app, such as an Alpine Linux base with a static executable in it, is also much more secure since your attack surface is significantly smaller.
kortex 7y ago
Clearly you've never deployed a 30 GB image, where 26 GB is the final, application building layer. And that's with staged building and working to minimize the size of overlay layers, redundancy, etc. Okay, so that might be a particularly pathological case, but still, resources are resources. I'm fine with a multi-gig machine learning image where the model and most of the heavy stuff is in a middle layer and the only redeploy is a few MB of python code. But that architecture isn't always easy, either.

itamarst 7y ago

Multi-GB images still seem... big. Though to be fair in many situations those 60 seconds are meaningless, so might not be worth the time to optimize. Often images that large are the result of including build toolchains, which you can omit with multi-stage builds (e.g. for Python https://pythonspeed.com/articles/multi-stage-docker-python/ https://pythonspeed.com/articles/multi-stage-docker-python/.

tybit 7y ago

I agree it’s oversold for most use cases, clouds do have pretty decent networking after all. Having said that as an Australian with sub par internet at home, I do appreciate not making images gratuitously large. Just because SDK images won’t ever be deployed to prod doesn’t mean you shouldn’t strip out unnecessary junk.

igor47 7y ago
Similarly, just yesterday I had to go home from the cafe I was working in because I had to push a docker image. The wifi just wasn't fast enough for that, though plenty fast for googling documentation

cosmotic 7y ago

Adding a minute to some deployment process can be fairly significant. The docker push and pull is probably 70% of my deployment process and I already have ~100MB or less images. I don't think the concern is 'how long does deployment take' but 'how fast can we iterate?'; Building and loading the images on a local dev machine to test a 2 second change would take much longer with larger images. Getting feedback of a PR merge from a CI build agent would take minutes longer. I don't think the importance has been oversold.

dcolkitt 7y ago
> I don't think the concern is 'how long does deployment take' but 'how fast can we iterate? And for iteration, the only thing that matters is the size of the top layers that are being iterated on. Not the overall image size itself. You can put `RUN apt-get [kitchen sink]` at the beginning of the Dockerfile and it pretty much won't matter. When you change anything in the project repository, that bigass giant base layer doesn't get re-pulled because it doesn't change. To validate that a layer, Docker daemon just compares the hashes. So, for unmodified layers, docker pull is constant time with image size. The Docker daemon only downloads from the registry starting with the bottom-most modified layer relative to its cache. If anything, throwing the kitchen sink in the base layer is better for fast iteration. When you're being parsimonious about third-party libraries and packages, you'll frequently have to rebuild the base image. If you `RUN apt-get [everything]`, then you'll hardly ever never need to rebuild/re-push/re-pull that layer, because you'll always have whatever you need already available.
- echevil 7y ago
  It depends on the size of project and what you want to optimize. The time spent on push and pull of docker images used to be a real deal breaker once upon a time when I tried to improve the deployment speed of a project by switching from heroku build to docker based solution
- cosmotic 7y ago
  In a fresh pod/vm it still needs to pull all the layers, even the big, old one.

paulddraper 7y ago

> startup even on a fresh node only takes 60 seconds or so There are very complex web applications that download and start in < 10s. It's nice when things are fast not slow.

kube-system 7y ago

In our environment, we update way more frequently for security updates of dependencies than we do for application updates. These are usually updates to base images. A couple of gigs is not bad one time. But it's multiplicative, even without updating our application at all, we're transferring: [avg size] * [# of images] * [# of security updates] * [# of nodes] Given that we are usually able to get small images just by using slim base images, it's a no-brainier for us. No, we don't pull our hair out trying to save a meg, but we're not inheriting Ubuntu as a base image for a 1MB microservice. By using slimmer images we have fewer security updates to apply as well. Since we usually give security updates some human attention, that's fewer man-hours we have to spend on this too, which is way more expensive than bandwidth.

deleted 7y ago
[deleted]

PopeDotNinja 7y ago

Having needed to build a 25GB container for production, I'm all for containers being as small as possible :) For the curious, we had a business case that required in cluster file storage with an AWS S3 compatible API which was better than the alternative options at the moment. I learned a lot about how to keep the container image to "only" 25 GB. I had to download files into the container, start Minio (the object store) at build time, upload the files to Minio to generate some needed metadata, and then delete the downloaded copy of the file. All of this on a build server that had 80gb of storage. I tell myself I have embedded storage at scale :)

GordonS 7y ago
I've never seen a container image anywhere near that big, so still curious :) What exactly did the 25GB include?
- PopeDotNinja 7y ago
  It came preloaded with a bunch of different versions of software package for various operating systems and architectures. This was for an on premise deploy of Kubernetes where downloading files from outside the cluster was not an option. We were in a rush and this was the best idea anyone had at the time.

oppositelock 7y ago

Small images are useful in many scenarios. Developers iterating locally don't have to wait as long to produce them, when you deploy them to many machines, you don't have to worry about as much bandwidth due to the thundering herd problem, and you're paying less for storage for your docker registry, this all adds up. I've got thousands of docker images in my registry currently, which is SaaS Artifactory, and the transfer and storage costs are significant. A tiny image for running your app, such as an Alpine Linux base with a static executable in it, is also much more secure since your attack surface is significantly smaller.

kortex 7y ago

Clearly you've never deployed a 30 GB image, where 26 GB is the final, application building layer. And that's with staged building and working to minimize the size of overlay layers, redundancy, etc. Okay, so that might be a particularly pathological case, but still, resources are resources. I'm fine with a multi-gig machine learning image where the model and most of the heavy stuff is in a middle layer and the only redeploy is a few MB of python code. But that architecture isn't always easy, either.