6 ms·

One thing I'm wondering -- why are people obsessed with container size? A lot of enterprise is Java shops which deal with huge dependencies, huge distributions,

One thing I'm wondering -- why are people obsessed with container size? A lot of enterprise is Java shops which deal with huge dependencies, huge distributions, and generally pretty heavy software -- with internet speeds what they are today, it's not hard to throw 100-500MB around internal networks (building the code closer to the deployment area is also a solution). Even gigabyte size containers are not impossible to deploy these days with the right caching (lord help you if you invalidate a bunch of layers though). In a world of xTB thumbdrives, why are we so worried about containers being <1MB? Security is one thing, minimalism is another, but I do sometimes wonder about this. Anyway, if you want to build minimal containers and want to do it with a bit more backing than this one (pretty well written) blog post, probably also check out distroless: https://github.com/GoogleContainerTools/distroless https://github.com/GoogleContainerTools/distroless I personally find most of the time alpine-based containers are more than small enough and good enough with nice ergonomics (sometimes you just want to shell in to the container), but maybe it makes sense to build containers with and without debugging/inspection tools (a `project:vx.x.x` and a `project:vx.x.x-debug`) and switch them out when problems arise.

thrwaeasddsaf 6y ago
I don't know about you but I work on a laptop, and I work on multiple projects, and each project might use dozens of containers for builds, testing, releases, etcetra. If every container were a gigabyte in size, majority of the laptop's internal storage would be wasted on them, nevermind the time wasted pushing and pulling updates.
- hardwaresofton 6y ago
  I do too (my laptop is pretty beefy though), and yes you're right this would be a problem, if containers weren't built to combat this with Copy on Write filesystems. That said, dependencies do take up a lot of space, but machines these days also come with a lot of space. People also run prune and develop caching strategies (ex. NPM) as a result. It's not like every container will be a GB in size, as it's not reasonable to have every container have that much content, but how many people are out there using 950MB dependency-filled base containers with 50MB of actual relevant application code? Most people learned the run-it-on-alpine lesson like 5+ years ago at this point.
  - lanstin 6y ago
    My new laptop has 3T of disk. I have the old feeling of what can I possibly do with it all. I have two copies of all my photos and text files etc. and dual boot Arch and Ubuntu but still it is so Empty. I don’t think even a very aggressive go get campaign can fill it.
    - thrwaeasddsaf 6y ago
      My work laptop has 0.5T SSD and it's 90% full. Turns out VMs, large repos, and OS projects are big. I know there are some large docker images too.
      - hardwaresofton 6y ago
        Do you run `docker system prune`/`docker image prune` and related commands from time to time? I assume you already know about it but just in case. Just recently I found that I had like 100s of GBs used up on my hard drive because I'd installed and used podman for a while and it's image store was separate and just... sitting there, long after I'd moved back to docker for reasons.
emptyparadise 6y ago
Why create bloat when it can be avoided? The 6kB server might be overkill but the other options in that article seem pretty good.
- tluyben2 6y ago
  A lot of companies that sell products (non open source ones anyway) actually try to bloat the binaries and deployments to make them look more significant. Many gigabyte installers are not that weird for some products, where I found (for fun as corp wise we cannot really care) the relevant software we are paying for to be a few mb and it doesn't need or even touch the rest the rest in the package for 99% of cases. But it seems/feels like you are paying those $100000s for some serious big software. When software was still delivered in physical boxes, I noticed the same thing and I even found products in the past where the .exe had megabytes of random data attached to it to make it look bigger. With Docker images it's all a lot easier to bloat things up and 'hide things'; I used to pack up our main product in a chroot tgz (this is early 2000s) with an installer, so I did not have to actually think about different linux distro's or installers; I know for a fact that some people bought our product over competitors because the installer was so large in size so it looks like better value. People told me when I asked them afterwards why they did not go for the competition. And this was not intentional; it just had a full Debian install in a chroot.
  - daniellarusso 6y ago
    This was something I never considered, but reminds me of the Larry Ellison tactic of never naming anything v1, or the concept that because something costs the most, it therefore must be the best.
    - tluyben2 6y ago
      Well, Oracle was (not sure if they still are) one of these companies. Some of their products we used in the past where artificially bloated and packed over multi CDs etc to make it all look bigger.
El_RIDO 6y ago
I'm working at a shop where we have to use extremely large images (10 - 40 GiB) on a platform of 60+ medium sized blades (4 - 12 cores, 32 - 256 GiB RAM, 200 - 1000 GiB SSDs). Startup time for running such huge images often is 5 - 10 minutes. They contain prepackaged test data so that they can be run in parallel against newly built versions of our software - there are about 30ish of these images. You not only store these images once in the registry, but on all container hosts that they might run on. And you need to transmit them, if they aren't cached there, yet. And you may want to redeploy new images several times a day, as they get updated, new ones added, etc. To be fair, the above is not the use case containers were originally intended for. And if your 10 GiB container image only runs on 3 or 5 nodes and only gets updated once a week, you obviously don't have to worry about the overhead too much. But at scale (beyond 10 nodes) and/or on very fast development cycles (=more then one deployment per day), size IMHO starts to matter.
- hardwaresofton 6y ago
  I can totally see this starting to matter, but I do want to point out that it hasn't ground your business to a halt just yet and is very very unlikely to. From what I've seen most ops pain comes from a lack of automation when deploying -- as in if the process is 5-10 or even 15 minutes, no one really cares (outside of an emergency) if it's completely automated. Do you find that is true with your work as well? I can certainly see that it can be a painful problem (10-40GiB is huge), but I expect most smaller shops that aren't running on their own servers/colo as such these image sizes were never even an option (I can't imagine trying to upload 40GB across the public internet to launch one instance!).
  - ethbr0 6y ago
    I've worked in the general automation space for awhile, and agree that time-to-result is rarely the limiting factor. It's also not usually total-person-clock-time. It's number-of-distinct-manual-steps. (5 minutes manual) + (15 minutes automation) + (5 minutes manual) + (30 minutes automation) = everyone hates doing it (10 minutes manual) + (45 minutes automation) = everyone's happy and can use their time productively
ldng 6y ago
Wasted energy and bandwith do matter. Now, there is a gap between a few kB and multiple GB.
harpratap 6y ago
At scale every tiny bit helps. Especially now that Edge computing is becoming a reality, it's much better to pull and run a 1MB container image than a 100MB one on limited hardware and bandwidth
- hardwaresofton 6y ago
  Agreed -- edge computing is a different world, and the difference between 100kb-1mb-10mb-100mb is huge. What I was discussing is definitely not a fit at that scale/usecase -- I'd argue that most infrastructure using containers (and mos t people who see this blog post) are not built for this scale/use-case, and I'm not sure containers even make sense in the edge computing paradigm just yet.
geek_at 6y ago
oh god I fell in love with everthing related to Alpine Linux. Found out about it when I started playing around with docker but then ported all my home servers and Raspberry Pis to Alpine. It's so damn small and has little overhead. Need docker? `apk add docker`, done! I've written guides on how to boot Alpine (even with GUI) over PXE [1] and how to set up a perfect file server that runs from a RAMdisk USB Thumb drive with full disk encryption [2] [1] https://blog.haschek.at/2019/build-your-own-datacenter-with-pxe-and-alpine.html https://blog.haschek.at/2019/build-your-own-datacenter-with-... [2] https://blog.haschek.at/2020/the-perfect-file-server.html https://blog.haschek.at/2020/the-perfect-file-server.html
- Annatar 6y ago
  https://rom-o-matic.eu/ https://rom-o-matic.eu/ is a broken link. https://blog.haschek.at/2020/the-perfect-file-server.html https://blog.haschek.at/2020/the-perfect-file-server.html is made obsolete by SmartOS, which does the exact same thing, in addition to offering Triton (enterprise web GUI for virtual systems), OpenZFS (end-to-end data protection, ease of administration), zones (full-blown, running-at-the-speed-of-metal virtual UNIX servers), DTrace (for production-safe, real-time, deep machine state instrumentation and inspection), Bardiche (for building firewalls), FireEngine / Crossbow (high-performance TCP/IP stack for building virtual switches and routers) and imgadm(1M) / vmadm(1M) (for virtual server provisioning and software management), and last but not least, a reference implementation of the NFS V4 protocol.
  - geek_at 6y ago
    > is made obsolete by SmartOS or by proxmox or by any other OS. That's not what it's about. If it comes with a web gui, it's already bloated for me. This project was about the minimal perfect setup for my needs and I only need SSH access
    - Annatar 6y ago
      SmartOS does not come with a web GUI; Triton is completely optional and isn't required at all, but if one is running 100,000 servers or more, it's there as a gratis option if one needs one. As far as I am aware, and please correct me if I am wrong, but "proxmox" does not have nearly the same list of capabilities as SmartOS, all the while running from a ~650 MB, read-only RAMDisk.
blowski 6y ago
I wonder the same thing. I can see the value in shrinking a 1GB container by 90%, but I suspect there are diminishing returns by the time you're shrinking a 5MB container below 1MB. In most organisations, there are probably bigger opportunities for improving security, performance, etc.
- viraptor 6y ago
  It's worth remembering the MBs add up in multiple places. Sure, it may not be worth for 5MB. But once you're in tens/hundreds, they add up in storage (how many old versions do you keep), transfer times (how quickly can a fresh instance pull all tasks and how quickly can a developer (at home rather than in us-east-1) start with a project), transfer costs (how many times does a CI node need to download/upload the last layer). Saving 10% in size may actually translate to a non-trivial cost saving - both obvious and cost-as-in-paid-waiting-time.
  - lanstin 6y ago
    Also all those dependencies are security risks. A lot quicker and easier to scan a container with just one go executable than even the most benign distribution of core utils and so on. And if your containers are just one binary and you can read the logs then you don’t need to go into the container any more than you need to go into your binary. Then your k8s node or other container runner host is just like your old host running a bunch of processes but with much better process isolation and file system isolation.
    - blowski 6y ago
      There are some teams where this will be the best use of their time, and I'd be well out of my depth suggesting to them otherwise. But the vast majority of teams in my experience have spent ages farting around with these kind of micro-optmisations, whilst there were far better opportunities to improve the infrastructure. And in the process, they actually made it more fragile and harder to debug. Simply put, this is a niche skill with niche value. Most people trying to do this probably can't and shouldn't do it.
nfrankel 6y ago
This. I found that most people focus on size and forget about reuse. Sure, each layer adds size to your final image, but you can reuse them, so that you don't need to transfer the complete image but just the layers on top of the one that changed. Flatten everything and you need to ship the whole stuff.
paxys 6y ago
Plus, figuring out optimal sizing for Docker images is in general complicated and often counterintuitive. A company I worked with mandated that every image must be written from scratch to reduce size on disk and save on transfer bandwidth/time. The end result though was that there were no sharable layers, and so every server had to redownload every image wherever there was the tiniest update. Moving to a set of common 1GB+ base images with all dependencies included made the entire system smaller and faster.
zokier 6y ago
Container size generally impacts startup/deploy times. 500MB container still takes almost a second to copy over gigabit connection, and typically you need to copy it at least twice: from build server to repository and from repository to whereever its going to run.
- AdrianB1 6y ago
  A Gigabit Ethernet connection means 1Gbps minus some overhead, effective rate is around 117-120 MB/sec, so the 500 MB container takes at least 4 seconds to copy. 10 Gbps gets you to less than a second, but that works usually between servers, not from the regular laptops.
- chrisandchris 6y ago
  Which also means that _if_ you pull at full speed, this node has no bandwith at all for serving users (more or less).
goodpoint 6y ago
> why are people obsessed with container size Good question! Security should be a priority instead. A lot of containers on docker hub have serious vulnerabilities, as proven by various research papers. Removing bloat is good for security. However, using a well-tested distro and receiving security updates from its security team is more important than having a tiny footprint. This is a good reason to use Debian stable in containers and avoid alpine.
bob1029 6y ago
> building the code closer to the deployment area is also a solution This is exactly what we do. Once the first build of our application is installed on our customer's servers, it rebuilds itself from source each time. This is also really good at firewalling CI/CD compromises away from your customers, as long as you don't merge cryptolockers/miners into your private github repos.
AdrianB1 6y ago
I am working in Manufacturing Execution Systems area and the internal sites are spread around the globe on all continents; speed does matter, there are dozens of places hundreds of kilometers away from a place with Gigabit Internet (for WAN). Taking backups off site is faster by physically shipping hard drives than copying over the WAN, I don't like any large software. In one place in Africa driving to another site 3 hours away gets network speed 10x better, so we do this from time to time.

thrwaeasddsaf 6y ago

I don't know about you but I work on a laptop, and I work on multiple projects, and each project might use dozens of containers for builds, testing, releases, etcetra. If every container were a gigabyte in size, majority of the laptop's internal storage would be wasted on them, nevermind the time wasted pushing and pulling updates.

hardwaresofton 6y ago
I do too (my laptop is pretty beefy though), and yes you're right this would be a problem, if containers weren't built to combat this with Copy on Write filesystems. That said, dependencies do take up a lot of space, but machines these days also come with a lot of space. People also run prune and develop caching strategies (ex. NPM) as a result. It's not like every container will be a GB in size, as it's not reasonable to have every container have that much content, but how many people are out there using 950MB dependency-filled base containers with 50MB of actual relevant application code? Most people learned the run-it-on-alpine lesson like 5+ years ago at this point.
- lanstin 6y ago
  My new laptop has 3T of disk. I have the old feeling of what can I possibly do with it all. I have two copies of all my photos and text files etc. and dual boot Arch and Ubuntu but still it is so Empty. I don’t think even a very aggressive go get campaign can fill it.
  - thrwaeasddsaf 6y ago
    My work laptop has 0.5T SSD and it's 90% full. Turns out VMs, large repos, and OS projects are big. I know there are some large docker images too.
    - hardwaresofton 6y ago
      Do you run `docker system prune`/`docker image prune` and related commands from time to time? I assume you already know about it but just in case. Just recently I found that I had like 100s of GBs used up on my hard drive because I'd installed and used podman for a while and it's image store was separate and just... sitting there, long after I'd moved back to docker for reasons.

emptyparadise 6y ago

Why create bloat when it can be avoided? The 6kB server might be overkill but the other options in that article seem pretty good.

tluyben2 6y ago
A lot of companies that sell products (non open source ones anyway) actually try to bloat the binaries and deployments to make them look more significant. Many gigabyte installers are not that weird for some products, where I found (for fun as corp wise we cannot really care) the relevant software we are paying for to be a few mb and it doesn't need or even touch the rest the rest in the package for 99% of cases. But it seems/feels like you are paying those $100000s for some serious big software. When software was still delivered in physical boxes, I noticed the same thing and I even found products in the past where the .exe had megabytes of random data attached to it to make it look bigger. With Docker images it's all a lot easier to bloat things up and 'hide things'; I used to pack up our main product in a chroot tgz (this is early 2000s) with an installer, so I did not have to actually think about different linux distro's or installers; I know for a fact that some people bought our product over competitors because the installer was so large in size so it looks like better value. People told me when I asked them afterwards why they did not go for the competition. And this was not intentional; it just had a full Debian install in a chroot.
- daniellarusso 6y ago
  This was something I never considered, but reminds me of the Larry Ellison tactic of never naming anything v1, or the concept that because something costs the most, it therefore must be the best.
  - tluyben2 6y ago
    Well, Oracle was (not sure if they still are) one of these companies. Some of their products we used in the past where artificially bloated and packed over multi CDs etc to make it all look bigger.

El_RIDO 6y ago

I'm working at a shop where we have to use extremely large images (10 - 40 GiB) on a platform of 60+ medium sized blades (4 - 12 cores, 32 - 256 GiB RAM, 200 - 1000 GiB SSDs). Startup time for running such huge images often is 5 - 10 minutes. They contain prepackaged test data so that they can be run in parallel against newly built versions of our software - there are about 30ish of these images. You not only store these images once in the registry, but on all container hosts that they might run on. And you need to transmit them, if they aren't cached there, yet. And you may want to redeploy new images several times a day, as they get updated, new ones added, etc. To be fair, the above is not the use case containers were originally intended for. And if your 10 GiB container image only runs on 3 or 5 nodes and only gets updated once a week, you obviously don't have to worry about the overhead too much. But at scale (beyond 10 nodes) and/or on very fast development cycles (=more then one deployment per day), size IMHO starts to matter.

hardwaresofton 6y ago
I can totally see this starting to matter, but I do want to point out that it hasn't ground your business to a halt just yet and is very very unlikely to. From what I've seen most ops pain comes from a lack of automation when deploying -- as in if the process is 5-10 or even 15 minutes, no one really cares (outside of an emergency) if it's completely automated. Do you find that is true with your work as well? I can certainly see that it can be a painful problem (10-40GiB is huge), but I expect most smaller shops that aren't running on their own servers/colo as such these image sizes were never even an option (I can't imagine trying to upload 40GB across the public internet to launch one instance!).
- ethbr0 6y ago
  I've worked in the general automation space for awhile, and agree that time-to-result is rarely the limiting factor. It's also not usually total-person-clock-time. It's number-of-distinct-manual-steps. (5 minutes manual) + (15 minutes automation) + (5 minutes manual) + (30 minutes automation) = everyone hates doing it (10 minutes manual) + (45 minutes automation) = everyone's happy and can use their time productively

ldng 6y ago

Wasted energy and bandwith do matter. Now, there is a gap between a few kB and multiple GB.

harpratap 6y ago

At scale every tiny bit helps. Especially now that Edge computing is becoming a reality, it's much better to pull and run a 1MB container image than a 100MB one on limited hardware and bandwidth

hardwaresofton 6y ago
Agreed -- edge computing is a different world, and the difference between 100kb-1mb-10mb-100mb is huge. What I was discussing is definitely not a fit at that scale/usecase -- I'd argue that most infrastructure using containers (and mos t people who see this blog post) are not built for this scale/use-case, and I'm not sure containers even make sense in the edge computing paradigm just yet.

geek_at 6y ago

oh god I fell in love with everthing related to Alpine Linux. Found out about it when I started playing around with docker but then ported all my home servers and Raspberry Pis to Alpine. It's so damn small and has little overhead. Need docker? `apk add docker`, done! I've written guides on how to boot Alpine (even with GUI) over PXE [1] and how to set up a perfect file server that runs from a RAMdisk USB Thumb drive with full disk encryption [2] [1] https://blog.haschek.at/2019/build-your-own-datacenter-with-pxe-and-alpine.html https://blog.haschek.at/2019/build-your-own-datacenter-with-... [2] https://blog.haschek.at/2020/the-perfect-file-server.html https://blog.haschek.at/2020/the-perfect-file-server.html

Annatar 6y ago
https://rom-o-matic.eu/ https://rom-o-matic.eu/ is a broken link. https://blog.haschek.at/2020/the-perfect-file-server.html https://blog.haschek.at/2020/the-perfect-file-server.html is made obsolete by SmartOS, which does the exact same thing, in addition to offering Triton (enterprise web GUI for virtual systems), OpenZFS (end-to-end data protection, ease of administration), zones (full-blown, running-at-the-speed-of-metal virtual UNIX servers), DTrace (for production-safe, real-time, deep machine state instrumentation and inspection), Bardiche (for building firewalls), FireEngine / Crossbow (high-performance TCP/IP stack for building virtual switches and routers) and imgadm(1M) / vmadm(1M) (for virtual server provisioning and software management), and last but not least, a reference implementation of the NFS V4 protocol.
- geek_at 6y ago
  > is made obsolete by SmartOS or by proxmox or by any other OS. That's not what it's about. If it comes with a web gui, it's already bloated for me. This project was about the minimal perfect setup for my needs and I only need SSH access
  - Annatar 6y ago
    SmartOS does not come with a web GUI; Triton is completely optional and isn't required at all, but if one is running 100,000 servers or more, it's there as a gratis option if one needs one. As far as I am aware, and please correct me if I am wrong, but "proxmox" does not have nearly the same list of capabilities as SmartOS, all the while running from a ~650 MB, read-only RAMDisk.

blowski 6y ago

I wonder the same thing. I can see the value in shrinking a 1GB container by 90%, but I suspect there are diminishing returns by the time you're shrinking a 5MB container below 1MB. In most organisations, there are probably bigger opportunities for improving security, performance, etc.

viraptor 6y ago
It's worth remembering the MBs add up in multiple places. Sure, it may not be worth for 5MB. But once you're in tens/hundreds, they add up in storage (how many old versions do you keep), transfer times (how quickly can a fresh instance pull all tasks and how quickly can a developer (at home rather than in us-east-1) start with a project), transfer costs (how many times does a CI node need to download/upload the last layer). Saving 10% in size may actually translate to a non-trivial cost saving - both obvious and cost-as-in-paid-waiting-time.
- lanstin 6y ago
  Also all those dependencies are security risks. A lot quicker and easier to scan a container with just one go executable than even the most benign distribution of core utils and so on. And if your containers are just one binary and you can read the logs then you don’t need to go into the container any more than you need to go into your binary. Then your k8s node or other container runner host is just like your old host running a bunch of processes but with much better process isolation and file system isolation.
  - blowski 6y ago
    There are some teams where this will be the best use of their time, and I'd be well out of my depth suggesting to them otherwise. But the vast majority of teams in my experience have spent ages farting around with these kind of micro-optmisations, whilst there were far better opportunities to improve the infrastructure. And in the process, they actually made it more fragile and harder to debug. Simply put, this is a niche skill with niche value. Most people trying to do this probably can't and shouldn't do it.

nfrankel 6y ago

This. I found that most people focus on size and forget about reuse. Sure, each layer adds size to your final image, but you can reuse them, so that you don't need to transfer the complete image but just the layers on top of the one that changed. Flatten everything and you need to ship the whole stuff.

paxys 6y ago

Plus, figuring out optimal sizing for Docker images is in general complicated and often counterintuitive. A company I worked with mandated that every image must be written from scratch to reduce size on disk and save on transfer bandwidth/time. The end result though was that there were no sharable layers, and so every server had to redownload every image wherever there was the tiniest update. Moving to a set of common 1GB+ base images with all dependencies included made the entire system smaller and faster.

zokier 6y ago

Container size generally impacts startup/deploy times. 500MB container still takes almost a second to copy over gigabit connection, and typically you need to copy it at least twice: from build server to repository and from repository to whereever its going to run.

AdrianB1 6y ago
A Gigabit Ethernet connection means 1Gbps minus some overhead, effective rate is around 117-120 MB/sec, so the 500 MB container takes at least 4 seconds to copy. 10 Gbps gets you to less than a second, but that works usually between servers, not from the regular laptops.
chrisandchris 6y ago
Which also means that _if_ you pull at full speed, this node has no bandwith at all for serving users (more or less).

goodpoint 6y ago

> why are people obsessed with container size Good question! Security should be a priority instead. A lot of containers on docker hub have serious vulnerabilities, as proven by various research papers. Removing bloat is good for security. However, using a well-tested distro and receiving security updates from its security team is more important than having a tiny footprint. This is a good reason to use Debian stable in containers and avoid alpine.

bob1029 6y ago

> building the code closer to the deployment area is also a solution This is exactly what we do. Once the first build of our application is installed on our customer's servers, it rebuilds itself from source each time. This is also really good at firewalling CI/CD compromises away from your customers, as long as you don't merge cryptolockers/miners into your private github repos.

AdrianB1 6y ago

I am working in Manufacturing Execution Systems area and the internal sites are spread around the globe on all continents; speed does matter, there are dozens of places hundreds of kilometers away from a place with Gigabit Internet (for WAN). Taking backups off site is faster by physically shipping hard drives than copying over the WAN, I don't like any large software. In one place in Africa driving to another site 3 hours away gets network speed 10x better, so we do this from time to time.