13 ms·

I have always been a bit surprised at the popularity of Alpine Linux for docker images. It’s awesome that the images are pretty small, but a wide variety of sof

I have always been a bit surprised at the popularity of Alpine Linux for docker images. It’s awesome that the images are pretty small, but a wide variety of software has been shown to run noticeably slower on Alpine compared to other distributions, in part due to its usage of musl instead of glibc. I’d think that a few megabytes of disk isn’t as valuable as the extra cpu cycles.

sosodev 7y ago
I guess the question is how much slower?
- Jonanin 7y ago
  Benchmarks here: https://www.phoronix.com/scan.php?page=article&item=docker-summer-2018&num=4 https://www.phoronix.com/scan.php?page=article&item=docker-s... For Python in particular, significantly slower.
  - nine_k 7y ago
    The Python test is indeed a large outlier! I wonder how reproducible it is, and what the reason might be.
    - aw3c2 7y ago
      That is very likely due to processor/architecture optimized precompiled Python. See e.g. https://clearlinux.org/blogs/transparent-use-library-packages-optimized-intel-architecture https://clearlinux.org/blogs/transparent-use-library-package... and https://clearlinux.org/blogs/boosting-python-profile-guided-platform-specific-optimizations https://clearlinux.org/blogs/boosting-python-profile-guided-...
- blopker 7y ago
  I did an analysis with the official Docker Python image [0] with the --enable-optimizations flag. Although not the point of the results [1], they show a decent speed up when using Debian vs Alpine as well. [0] https://github.com/docker-library/python/issues/160 https://github.com/docker-library/python/issues/160 [1] https://gist.github.com/blopker/9fff37e67f14143d2757f9f2172c8ea0 https://gist.github.com/blopker/9fff37e67f14143d2757f9f2172c...
chrisseaton 7y ago
> I’d think that a few megabytes of disk isn’t as valuable as the extra cpu cycles. Depends on your workload, of course. Some people want to run a huge number of containers and each isn’t compute intensive. Or maybe you don’t use libc at all in your fast path? Lots of cases it makes sense.
- icebraining 7y ago
  If you reuse the base image, the libc files will be shared anyway.
  - jacques_chester 7y ago
    This is the part most people don't realise. They see Alpine at 5Mb and Ubuntu at 80Mb. They mentally multiply, without realising that each of these will be pulled once for each image built on top of them. For a large cluster it's a wash. You might as well use Ubuntu, Centos -- anything where there are people working fulltime to fix CVEs quickly.
    - deleted 7y ago
      [deleted]
    - ShinTakuya 7y ago
      As far as I'm aware you'd have to load that 80mb into memory for each docker container you run so that can add up if you want to run a bunch of containers on a cheap host with 1GB of RAM. I do agree that people prematurely optimise and mainly incorrectly consider disk space but I think there's a decent use case for tiny images.
      - jacques_chester 7y ago
        Not quite. The 80Mb represents the uncompressed on-disk size. Those bits can appear in memory in two main ways. It can be executable, in which case large parts will be shared (remember, containers are not like VMs). Or it can in the FS cache, in which case bits will be evicted as necessary to make room for executables. There's a case for tiny images, but it's in severely-constrained environments. Otherwise folks are fetishising the wrong thing based on a misunderstanding of how container images and runtimes work.
nine_k 7y ago
If your software is heavily CPU-bound, and the difference matters for you, you can likely find or build a highly optimized image for your particular number-crunching task. If your software is I/O-bound, and is written in a language like Python or Ruby, and sits idle waiting for requests for significant time anyway, CPU performance is likely not key for you. This also represents the majority case, AFAICT.
yegle 7y ago
For anyone who want a small image but with glibc, https://github.com/GoogleContainerTools/distroless https://github.com/GoogleContainerTools/distroless is a good choice, especially if you are writing in static linked language e.g. Go and Rust.
- Jonanin 7y ago
  For Python I'd also highly recommend Clear Linux for raw performance. It's not quite as easy to get started as with something like Ubuntu though.
  - mixmastamyk 7y ago
    Checked it out, is Intel supported with compiler tweaks to get the most out of their CPUs, for libs like math, pandas, etc.
  - pella 7y ago
    docker pull clearlinux/python https://github.com/clearlinux/dockerfiles/tree/master/python https://github.com/clearlinux/dockerfiles/tree/master/python
  - svacko 7y ago
    Clear Linux is definitely an interesting project, however, using base image with 'latest' tag (only tag existing for https://hub.docker.com/r/clearlinux/python/tags https://hub.docker.com/r/clearlinux/python/tags) in the production is not a best strategy as the breaking changes can arrive anytime
- snarfy 7y ago
  For statically linked binaries, why wouldn't you use the SCRATCH (0 kb) 'image'?
  - hssys 7y ago
    You might want at least a shell in the container for debugging?
polskibus 7y ago
What is the better and safer alternative for cases where some extra megabytes are ok?
sofaofthedamned 7y ago
I've no idea why you wouldn't use Ubuntu which is only around 40mb, has a sane package manager and a standard glibc.
- peterwwillis 7y ago
  The base image may be small, but all the packages and metadata are large, and the dependencies are many. Alpine always leans to conservative options and intentionally removes mostly-unnecessary things. So average image sizes are higher with an Ubuntu base compared to Alpine.
- nine_k 7y ago
  40mb vs 5mb is like 5x difference. There are also slimmed down images based on Debian or Ubuntu. A number of packages is a bit older versions, though.
  - koolba 7y ago
    You're only playing that 40mb once though. Multiple containers sharing the same parent layers will not require additional storage for the core OS layer.
    - sofaofthedamned 7y ago
      Exactly. With copy on write and samepage merging it's more important to use the same base image for all your deployments.
    - outworlder 7y ago
      Are you guys running everything on a single box? Do you get all developers to agree on which base image to build all their services from? I heard about this "oh, it's shared, don't worry" thing before. It started with 40MB. Now that supposedly shared image is half a gig. "Don't worry, it's shared anyway". Expect when it isn't. And when it is, it still slow us down in bringing up new nodes. And guess what, turns out that not everyone is starting from the same point, so there is a multitude of 'shared' images now. Storage is cheap, but bandwidth may not be. And it still takes time to download. Try to keep your containers as small as possible for as long as possible. Your tech debt may grow slower that way.
peterwwillis 7y ago
It's not a few megabytes, it's a few hundred to a thousand megabytes saved, on average. Multiplied by a thousand containers, and much larger layers on build servers, plus bandwidth, it makes a difference. Worst case for slower processes, things take longer. Worst case for more disk use, things start crashing. For general cases, the former is preferable.
- sofaofthedamned 7y ago
  No, it's not. With samepage merging it's nothing, let alone docker only loading the image once.
  - atombender 7y ago
    This assumes everyone is on exactly the same version. Larger organizations can afford to appoint SREs who can institute a broad range of security and optimization policies (including "all apps should use the same Alpine base image version") and enforce them programmatically as well as ensure that apps are continually updated to match them. That kind of thing is expensive, resource-wise, for smaller organizations.
  - peterwwillis 7y ago
    I'm not talking about image waste, I'm talking a single box with a half dozen different base image versions and a slew of extra packages thrown in. Every app built against glibc is going to get bigger, and not all Ubuntu packages are built with small size in mind. Compare these systems with Ubuntu vs Alpine base images and the average size for an Ubuntu ecosystem is substantially larger.
whatshisface 7y ago
Why is musl slower?
- saagarjha 7y ago
  It's much simpler and more oriented towards POSIX compatibility than performance.
- Nelson69 7y ago
  I don't want to disparage any project or guess the motivation but there have been some undercurrents of anti-GPL sentiment at times and anti-complexity. Folks have sort of backed it up by posting some links to things that aren't as you might think. GLIBC in particular looks nothing like how you might imagine it. You can look at strlen in the K&R book and it's beautiful, like a textbook: int strlen(char s[]) { int i; i = 0; while( s[i] != '\0') ++i return i; } then you look at: https://sourceware.org/git/?p=glibc.git;a=blob;f=string/strlen.c;h=279749659f2be51a1f643a9aafdc4e1d3320136d;hb=HEAD https://sourceware.org/git/?p=glibc.git;a=blob;f=string/strl... and your mind will sort of explode for a bit. The difference is the GLIBC version is dramatically faster; take the comments away and most of us wouldn't even know that's strlen. It's more complex, no question, but it's much faster. GLIBC is full of stuff like that. qsort and memcpy are non-obvious to many folks. It's not complexity for no reason, you'd be challenged to build a better qsort than the one in glibc, it's not easy.
  - username223 7y ago
    This reminds me of a classic blog post: http://ridiculousfish.com/blog/posts/old-age-and-treachery.html All of those old Unix programs aren't fast by being simple and clean on the inside. Old age and treachery...
    - globuous 7y ago
      And this guy also https://news.ycombinator.com/item?id=14543536 https://news.ycombinator.com/item?id=14543536 (which comes with this: https://www.gnu.org/prep/standards/standards.html#Reading-Non_002dFree-Code https://www.gnu.org/prep/standards/standards.html#Reading-No...)
bogomipz 7y ago
>"It’s awesome that the images are pretty small, but a wide variety of software has been shown to run noticeably slower on Alpine compared to other distributions, in part due to its usage of musl instead of glibc" Might you have any citation(s) for this? I have not heard this before. Could you elaborate on what some the "wide variety of software" is? Thanks.
- creatornator 7y ago
  I've heard that having shared libraries like glibc allows for commonly shared hotpaths to remain in the CPU cache, making it faster. Whereas in musl, since the binaries all have their own copies of procedures, they are more often kicked out of cache. I don't imagine it has a big impact on a container where you are only running one or two binaries. Perhaps in a desktop environment with hundreds or thousands of different programs running, it might make a difference.
cs02rm0 7y ago
Because it's tiny, I tend to default to Alpine and then move away from it where necessary. Rather than worrying about potential CPU performance requirements upfront - premature optimisation and all that.
- mikepurvis 7y ago
  Isn't there an argument that using Alpine instead of something like Ubuntu is premature optimization for space?
  - akvadrako 7y ago
    It's faster in terms of development time. Faster to install packages and to push to docker repos.
    - icebraining 7y ago
      Why? You should only need to push the base image once. Then only upper layers will have to be pushed.
  - draw_down 7y ago
    Optimization is only premature when other people do it. :)
- emilfihlman 7y ago
  Making it smaller than necessary is a bigger premature optimisation, though. Rather than worrying about potential disk space issues upfront just use glibc and everything works fast and fine.
bayesian_horse 7y ago
The small size of an image can be an actual issue. Certainly for embedded devices, probably also for clusters. For me it's mostly a non-issue. But things like redis or nginx work nicely with alpine as a base. And more than likely, whatever I want to containerize has already been containerized for Alpine. If not, getting something to work on Alpine may just not be worth it...
dnautics 7y ago
It's not just a few megabytes, it's more like hundreds. If you're moving an image across a bad network (like the internet) that could be hundreds of milliseconds.
- geezerjay 7y ago
  I would also add that data transfers cost money, and having to transfer a few hundred MBs each time a container image is passed around can reflect in the expenses.

sosodev 7y ago

I guess the question is how much slower?

Jonanin 7y ago
Benchmarks here: https://www.phoronix.com/scan.php?page=article&item=docker-summer-2018&num=4 https://www.phoronix.com/scan.php?page=article&item=docker-s... For Python in particular, significantly slower.
- nine_k 7y ago
  The Python test is indeed a large outlier! I wonder how reproducible it is, and what the reason might be.
  - aw3c2 7y ago
    That is very likely due to processor/architecture optimized precompiled Python. See e.g. https://clearlinux.org/blogs/transparent-use-library-packages-optimized-intel-architecture https://clearlinux.org/blogs/transparent-use-library-package... and https://clearlinux.org/blogs/boosting-python-profile-guided-platform-specific-optimizations https://clearlinux.org/blogs/boosting-python-profile-guided-...
blopker 7y ago
I did an analysis with the official Docker Python image [0] with the --enable-optimizations flag. Although not the point of the results [1], they show a decent speed up when using Debian vs Alpine as well. [0] https://github.com/docker-library/python/issues/160 https://github.com/docker-library/python/issues/160 [1] https://gist.github.com/blopker/9fff37e67f14143d2757f9f2172c8ea0 https://gist.github.com/blopker/9fff37e67f14143d2757f9f2172c...

chrisseaton 7y ago

> I’d think that a few megabytes of disk isn’t as valuable as the extra cpu cycles. Depends on your workload, of course. Some people want to run a huge number of containers and each isn’t compute intensive. Or maybe you don’t use libc at all in your fast path? Lots of cases it makes sense.

icebraining 7y ago
If you reuse the base image, the libc files will be shared anyway.
- jacques_chester 7y ago
  This is the part most people don't realise. They see Alpine at 5Mb and Ubuntu at 80Mb. They mentally multiply, without realising that each of these will be pulled once for each image built on top of them. For a large cluster it's a wash. You might as well use Ubuntu, Centos -- anything where there are people working fulltime to fix CVEs quickly.
  - deleted 7y ago
    [deleted]
  - ShinTakuya 7y ago
    As far as I'm aware you'd have to load that 80mb into memory for each docker container you run so that can add up if you want to run a bunch of containers on a cheap host with 1GB of RAM. I do agree that people prematurely optimise and mainly incorrectly consider disk space but I think there's a decent use case for tiny images.
    - jacques_chester 7y ago
      Not quite. The 80Mb represents the uncompressed on-disk size. Those bits can appear in memory in two main ways. It can be executable, in which case large parts will be shared (remember, containers are not like VMs). Or it can in the FS cache, in which case bits will be evicted as necessary to make room for executables. There's a case for tiny images, but it's in severely-constrained environments. Otherwise folks are fetishising the wrong thing based on a misunderstanding of how container images and runtimes work.

nine_k 7y ago

If your software is heavily CPU-bound, and the difference matters for you, you can likely find or build a highly optimized image for your particular number-crunching task. If your software is I/O-bound, and is written in a language like Python or Ruby, and sits idle waiting for requests for significant time anyway, CPU performance is likely not key for you. This also represents the majority case, AFAICT.

yegle 7y ago

For anyone who want a small image but with glibc, https://github.com/GoogleContainerTools/distroless https://github.com/GoogleContainerTools/distroless is a good choice, especially if you are writing in static linked language e.g. Go and Rust.

Jonanin 7y ago
For Python I'd also highly recommend Clear Linux for raw performance. It's not quite as easy to get started as with something like Ubuntu though.
- mixmastamyk 7y ago
  Checked it out, is Intel supported with compiler tweaks to get the most out of their CPUs, for libs like math, pandas, etc.
- pella 7y ago
  docker pull clearlinux/python https://github.com/clearlinux/dockerfiles/tree/master/python https://github.com/clearlinux/dockerfiles/tree/master/python
- svacko 7y ago
  Clear Linux is definitely an interesting project, however, using base image with 'latest' tag (only tag existing for https://hub.docker.com/r/clearlinux/python/tags https://hub.docker.com/r/clearlinux/python/tags) in the production is not a best strategy as the breaking changes can arrive anytime
snarfy 7y ago
For statically linked binaries, why wouldn't you use the SCRATCH (0 kb) 'image'?
- hssys 7y ago
  You might want at least a shell in the container for debugging?

polskibus 7y ago

What is the better and safer alternative for cases where some extra megabytes are ok?

sofaofthedamned 7y ago

I've no idea why you wouldn't use Ubuntu which is only around 40mb, has a sane package manager and a standard glibc.

peterwwillis 7y ago
The base image may be small, but all the packages and metadata are large, and the dependencies are many. Alpine always leans to conservative options and intentionally removes mostly-unnecessary things. So average image sizes are higher with an Ubuntu base compared to Alpine.
nine_k 7y ago
40mb vs 5mb is like 5x difference. There are also slimmed down images based on Debian or Ubuntu. A number of packages is a bit older versions, though.
- koolba 7y ago
  You're only playing that 40mb once though. Multiple containers sharing the same parent layers will not require additional storage for the core OS layer.
  - sofaofthedamned 7y ago
    Exactly. With copy on write and samepage merging it's more important to use the same base image for all your deployments.
  - outworlder 7y ago
    Are you guys running everything on a single box? Do you get all developers to agree on which base image to build all their services from? I heard about this "oh, it's shared, don't worry" thing before. It started with 40MB. Now that supposedly shared image is half a gig. "Don't worry, it's shared anyway". Expect when it isn't. And when it is, it still slow us down in bringing up new nodes. And guess what, turns out that not everyone is starting from the same point, so there is a multitude of 'shared' images now. Storage is cheap, but bandwidth may not be. And it still takes time to download. Try to keep your containers as small as possible for as long as possible. Your tech debt may grow slower that way.

peterwwillis 7y ago

It's not a few megabytes, it's a few hundred to a thousand megabytes saved, on average. Multiplied by a thousand containers, and much larger layers on build servers, plus bandwidth, it makes a difference. Worst case for slower processes, things take longer. Worst case for more disk use, things start crashing. For general cases, the former is preferable.

sofaofthedamned 7y ago
No, it's not. With samepage merging it's nothing, let alone docker only loading the image once.
- atombender 7y ago
  This assumes everyone is on exactly the same version. Larger organizations can afford to appoint SREs who can institute a broad range of security and optimization policies (including "all apps should use the same Alpine base image version") and enforce them programmatically as well as ensure that apps are continually updated to match them. That kind of thing is expensive, resource-wise, for smaller organizations.
- peterwwillis 7y ago
  I'm not talking about image waste, I'm talking a single box with a half dozen different base image versions and a slew of extra packages thrown in. Every app built against glibc is going to get bigger, and not all Ubuntu packages are built with small size in mind. Compare these systems with Ubuntu vs Alpine base images and the average size for an Ubuntu ecosystem is substantially larger.

whatshisface 7y ago

Why is musl slower?

saagarjha 7y ago
It's much simpler and more oriented towards POSIX compatibility than performance.
Nelson69 7y ago
I don't want to disparage any project or guess the motivation but there have been some undercurrents of anti-GPL sentiment at times and anti-complexity. Folks have sort of backed it up by posting some links to things that aren't as you might think. GLIBC in particular looks nothing like how you might imagine it. You can look at strlen in the K&R book and it's beautiful, like a textbook: int strlen(char s[]) { int i; i = 0; while( s[i] != '\0') ++i return i; } then you look at: https://sourceware.org/git/?p=glibc.git;a=blob;f=string/strlen.c;h=279749659f2be51a1f643a9aafdc4e1d3320136d;hb=HEAD https://sourceware.org/git/?p=glibc.git;a=blob;f=string/strl... and your mind will sort of explode for a bit. The difference is the GLIBC version is dramatically faster; take the comments away and most of us wouldn't even know that's strlen. It's more complex, no question, but it's much faster. GLIBC is full of stuff like that. qsort and memcpy are non-obvious to many folks. It's not complexity for no reason, you'd be challenged to build a better qsort than the one in glibc, it's not easy.
- username223 7y ago
  This reminds me of a classic blog post: http://ridiculousfish.com/blog/posts/old-age-and-treachery.html All of those old Unix programs aren't fast by being simple and clean on the inside. Old age and treachery...
  - globuous 7y ago
    And this guy also https://news.ycombinator.com/item?id=14543536 https://news.ycombinator.com/item?id=14543536 (which comes with this: https://www.gnu.org/prep/standards/standards.html#Reading-Non_002dFree-Code https://www.gnu.org/prep/standards/standards.html#Reading-No...)

bogomipz 7y ago

>"It’s awesome that the images are pretty small, but a wide variety of software has been shown to run noticeably slower on Alpine compared to other distributions, in part due to its usage of musl instead of glibc" Might you have any citation(s) for this? I have not heard this before. Could you elaborate on what some the "wide variety of software" is? Thanks.

creatornator 7y ago
I've heard that having shared libraries like glibc allows for commonly shared hotpaths to remain in the CPU cache, making it faster. Whereas in musl, since the binaries all have their own copies of procedures, they are more often kicked out of cache. I don't imagine it has a big impact on a container where you are only running one or two binaries. Perhaps in a desktop environment with hundreds or thousands of different programs running, it might make a difference.

cs02rm0 7y ago

Because it's tiny, I tend to default to Alpine and then move away from it where necessary. Rather than worrying about potential CPU performance requirements upfront - premature optimisation and all that.

mikepurvis 7y ago
Isn't there an argument that using Alpine instead of something like Ubuntu is premature optimization for space?
- akvadrako 7y ago
  It's faster in terms of development time. Faster to install packages and to push to docker repos.
  - icebraining 7y ago
    Why? You should only need to push the base image once. Then only upper layers will have to be pushed.
- draw_down 7y ago
  Optimization is only premature when other people do it. :)
emilfihlman 7y ago
Making it smaller than necessary is a bigger premature optimisation, though. Rather than worrying about potential disk space issues upfront just use glibc and everything works fast and fine.

bayesian_horse 7y ago

The small size of an image can be an actual issue. Certainly for embedded devices, probably also for clusters. For me it's mostly a non-issue. But things like redis or nginx work nicely with alpine as a base. And more than likely, whatever I want to containerize has already been containerized for Alpine. If not, getting something to work on Alpine may just not be worth it...

dnautics 7y ago

It's not just a few megabytes, it's more like hundreds. If you're moving an image across a bad network (like the internet) that could be hundreds of milliseconds.

geezerjay 7y ago
I would also add that data transfers cost money, and having to transfer a few hundred MBs each time a container image is passed around can reflect in the expenses.