13 ms·

There is perhaps no greater example of a cargo-culted non-optimization than the choice of Alpine as a base image. This has cumulatively wasted millions of hours

There is perhaps no greater example of a cargo-culted non-optimization than the choice of Alpine as a base image. This has cumulatively wasted millions of hours of developer time. And why? To reduce the base image size. Image layers are cached and shared! If you have one image based on Ubuntu in your stack, you may as well base them all on Ubuntu, because you only need to download (and store!) the common base image once. And if you're such a purist that you've avoided anything but Alpine base images, what have you actually gained? Your deploy time is slightly faster? Your first cold boot is slightly faster? Outside of a few serverless use-cases, this is basically meaningless and almost never a real bottleneck (and even for serverless, there is probably some level of image caching across boots). Meanwhile what did you give up? You opted out of the most well-tested and maintained versions of binaries, and forced all software in your image to link with musl instead of glibc. Congratulations on creating an ongoing headache for yourself. You've increased maintenance costs and probably even decreased security by preferring a less-audited codebase. I suppose one positive side-effect of the Alpine cargo-cult is that some of its more technically-inclined adherents at least contributed back to projects to improve their cross-compilation toolchains. But otherwise, it's just been a huge waste of time for pretty much everybody.

dharmab 3y ago
> If you have one image based on Ubuntu in your stack, you may as well base them all on Ubuntu, because you only need to download (and store!) the common base image once This is only true if your infrastructure is static. If your infrastructure is highly elastic, image size has an impact on your time to scale up. A great example is using spot instances; in my line of work we need to create thousands of spot instances to run many millions of time-sensitive batch jobs at certain times and dates, then scale those instances down rapidly to reduce the bill. Of course, there are better choices than Alpine to optimize image size. Distroless (https://github.com/GoogleContainerTools/distroless https://github.com/GoogleContainerTools/distroless) is a good example.
- chatmasta 3y ago
  That's true. But the size difference between alpine and debian:*-slim (for example) is on the order of < 20mb. Datacenter hardware can download this in milliseconds. There may be some rare scenarios where this is a meaningful delay, but usually provisioning time is bottlenecked somewhere else. Heck, it probably takes longer to download the VM image (and if you're optimizing for provisioning speed, you shouldn't be downloading Docker images on boot anyway - you should be baking VM images ahead of time.)
  - dharmab 3y ago
    Baking the VM images trades a lot of flexibility for some latency, and it doesn't apply to container-as-a-service infrastructure where you don't control the VM. My point is you can get most of the way there with slim/distroless without making the tradeoffs of Alpine, and that's probably enough for most use cases.
  - pella 3y ago
    > But the size difference between alpine and debian:-slim (for example) is on the order of < 20mb. compressed size(amd64): 17.59MB vs. 45.35MB ( > 2.5x ) uncompressed size(amd64):52.00MB vs. 130.00MB ( > 2.5x ) $ docker images python:3.12.1* REPOSITORY TAG IMAGE ID CREATED SIZE python 3.12.1-alpine3.19 0108959914c2 2 weeks ago 51.7MB python 3.12.1-slim-bookworm 7c5c05471c3a 2 weeks ago 130MB**
    - mikeravkine 3y ago
      There is no difference in download time for 20mb vs 50mb in a datacenter, but the amount of time lost to alpine being "quirky" is incalculable.
    - dharmab 3y ago
      The compressed size is what matters when discussing image download time- CPUs made after around 2010 can uncompress files ludicrously quickly.
    - chatmasta 3y ago
- mkesper 3y ago
  Distroless is Debian packages with worse security and without package management.
  - lkjasdlfwfasd 3y ago
    > without package management. distroless is for the TERMINAL layer, you should already be done with the package manager prior to entering distroless.
yjftsjthsd-h 3y ago
> Image layers are cached and shared! If you have one image based on Ubuntu in your stack, you may as well base them all on Ubuntu, because you only need to download (and store!) the common base image once. AIUI that's only helps if everything uses exactly the same base image. Every time the base image gets updated, your images will diverge as they're rebuilt. > You opted out of the most well-tested and maintained versions of binaries, Strictly true, though Alpine is pretty widely used and tested these days. > and forced all software in your image to link with musl instead of glibc. Well... Yeah, that's how that works. > Congratulations on creating an ongoing headache for yourself. You've increased maintenance costs I keep hearing this, but in years of shipping Alpine based images into prod the problems never materialized. > and probably even decreased security by preferring a less-audited codebase. Or, since we're handwaving, you've increased security because musl is better hardened and an attacker's binaries are less likely to work on it.
- dharmab 3y ago
  > I keep hearing this, but in years of shipping Alpine based images into prod the problems never materialized. DNS was the big one; for many years musl's implementation of DNS was both divergent from glibc, and in some cases, broken due to lack of TCP support. This only got fixed earlier this year: https://gitlab.alpinelinux.org/alpine/tsc/-/issues/43 https://gitlab.alpinelinux.org/alpine/tsc/-/issues/43 TCP support can be a really big deal for reliability at very large scales. When I led the container infrastructure team at $BIGTECHCOMPANY we eventually published a recommendation against using Alpine because musl's resolver was demonstrably less reliable than glibc's- we had the production data to show it. The other case was when developers copied binaries compiled for glibc into alpine containers, which is user error to be fair, but it was easier to tell everyone to use glibc instead of teaching thousands of Python devs about how the compiler and linker works ;_;
- chatmasta 3y ago
  > AIUI that's only helps if everything uses exactly the same base image. Every time the base image gets updated, your images will diverge as they're rebuilt. Yes this is true. In our stack we pin all base images to a common hash, which makes builds fast and also increases reliability of CI pipelines since we don't get unexpected changes to builds. The tradeoff is you need to be deliberate about updating the pinned image with new security patches, but this feature is available out-of-the-box with update bots like Renovate. > the problems never materialized What about every hour you spent recompiling binaries to make them work with musl? That's what I meant by maintenance cost - every time you add a new binary into your minimalist image regime, you need to either pay the cost of making it work with musl, or abandon the effort, effectively nullifying the benefits of your other images using Alpine (since once you have one image based on a larger base image, you may as well base them all on it). This is a kind of Stockholm Syndrome where the cargo-culted optimization becomes forever embedded in your software requirements even when the options may have changed (for example, once debian-slim was introduced, there was little reason to continue using Alpine).
sgarland 3y ago
100% agreed. For every bad example posted where the replies are full of, "yeah, but that was patched X months/years ago," I guarantee there are more edge cases waiting to be found. Alpine is not worth the engineering headache, full stop. "But it's a _correct_ implementation, glibc is buggy!" Yeah, and it's been the accepted standard since forever. Everything expects it, and is used to it.
- dharmab 3y ago
  You're getting downvoted for a pragmatic observation. The documented "correct" approach is not necessarily the most likely to work reliably. https://en.m.wikipedia.org/wiki/Protocol_ossification https://en.m.wikipedia.org/wiki/Protocol_ossification
deleted 3y ago
[deleted]
Avamander 3y ago
You've forgotten one additional nuance - security. It's easier to look out for outdated/vulnerable components. A slim image also makes it quite a bit more annoying for attackers, if you don't give them all the tools they need straight away.
- dharmab 3y ago
  There's usually enough gadgets in slim images to infiltrate/exfiltrate, especially if you include package managers in your final image. Using a multi-stage build with a final image from scratch is a good practice where possible, especially now that you can attach debugging sidecar containers to your application as needed.
bravetraveler 3y ago
Registries are painfully easy to host, too. One person can manage that on behalf of everyone expected to maintain this alpine ritual. Scaling can be solved in a way that keeps familiarity

dharmab 3y ago

> If you have one image based on Ubuntu in your stack, you may as well base them all on Ubuntu, because you only need to download (and store!) the common base image once This is only true if your infrastructure is static. If your infrastructure is highly elastic, image size has an impact on your time to scale up. A great example is using spot instances; in my line of work we need to create thousands of spot instances to run many millions of time-sensitive batch jobs at certain times and dates, then scale those instances down rapidly to reduce the bill. Of course, there are better choices than Alpine to optimize image size. Distroless (https://github.com/GoogleContainerTools/distroless https://github.com/GoogleContainerTools/distroless) is a good example.

chatmasta 3y ago
That's true. But the size difference between alpine and debian:*-slim (for example) is on the order of < 20mb. Datacenter hardware can download this in milliseconds. There may be some rare scenarios where this is a meaningful delay, but usually provisioning time is bottlenecked somewhere else. Heck, it probably takes longer to download the VM image (and if you're optimizing for provisioning speed, you shouldn't be downloading Docker images on boot anyway - you should be baking VM images ahead of time.)
- dharmab 3y ago
  Baking the VM images trades a lot of flexibility for some latency, and it doesn't apply to container-as-a-service infrastructure where you don't control the VM. My point is you can get most of the way there with slim/distroless without making the tradeoffs of Alpine, and that's probably enough for most use cases.
- pella 3y ago
  > But the size difference between alpine and debian:-slim (for example) is on the order of < 20mb. compressed size(amd64): 17.59MB vs. 45.35MB ( > 2.5x ) uncompressed size(amd64):52.00MB vs. 130.00MB ( > 2.5x ) $ docker images python:3.12.1* REPOSITORY TAG IMAGE ID CREATED SIZE python 3.12.1-alpine3.19 0108959914c2 2 weeks ago 51.7MB python 3.12.1-slim-bookworm 7c5c05471c3a 2 weeks ago 130MB**
  - mikeravkine 3y ago
    There is no difference in download time for 20mb vs 50mb in a datacenter, but the amount of time lost to alpine being "quirky" is incalculable.
  - dharmab 3y ago
    The compressed size is what matters when discussing image download time- CPUs made after around 2010 can uncompress files ludicrously quickly.
  - chatmasta 3y ago
mkesper 3y ago
Distroless is Debian packages with worse security and without package management.
- lkjasdlfwfasd 3y ago
  > without package management. distroless is for the TERMINAL layer, you should already be done with the package manager prior to entering distroless.

yjftsjthsd-h 3y ago

> Image layers are cached and shared! If you have one image based on Ubuntu in your stack, you may as well base them all on Ubuntu, because you only need to download (and store!) the common base image once. AIUI that's only helps if everything uses exactly the same base image. Every time the base image gets updated, your images will diverge as they're rebuilt. > You opted out of the most well-tested and maintained versions of binaries, Strictly true, though Alpine is pretty widely used and tested these days. > and forced all software in your image to link with musl instead of glibc. Well... Yeah, that's how that works. > Congratulations on creating an ongoing headache for yourself. You've increased maintenance costs I keep hearing this, but in years of shipping Alpine based images into prod the problems never materialized. > and probably even decreased security by preferring a less-audited codebase. Or, since we're handwaving, you've increased security because musl is better hardened and an attacker's binaries are less likely to work on it.

dharmab 3y ago
> I keep hearing this, but in years of shipping Alpine based images into prod the problems never materialized. DNS was the big one; for many years musl's implementation of DNS was both divergent from glibc, and in some cases, broken due to lack of TCP support. This only got fixed earlier this year: https://gitlab.alpinelinux.org/alpine/tsc/-/issues/43 https://gitlab.alpinelinux.org/alpine/tsc/-/issues/43 TCP support can be a really big deal for reliability at very large scales. When I led the container infrastructure team at $BIGTECHCOMPANY we eventually published a recommendation against using Alpine because musl's resolver was demonstrably less reliable than glibc's- we had the production data to show it. The other case was when developers copied binaries compiled for glibc into alpine containers, which is user error to be fair, but it was easier to tell everyone to use glibc instead of teaching thousands of Python devs about how the compiler and linker works ;_;
chatmasta 3y ago
> AIUI that's only helps if everything uses exactly the same base image. Every time the base image gets updated, your images will diverge as they're rebuilt. Yes this is true. In our stack we pin all base images to a common hash, which makes builds fast and also increases reliability of CI pipelines since we don't get unexpected changes to builds. The tradeoff is you need to be deliberate about updating the pinned image with new security patches, but this feature is available out-of-the-box with update bots like Renovate. > the problems never materialized What about every hour you spent recompiling binaries to make them work with musl? That's what I meant by maintenance cost - every time you add a new binary into your minimalist image regime, you need to either pay the cost of making it work with musl, or abandon the effort, effectively nullifying the benefits of your other images using Alpine (since once you have one image based on a larger base image, you may as well base them all on it). This is a kind of Stockholm Syndrome where the cargo-culted optimization becomes forever embedded in your software requirements even when the options may have changed (for example, once debian-slim was introduced, there was little reason to continue using Alpine).

sgarland 3y ago

100% agreed. For every bad example posted where the replies are full of, "yeah, but that was patched X months/years ago," I guarantee there are more edge cases waiting to be found. Alpine is not worth the engineering headache, full stop. "But it's a _correct_ implementation, glibc is buggy!" Yeah, and it's been the accepted standard since forever. Everything expects it, and is used to it.

dharmab 3y ago
You're getting downvoted for a pragmatic observation. The documented "correct" approach is not necessarily the most likely to work reliably. https://en.m.wikipedia.org/wiki/Protocol_ossification https://en.m.wikipedia.org/wiki/Protocol_ossification

deleted 3y ago

[deleted]

Avamander 3y ago

You've forgotten one additional nuance - security. It's easier to look out for outdated/vulnerable components. A slim image also makes it quite a bit more annoying for attackers, if you don't give them all the tools they need straight away.

dharmab 3y ago
There's usually enough gadgets in slim images to infiltrate/exfiltrate, especially if you include package managers in your final image. Using a multi-stage build with a final image from scratch is a good practice where possible, especially now that you can attach debugging sidecar containers to your application as needed.

bravetraveler 3y ago

Registries are painfully easy to host, too. One person can manage that on behalf of everyone expected to maintain this alpine ritual. Scaling can be solved in a way that keeps familiarity