3 ms·

> Image layers are cached and shared! If you have one image based on Ubuntu in your stack, you may as well base them all on Ubuntu, because you only need to dow

> Image layers are cached and shared! If you have one image based on Ubuntu in your stack, you may as well base them all on Ubuntu, because you only need to download (and store!) the common base image once. AIUI that's only helps if everything uses exactly the same base image. Every time the base image gets updated, your images will diverge as they're rebuilt. > You opted out of the most well-tested and maintained versions of binaries, Strictly true, though Alpine is pretty widely used and tested these days. > and forced all software in your image to link with musl instead of glibc. Well... Yeah, that's how that works. > Congratulations on creating an ongoing headache for yourself. You've increased maintenance costs I keep hearing this, but in years of shipping Alpine based images into prod the problems never materialized. > and probably even decreased security by preferring a less-audited codebase. Or, since we're handwaving, you've increased security because musl is better hardened and an attacker's binaries are less likely to work on it.

dharmab 3y ago
> I keep hearing this, but in years of shipping Alpine based images into prod the problems never materialized. DNS was the big one; for many years musl's implementation of DNS was both divergent from glibc, and in some cases, broken due to lack of TCP support. This only got fixed earlier this year: https://gitlab.alpinelinux.org/alpine/tsc/-/issues/43 https://gitlab.alpinelinux.org/alpine/tsc/-/issues/43 TCP support can be a really big deal for reliability at very large scales. When I led the container infrastructure team at $BIGTECHCOMPANY we eventually published a recommendation against using Alpine because musl's resolver was demonstrably less reliable than glibc's- we had the production data to show it. The other case was when developers copied binaries compiled for glibc into alpine containers, which is user error to be fair, but it was easier to tell everyone to use glibc instead of teaching thousands of Python devs about how the compiler and linker works ;_;
chatmasta 3y ago
> AIUI that's only helps if everything uses exactly the same base image. Every time the base image gets updated, your images will diverge as they're rebuilt. Yes this is true. In our stack we pin all base images to a common hash, which makes builds fast and also increases reliability of CI pipelines since we don't get unexpected changes to builds. The tradeoff is you need to be deliberate about updating the pinned image with new security patches, but this feature is available out-of-the-box with update bots like Renovate. > the problems never materialized What about every hour you spent recompiling binaries to make them work with musl? That's what I meant by maintenance cost - every time you add a new binary into your minimalist image regime, you need to either pay the cost of making it work with musl, or abandon the effort, effectively nullifying the benefits of your other images using Alpine (since once you have one image based on a larger base image, you may as well base them all on it). This is a kind of Stockholm Syndrome where the cargo-culted optimization becomes forever embedded in your software requirements even when the options may have changed (for example, once debian-slim was introduced, there was little reason to continue using Alpine).
- yjftsjthsd-h 3y ago
  > What about every hour you spent recompiling binaries to make them work with musl? We were a mostly non-Python shop, and what Python images we did use, IIRC, either 1. didn't use libraries with native code so not a problem, 2. just `apk add`ed the Alpine-provided libraries, or 2. just used the Debian based image for that particular image. I will grant that if you're working in a Python-heavy environment the calculus is likely different. > That's what I meant by maintenance cost - every time you add a new binary into your minimalist image regime, you need to either pay the cost of making it work with musl, or abandon the effort, effectively nullifying the benefits of your other images using Alpine (since once you have one image based on a larger base image, you may as well base them all on it). IME it just wasn't that much cost, and it dropped off even more once musl wheels started becoming normal. And we had enough churn in images that basing everything on ex. a single static version of Debian was 1. never going to happen, and 2. wouldn't have reduced our image sizes since the base image wasn't constant anyways. > This is a kind of Stockholm Syndrome where the cargo-culted optimization becomes forever embedded in your software requirements even when the options may have changed Or, it's a perfectly reasonable rule of thumb and if actual evidence shows that something else is better you can pick it up then. I mean, the same thing goes in reverse, right? You've cargo culted "Alpine slow" and have embedded that in your stack even though the reasons for it no longer exist. > (for example, once debian-slim was introduced, there was little reason to continue using Alpine). Alpine is still smaller. Debian shipping slim images reduced the gap, but it's still there, and installing packages will exacerbate it (since Alpine packages also seem to be smaller than the Debian equivalents.
jauntywundrkind 3y ago
> I keep hearing this, but in years of shipping Alpine based images into prod the problems never materialized. Postgres has very subtly different sort ordering when we use the alpine based images. That the dev trying the upgrade locally happened to pick alpine got what should have been a very simple couple person-days upgrade backlogged. And drawn out & stumbled over. Multiple people wading through a long series of changelogs. We eventually figured out what was going on, but it was a miserable long soggy sad trudge to get ourselves back on track. All because someone used a not-so-standard library. Musl causes a more regular pain of node add-ons needing to be specially compiled, but at least we typically see these coming. Time lost, but at least predictable time lost. One other gripe: musl also is not designed for server workloads at all. It's designed for space efficiency & small systems. Sometimes memory can be a dominant cost or a crucial factor for online services, but for most workloads, you are probably misoptimizing your system by misapplying what was intended as a niche standard library to your system.
- sgarland 3y ago
  I think the Postgres weirdness is due to collations, which tbf are a nightmare to deal with in DBs on their own. But in any case, it would be tremendously irritating to troubleshoot.

dharmab 3y ago

> I keep hearing this, but in years of shipping Alpine based images into prod the problems never materialized. DNS was the big one; for many years musl's implementation of DNS was both divergent from glibc, and in some cases, broken due to lack of TCP support. This only got fixed earlier this year: https://gitlab.alpinelinux.org/alpine/tsc/-/issues/43 https://gitlab.alpinelinux.org/alpine/tsc/-/issues/43 TCP support can be a really big deal for reliability at very large scales. When I led the container infrastructure team at $BIGTECHCOMPANY we eventually published a recommendation against using Alpine because musl's resolver was demonstrably less reliable than glibc's- we had the production data to show it. The other case was when developers copied binaries compiled for glibc into alpine containers, which is user error to be fair, but it was easier to tell everyone to use glibc instead of teaching thousands of Python devs about how the compiler and linker works ;_;

chatmasta 3y ago

> AIUI that's only helps if everything uses exactly the same base image. Every time the base image gets updated, your images will diverge as they're rebuilt. Yes this is true. In our stack we pin all base images to a common hash, which makes builds fast and also increases reliability of CI pipelines since we don't get unexpected changes to builds. The tradeoff is you need to be deliberate about updating the pinned image with new security patches, but this feature is available out-of-the-box with update bots like Renovate. > the problems never materialized What about every hour you spent recompiling binaries to make them work with musl? That's what I meant by maintenance cost - every time you add a new binary into your minimalist image regime, you need to either pay the cost of making it work with musl, or abandon the effort, effectively nullifying the benefits of your other images using Alpine (since once you have one image based on a larger base image, you may as well base them all on it). This is a kind of Stockholm Syndrome where the cargo-culted optimization becomes forever embedded in your software requirements even when the options may have changed (for example, once debian-slim was introduced, there was little reason to continue using Alpine).

yjftsjthsd-h 3y ago
> What about every hour you spent recompiling binaries to make them work with musl? We were a mostly non-Python shop, and what Python images we did use, IIRC, either 1. didn't use libraries with native code so not a problem, 2. just `apk add`ed the Alpine-provided libraries, or 2. just used the Debian based image for that particular image. I will grant that if you're working in a Python-heavy environment the calculus is likely different. > That's what I meant by maintenance cost - every time you add a new binary into your minimalist image regime, you need to either pay the cost of making it work with musl, or abandon the effort, effectively nullifying the benefits of your other images using Alpine (since once you have one image based on a larger base image, you may as well base them all on it). IME it just wasn't that much cost, and it dropped off even more once musl wheels started becoming normal. And we had enough churn in images that basing everything on ex. a single static version of Debian was 1. never going to happen, and 2. wouldn't have reduced our image sizes since the base image wasn't constant anyways. > This is a kind of Stockholm Syndrome where the cargo-culted optimization becomes forever embedded in your software requirements even when the options may have changed Or, it's a perfectly reasonable rule of thumb and if actual evidence shows that something else is better you can pick it up then. I mean, the same thing goes in reverse, right? You've cargo culted "Alpine slow" and have embedded that in your stack even though the reasons for it no longer exist. > (for example, once debian-slim was introduced, there was little reason to continue using Alpine). Alpine is still smaller. Debian shipping slim images reduced the gap, but it's still there, and installing packages will exacerbate it (since Alpine packages also seem to be smaller than the Debian equivalents.

jauntywundrkind 3y ago

> I keep hearing this, but in years of shipping Alpine based images into prod the problems never materialized. Postgres has very subtly different sort ordering when we use the alpine based images. That the dev trying the upgrade locally happened to pick alpine got what should have been a very simple couple person-days upgrade backlogged. And drawn out & stumbled over. Multiple people wading through a long series of changelogs. We eventually figured out what was going on, but it was a miserable long soggy sad trudge to get ourselves back on track. All because someone used a not-so-standard library. Musl causes a more regular pain of node add-ons needing to be specially compiled, but at least we typically see these coming. Time lost, but at least predictable time lost. One other gripe: musl also is not designed for server workloads at all. It's designed for space efficiency & small systems. Sometimes memory can be a dominant cost or a crucial factor for online services, but for most workloads, you are probably misoptimizing your system by misapplying what was intended as a niche standard library to your system.

sgarland 3y ago
I think the Postgres weirdness is due to collations, which tbf are a nightmare to deal with in DBs on their own. But in any case, it would be tremendously irritating to troubleshoot.