6 ms·

Using Alpine can make Python Docker builds 50× slower

61 points by bomewish 3y ago

ehutch79 3y ago
I thought python now provided wheels for alpine? Wouldn’t that make this article outdated? Also, real question, how often are you rebuilding docker images (not containers) that a few seconds matter?
avtar 3y ago
Having run into some of the musl-related issues mentioned in the article, I now opt for Debian-based images even for non-python use cases. Having to spend time troubleshooting Alpine specific issues just doesn’t seem worth it.
- dathinab 3y ago
  In my experience most issues are not really musel issues but incorrect usage of libc where glibc used some "magic" to make that incorrect usage somehow somewhat work, at the cost of higher complexity which more then one time lead to security issues due to a larger attack surface. So in general I would recommend testing with musel and fix whatever issues you run into in a non-musel specific way. Through then I guess at work many people do not have time/resources for stuff like that.
  - avtar 3y ago
    > So in general I would recommend testing with musel and fix whatever issues you run into in a non-musel specific way. But again, do all this just to save anywhere between 50-200mb of space compared to Debian slim images. I'd rather just use the latter and get on with my day.
    - dathinab 3y ago
      it's not to save 50-200mb but to correctly use libc to avoid any unexpected situations with glibc down the line, e.g. when a change needed to be done for security reasons or running it on a more hardened system might lead to similar slow downs as with musel
- asylteltine 3y ago
  I use scratch or distroless with a static binary. No nonsense needed! I love go.
chatmasta 3y ago
There is perhaps no greater example of a cargo-culted non-optimization than the choice of Alpine as a base image. This has cumulatively wasted millions of hours of developer time. And why? To reduce the base image size. Image layers are cached and shared! If you have one image based on Ubuntu in your stack, you may as well base them all on Ubuntu, because you only need to download (and store!) the common base image once. And if you're such a purist that you've avoided anything but Alpine base images, what have you actually gained? Your deploy time is slightly faster? Your first cold boot is slightly faster? Outside of a few serverless use-cases, this is basically meaningless and almost never a real bottleneck (and even for serverless, there is probably some level of image caching across boots). Meanwhile what did you give up? You opted out of the most well-tested and maintained versions of binaries, and forced all software in your image to link with musl instead of glibc. Congratulations on creating an ongoing headache for yourself. You've increased maintenance costs and probably even decreased security by preferring a less-audited codebase. I suppose one positive side-effect of the Alpine cargo-cult is that some of its more technically-inclined adherents at least contributed back to projects to improve their cross-compilation toolchains. But otherwise, it's just been a huge waste of time for pretty much everybody.
- dharmab 3y ago
  > If you have one image based on Ubuntu in your stack, you may as well base them all on Ubuntu, because you only need to download (and store!) the common base image once This is only true if your infrastructure is static. If your infrastructure is highly elastic, image size has an impact on your time to scale up. A great example is using spot instances; in my line of work we need to create thousands of spot instances to run many millions of time-sensitive batch jobs at certain times and dates, then scale those instances down rapidly to reduce the bill. Of course, there are better choices than Alpine to optimize image size. Distroless (https://github.com/GoogleContainerTools/distroless https://github.com/GoogleContainerTools/distroless) is a good example.
  - chatmasta 3y ago
    That's true. But the size difference between alpine and debian:*-slim (for example) is on the order of < 20mb. Datacenter hardware can download this in milliseconds. There may be some rare scenarios where this is a meaningful delay, but usually provisioning time is bottlenecked somewhere else. Heck, it probably takes longer to download the VM image (and if you're optimizing for provisioning speed, you shouldn't be downloading Docker images on boot anyway - you should be baking VM images ahead of time.)
yjftsjthsd-h 3y ago
Seems a little disingenuous to say that Alpine is slow because it can't use wheels, and only in a little note at the bottom mention that that's not true anymore.
- sp332 3y ago
  The blog post is three years old. The update is more recent.
  - yjftsjthsd-h 3y ago
    If you're going to update a blog post to mention that it's inaccurate, it would seem preferable to put that at the top (so people don't skim it and walk away misinformed) or edit the document in place (which in this case would result in a very short page, since the meat of the post is the part that's wrong now).
ofek 3y ago
As mentioned in an addendum at the bottom of the page, wheels can now target Alpine. You can try it yourself using the example package from OP: docker run --rm python:3.11-alpine pip install pandas
deleted 3y ago
[deleted]
oceanplexian 3y ago
I mean, the easy solution is don’t use Python if the container image size is important to you. Build it in Go and you don’t need to drag in a million weird dependencies. The whole idea you need a 300MB container image to run a simple Python program is a perfect demonstration of everything that’s wrong with software development.
- tensor 3y ago
  It's not that simple. Likely the root of this is that alpine uses musl which by default ships an allocator that is extremely slow in multithreaded code. I've seen Go programs be much much slower under alpine for this reason as well. Given how often this comes up, I think it's high time that alpine replaced the default allocator with something more modern.
- asylteltine 3y ago
  > I mean, the easy solution is don’t use Python :) People will be much better off and happier if they switch to Go for 99% of tasks. I’ll understand if it’s python for machine learning.
  - whalesalad 3y ago
    Go is such an ugly language. It has it’s place but it’s not the end-all be-all.
    - asylteltine 3y ago
      How? It’s SO easy to read and you don’t need semicolons. How can it possibly be ugly?
      - sgarland 3y ago
        if err != nil
        smegsicle 3y ago
dang 3y ago
Related: Alpine makes Python Docker builds slower, and images larger - https://news.ycombinator.com/item?id=22182226 https://news.ycombinator.com/item?id=22182226 - Jan 2020 (149 comments)

61 points by bomewish 3y ago

I thought python now provided wheels for alpine? Wouldn’t that make this article outdated? Also, real question, how often are you rebuilding docker images (not containers) that a few seconds matter?

avtar 3y ago

Having run into some of the musl-related issues mentioned in the article, I now opt for Debian-based images even for non-python use cases. Having to spend time troubleshooting Alpine specific issues just doesn’t seem worth it.

dathinab 3y ago
In my experience most issues are not really musel issues but incorrect usage of libc where glibc used some "magic" to make that incorrect usage somehow somewhat work, at the cost of higher complexity which more then one time lead to security issues due to a larger attack surface. So in general I would recommend testing with musel and fix whatever issues you run into in a non-musel specific way. Through then I guess at work many people do not have time/resources for stuff like that.
- avtar 3y ago
  > So in general I would recommend testing with musel and fix whatever issues you run into in a non-musel specific way. But again, do all this just to save anywhere between 50-200mb of space compared to Debian slim images. I'd rather just use the latter and get on with my day.
  - dathinab 3y ago
    it's not to save 50-200mb but to correctly use libc to avoid any unexpected situations with glibc down the line, e.g. when a change needed to be done for security reasons or running it on a more hardened system might lead to similar slow downs as with musel
asylteltine 3y ago
I use scratch or distroless with a static binary. No nonsense needed! I love go.

chatmasta 3y ago

There is perhaps no greater example of a cargo-culted non-optimization than the choice of Alpine as a base image. This has cumulatively wasted millions of hours of developer time. And why? To reduce the base image size. Image layers are cached and shared! If you have one image based on Ubuntu in your stack, you may as well base them all on Ubuntu, because you only need to download (and store!) the common base image once. And if you're such a purist that you've avoided anything but Alpine base images, what have you actually gained? Your deploy time is slightly faster? Your first cold boot is slightly faster? Outside of a few serverless use-cases, this is basically meaningless and almost never a real bottleneck (and even for serverless, there is probably some level of image caching across boots). Meanwhile what did you give up? You opted out of the most well-tested and maintained versions of binaries, and forced all software in your image to link with musl instead of glibc. Congratulations on creating an ongoing headache for yourself. You've increased maintenance costs and probably even decreased security by preferring a less-audited codebase. I suppose one positive side-effect of the Alpine cargo-cult is that some of its more technically-inclined adherents at least contributed back to projects to improve their cross-compilation toolchains. But otherwise, it's just been a huge waste of time for pretty much everybody.

dharmab 3y ago
> If you have one image based on Ubuntu in your stack, you may as well base them all on Ubuntu, because you only need to download (and store!) the common base image once This is only true if your infrastructure is static. If your infrastructure is highly elastic, image size has an impact on your time to scale up. A great example is using spot instances; in my line of work we need to create thousands of spot instances to run many millions of time-sensitive batch jobs at certain times and dates, then scale those instances down rapidly to reduce the bill. Of course, there are better choices than Alpine to optimize image size. Distroless (https://github.com/GoogleContainerTools/distroless https://github.com/GoogleContainerTools/distroless) is a good example.
- chatmasta 3y ago
  That's true. But the size difference between alpine and debian:*-slim (for example) is on the order of < 20mb. Datacenter hardware can download this in milliseconds. There may be some rare scenarios where this is a meaningful delay, but usually provisioning time is bottlenecked somewhere else. Heck, it probably takes longer to download the VM image (and if you're optimizing for provisioning speed, you shouldn't be downloading Docker images on boot anyway - you should be baking VM images ahead of time.)

yjftsjthsd-h 3y ago

Seems a little disingenuous to say that Alpine is slow because it can't use wheels, and only in a little note at the bottom mention that that's not true anymore.

sp332 3y ago
The blog post is three years old. The update is more recent.
- yjftsjthsd-h 3y ago
  If you're going to update a blog post to mention that it's inaccurate, it would seem preferable to put that at the top (so people don't skim it and walk away misinformed) or edit the document in place (which in this case would result in a very short page, since the meat of the post is the part that's wrong now).

ofek 3y ago

As mentioned in an addendum at the bottom of the page, wheels can now target Alpine. You can try it yourself using the example package from OP: docker run --rm python:3.11-alpine pip install pandas

deleted 3y ago

[deleted]

oceanplexian 3y ago

I mean, the easy solution is don’t use Python if the container image size is important to you. Build it in Go and you don’t need to drag in a million weird dependencies. The whole idea you need a 300MB container image to run a simple Python program is a perfect demonstration of everything that’s wrong with software development.

tensor 3y ago
It's not that simple. Likely the root of this is that alpine uses musl which by default ships an allocator that is extremely slow in multithreaded code. I've seen Go programs be much much slower under alpine for this reason as well. Given how often this comes up, I think it's high time that alpine replaced the default allocator with something more modern.
asylteltine 3y ago
> I mean, the easy solution is don’t use Python :) People will be much better off and happier if they switch to Go for 99% of tasks. I’ll understand if it’s python for machine learning.
- whalesalad 3y ago
  Go is such an ugly language. It has it’s place but it’s not the end-all be-all.
  - asylteltine 3y ago
    How? It’s SO easy to read and you don’t need semicolons. How can it possibly be ugly?
    - sgarland 3y ago
      if err != nil
      - smegsicle 3y ago

dang 3y ago

Related: Alpine makes Python Docker builds slower, and images larger - https://news.ycombinator.com/item?id=22182226 https://news.ycombinator.com/item?id=22182226 - Jan 2020 (149 comments)