13 ms·

Alpine makes Python Docker builds slower, and images larger

306 points by itamarst 7y ago

gtirloni 7y ago
Not really Alpine's problem that there aren't upstream wheels for your dependencies. I think you're being realistic with your suggestion but you're barking at the wrong tree.
- hprotagonist 7y ago
  musl seems very niche. That’s an ecosystem problem; that is on Alpine for swimming upstream.
- wzdd 7y ago
  Multi-stage container builds are also not exactly unusual. You would generally not want an entire C build system sitting around in your container.
  - hinkley 7y ago
    Isn’t this also a problem of requiring the compiler toolchain at install time? I think we will eventually reach consensus that package managers requiring a compiler is a misfeature. I think I’d be just fine having two base images, one for runtime, and a superset image for Continuous Integration that layers the compiler toolchain on top. But Node has this same behavior and it drives me nuts.
    - viraptor 7y ago
      > I think we will eventually reach consensus that package managers requiring a compiler is a misfeature What's your alternative solution? For simple native libraries there's always libffi, but things that need numerical processing will want to actually plug into the underlying value representation directly to not sacrifice speed.
    - bmurphy1976 7y ago
      >Isn’t this also a problem of requiring the compiler toolchain at install time? This isn't a new problem if you use docker. To address it you have to either use multistage builds or more traditionally you install the dev tools, build and install your artifacts, then remove your dev tools all in a single step. It's annoying, but it's a common pattern.
- dkarp 7y ago
  I Agree, since this is the case for all distros using musl. This was a decision made by python to not support musl in the wheel manylinux standard: https://github.com/pypa/manylinux/blob/46241e9debbaf4b045c888af55ec80311bf27d99/pep-513.rst#platform-detection-for-installers https://github.com/pypa/manylinux/blob/46241e9debbaf4b045c88... If you need more than a few packages installed using pip, then Alpine simply is not a good choice for python.
nwmcsween 7y ago
What exactly is pip erroring out on with the wheel install? if it's looking for glibc /lib/ld-$foo.so you could symlink musl and see if it works and if python touches some glibc specific junk just install gcompat - https://pkgs.alpinelinux.org/package/edge/community/x86/gcompat https://pkgs.alpinelinux.org/package/edge/community/x86/gcom...
- itamarst 7y ago
  On Alpine it doesn't even download the wheel at all, it just downloads the source code.
  - nwmcsween 7y ago
    Could you RUN apk add --no-cache gcompat then see what pip does? If pip is checking os-release for determining the libc then it is utterly broken.
favmutation 7y ago
From my own (limited) experience, a 25-minute build time seems pretty unusual for an Alpine image pulling in Python dependencies. The lack of wheel support on Alpine definitely seems like a compelling argument against using it for Python apps, but that build time indicates to me that some confounding factor might be at play here. Maybe matplotlib and/or pandas are just way bigger than the libraries I've pulled into Alpine images before? Curious to hear what others think.
- pletnes 7y ago
  Pandas and numpy (which all things data science depend on) require compilation of C/C++ code which comes in addition to the download/install steps. Wheel files can contain precompiled .so files.
  - mwarkentin 7y ago
    Some of the libraries which can be very slow to install due to compilation that we've ended up building our own alpine-compatible wheels for: Pillow PyXB cffi cryptography gevent lxml msgpack-python psycopg2 pycrypto uWSGI
ShakataGaNai 7y ago
I do a lot of Python and a lot of Docker. Mostly python in Docker. I've used both Alpine and Ubuntu. There is a fair amount right about this article, and a lot wrong. First "the Dockerfiles in this article are not examples of best practices" Well, that's a big mistake. Of course if you don't follow best practices you won't get the best results. In these examples the author doesn't even follow the basic recommendations from the Docker Alpine image page. Ex, use "apk add --no-cache PACKAGE". When you're caching apt & apk, of course the image is going to be a ton larger. On the flip side he does basically exactly that to clean up ubuntus apt cache. The real article should have been "should you use alpine for every python/docker project?" and the answer is "No". If you're doing something complicated that requires a lot of system libs, like say machine learning or imagine manipulation - don't use Alpine. It's a pain. On the flip side if all you need is a small flask app, Alpine is a great solution. Also, build times and sizes don't matter too much in the grand scheme of things. Unless you're changing the Dockerfile regularly, it won't matter. Why? Because Docker caches each layer of the build. So if all you do is add your app code (which changes, and is added at the end of the Dockerfile) - sure the initial build might be 10 mn, but after that it'll be a few seconds. Docker pull caches just the same, so the initial pull might be large, but then after that it's just the new layers.
- masklinn 7y ago
  > Also, build times and sizes don't matter too much in the grand scheme of things. In which case... why do you bother with alpine in the first place?
  - Rapzid 7y ago
    I don't. Never got on board with it and just stick to Ubuntu so I don't have to think about the differences. Never had a business driver come up for going with Alpine though.
  - ShakataGaNai 7y ago
    > Too much There are cases in which it does matter. Just like anything else, it strongly depends on your use case. If you're into the microservices thing and rebuild your containers, or change requirements frequently - maybe container size (as they'll be pulled a lot, by a lot of different hosts) matter. Maybe you're making something for the public consumption and want to make sure it doesn't take up a huge amount of space. Maybe you're making an image for IOT/RPi type devices. You get the idea. Personally, I like using Alpine where possible because it's got less stuff. Less software means less things that could potentially have a security issue needing fixing/patching/updating later. However my default container for anything else is "miniubuntu" build as it's got all the basics, it's 85mb in size, and I can install all the things I need for the more complicated projects.
dkarp 7y ago
The build issue is because musl wheels don't exist, so installing with pip takes forever. If you just need pandas, then you can install it with apk along with many other packages and it'll be much quicker: https://pkgs.alpinelinux.org/packages?page=1&branch=edge&name=py3-%2A https://pkgs.alpinelinux.org/packages?page=1&branch=edge&nam... Alpine are doing what they can, by maintaining common python packages on apk. It is pip that is tied to glibc
- avian 7y ago
  Unfortunately in many (most?) cases I've seen the prescribed deployment practice for Python software is to use virtualenv (or some variant of it). Python dependencies are to be handled exclusively via pip (requirements.txt, install_requires in setup.py, etc.) and using distro-packaged Python modules is frowned upon because they are considered to be outdated. I'm not saying that's the best practice. Personally I like to use distro-packaged stuff as much as possible because it's less volatile and tends to go through some more review than things on PyPI. But the virtualenv/pip exclusive method does seem to prevail in the field in my experience, so the advice to just use Alpine packages isn't useful in many cases.
  - jontaydev 7y ago
    It is best practice, but you probably don’t need virtualenv if you’re deploying a single application in a docker container.
    - brummm 7y ago
      Messing with the system python without using some form of virtual environment is a recipe for disaster.
      - dkarp 7y ago
        It's a docker box, you're already inside some form of virtual environment
      - kevin_thibedeau 7y ago
axyjo 7y ago
Definitely ran into "Alpine Linux can cause unexpected runtime bugs" -- with strftime, libc and musl don't have 100% equivalence of the date/time formatting strings.
- chungus_khan 7y ago
  C library variations between UNIX systems are the eternal curse of this species apparently.
deathanatos 7y ago
> And the resulting image is 851MB. If you include the dev dependencies and compiler toolchains in the final image. You can (and should) cut all that out of the final image.
- yjftsjthsd-h 7y ago
  The author sort of mentions this: > The image size can be made smaller, for example with multi-stage builds, but that means even more work. ...which I really don't find compelling, honestly. If you want a smaller image, then yes, you need to do one of the major things that produces smaller images. Don't install a whole dev chain, leave it in the image, and then complain that your image is big.
  - jrockway 7y ago
    I think the advice of "ship gcc and its dependencies to production to avoid writing 2 extra lines in your Dockerfile" is pretty awful advice. Less stuff in your containers means faster image pulls and less attack surface. Seems crazy not to do it. FROM clean-image, COPY --from=base-image ... I am guessing with Python the difficulty is in enumerating what files you need to copy, because there's more than one. But I am sure that can be figured out exactly once and you can save yourself some time and money on unnecessary data transfer.
danpalmer 7y ago
Lots of criticism of the dependency issues detailed here, but no one talking about the fact that musl looks to allocate roughly ~50% slower than glibc. For many applications, particularly those that aren't written with low-level performance details in mind, this is likely to cause significant slowdowns (obviously not the full 50% as many things will be CPU bound, but still likely noticeable). More generally, it seems that Alpine optimises for build/deployment, but glibc-based distros are going to be faster at runtime. While a fast build/deploy is nice, it's probably not worth it if it comes at a cost of significant runtime performance issues.
- ShakataGaNai 7y ago
  Like anything else, if you need to worry about speed - you should optimize. There are just as many people who will shit on Python for it's speed/scale capabilities as there are people who will point out "Well, it works for Instagram".
  - chungus_khan 7y ago
    The reason musl is slower is largely because the project's goals are about being lightweight and correct first and foremost, and this can often come into conflict with performance optimization. Python isn't the issue here, but almost any Python program will be heavily affected by this because the Python interpreter does a lot of allocation.
  - danpalmer 7y ago
    There are levels to caring about speed, and this could easily cause issues at a level above what they should, being a fairly low level detail (memory allocations). I work on a Python application where we don't care about memory allocations (we care about the number of database queries for example). Using musl may make memory allocations expensive enough (as they are 50% slower) that we may have to start paying attention to this in some areas of our code.
Deradon 7y ago
I'd structure the Dockerfile in another way with the "python:3.8-alpine" approach. E.g. something like: FROM python:3.8-alpine RUN apk add --no-cache --virtual build-dependencies gcc build-base freetype-dev libpng-dev openblas-dev && \ pip install --no-cache-dir matplotlib pandas && \ apk del build-dependencies This way you won't keep the build dependencies in the layer and the final image. Maybe not all of the packages are build-dependencies, but at least there is no need to keep the gcc and build-base package around. Long story short, wrap your "pip install", "bundle install", "yarn install" around some "apk add .. && apk del" within one RUN. Yes, it would still compile for quite some time, but the final image size is not "851MB" but "489MB". Still larger than the "363MB" of the "python-slim" version. Guess "pip install" will keep some build artifacts around?
- Arnavion 7y ago
  TIL about `--virtual`. This is very useful. >-t, --virtual NAME >Instead of adding all the packages to 'world', create a new virtual package with the listed dependencies and add that to 'world'; the actions of the command are easily reverted by deleting the virtual package
- daze42 7y ago
  Your guess is correct. Pip does keep a cache of packages around. It's easy to fix by setting the `--no-cache-dir` switch when installing: `pip install --no-cache-dir -r requirements.txt`
- je42 7y ago
  alternatively you can create a multi staged build. also you dont need clean up the builder, since the unused files are not used in the final image. also add a new file you can just append it to the builder. (not rebuild required of the previous installs required since you don't care about the size of the builder image ) FROM python:3.8-alpine as builder RUN apk add --virtual build-dependencies gcc build-base freetype-dev libpng-dev openblas-dev RUN pip install matplotlib pandas FROM python:3.8-alpine # a couple of copy commands to copy binaries and py files from builder to the final image. COPY --from=builder /usr/bin /usr/bin COPY --from=builder /usr/local/lib/ /usr/local/lib/
aeyes 7y ago
That happens as well on Debian if there are no wheels for your packages. It happens quite often when Python versions change. I just run a multistage build where I first install all the packages, building wheels were necessary. Then I copy them to a new clean image from the first container. Another benefit is that I don't need to have a compiler installed in my final image. The Dockerfile is much cleaner than having one huge RUN which installs and then uninstalls build dependencies.
- verst 7y ago
  This. The author should use a multi-stage build where one stage is used for building the missing wheels. Those should then be copied over to the final stage. No need to install any build tools in the final Alpine (or Debian) image. The Alpine image will definitely be much smaller.
raverbashing 7y ago
The issues in the article mostly apply to packages with (some) extensions. Matplotlib? Pandas? Those are not typical except for the datascience crowd. For your regular web stack? It works fine. And yes, installing the apk will save time in a lot of occasions.
- dkarp 7y ago
  FWIW, my current project is a pretty typical web stack (Flask/SQLAlchemy) and the build time was bad enough that I switched back to debian. In a web stack, you're likely to have a lot of external imports, so I'd say you're generally better off with debian. If you're doing data science and only importing Pandas then you'll be fine as you can switch to the apk.
  - jabl 7y ago
    Glibc libm has been heavily tuned for both speed and correctness over many years. Not sure I'd trust musl to the same extent in that area. (Assuming your data sciency workload uses those functions; if not, this doesn't apply)
  - raverbashing 7y ago
    That's weird, my guess is that it might be because of a DB driver or some other module that needs C In my use cases the image size download eclipsed the build time for the most part
dmuth 7y ago
Wow--that actually solves a mystery which I have been trying to figure out for a few months now! Awhile back, I was trying to Dockerize the Twint app, and ran into exactly the same problem. I ended up writing a "lite" version of the Dockerfile that would strike references to Pandas in the Python code: https://github.com/dmuth/splunk-twint/blob/master/Dockerfile-lite https://github.com/dmuth/splunk-twint/blob/master/Dockerfile... To this day, I'm amazed it worked. But it took build time from 15 minutes down to 30 seconds. I will have to revisit this, however, because I would prefer to have just one container that builds quickly instead of having to do a workaround like that.
StreamBright 7y ago
This article nicely summarizes the shortcomings of Python (depending on lots of C, C++ libs) and how it is being tide to GCC. If Alpine was a tier 1 platform for Python usage I am pretty sure there would be musl flavoured wheel packages. Comparing apples to oranges and concluding they are different.
LaserJesus 7y ago
What about https://github.com/sgerrand/alpine-pkg-glibc https://github.com/sgerrand/alpine-pkg-glibc? It provides a glibc apk for Alpine. The miniconda Dockerfile makes use of this before continuing to install miniconda. https://github.com/ContinuumIO/docker-images/blob/master/miniconda3/alpine/Dockerfile https://github.com/ContinuumIO/docker-images/blob/master/min.... Would this solve all of the issues raised in the article?
mychael 7y ago
There is so much misinformation in this article. Its hard to tell if its deliberate or not, so I'll assume the author is uninformed.
- yjftsjthsd-h 7y ago
  I tend to agree that the author is at best making some very specific judgement calls (particularly, I think that refusing to do multi-stage builds disqualifies any size-based comments), but your comment isn't very constructive if you're not actually addressing anything that they say.
atak1 7y ago
The article makes you click on another article for the answer of "well, what Python docker build should you use?" If you're not satisfied with Alpine and need slim builds, answer is python3.8:slim-buster (from other article). From empirical experience, slim-buster has been a good choice for our use case of ML services.
rcarmo 7y ago
I disagree with the author on many counts, largely because I maintain two stable, multi-platform Python images: - https://github.com/insightfulsystems/alpine-python https://github.com/insightfulsystems/alpine-python - https://github.com/rcarmo/ubuntu-python https://github.com/rcarmo/ubuntu-python The first uses stock Alpine packages, and the second builds Python from scratch (with some optimizations) atop Ubuntu LTS, and they serve two different use cases, but maintaining both made me learn a few things and there are a few factual errors in the article. For starters, yes, you can run manylinux wheels on Alpine. Here’s how to do it: https://github.com/insightfulsystems/alpine-python/blob/master/3.6/Dockerfile#L28 https://github.com/insightfulsystems/alpine-python/blob/mast... So no, you don’t need to recompile every single package for an Alpine Python runtime. As to runtime differences, yes, they exist, but are _extremely_ dependent on your use cases. I have not encountered any of the bugs - I did have a few crashes with GPU libraries and Tensorflow (which is why I also maintain the Ubuntu-based version), but the author points out third-party accounts, one of which seems to be attributable to locale settings (which you should always set anyway). Performance differences are negligible on Intel (at least for web apps using Sanic and asyncio - I don’t do much Django these days), and the inclusion of a link to buy a “production-ready template” mid-article is just... iffy. Give us data and working code - I’d like to see an objective benchmark, for instance, and might even set up one with my own images.
- dkarp 7y ago
  Thanks for sharing your manylinux wheels setup. Will the python packages just link the musl libc instead of glibc? If that's all that is needed then it makes me wonder why pip is only downloading wheels on glibc distros currently.
  - mwarkentin 7y ago
    I confirmed I was able to install lxml from a wheel and parse a web page with it.
  - rcarmo 7y ago
    In my experience the only things that broke were the CUDA/NVIDIA bindings that are “several turtles down” from Keras and the like, but, again, these images serve different purposes and I use the Alpine ones mostly for web services and vanilla sci-kit stuff.
yellowapple 7y ago
> Most or perhaps all of these problems have already been fixed, but no doubt there are more problems to discover. Random breakage of this sort is just one more thing to worry about. It's 2020, and yet we're still falling victim to textbook examples of FUD.
indygreg2 7y ago
The criticisms about musl's memory allocator being slower than glibc may be true. However, this specific issue can be resolved by telling the Python interpreter to use e.g. jemalloc as its allocator instead of the system allocator. Unfortunately, this requires building a custom binary linking jemalloc and calling into Python C APIs to configure the embedded interpreter and that's probably prohibitively too much effort for many users, who view the Python interpreter as a pre-built black box and therefore see musl's allocator as an unavoidable constant. Fortunately, tools like PyOxidizer exist to make this easier. (PyOxidizer supports using jemalloc as the allocator with a 1 line change and jemalloc can deliver even more performance than glib'c allocator.)
- shawn-butler 7y ago
  raw_allocator (string) Which memory allocator to use for the PYMEM_DOMAIN_RAW allocator. Values can be jemalloc, rust, or system. Important: the rust crate is not recommended because it introduces performance overhead. From the docucmentation
ameixaseca 7y ago
This post is actually reversing cause and effect: Python wheels are slower on anything that is not glibc or x86(-64). There's nothing wrong from Alpine side or anything that is not x86(-64) but with the mechanism used by Python. This is a problem I have with Python: everything is assumed to be a standard GNU/Linux/desktop environment, you are assumed to always have to have the latest versions of everything (or to have "up to" a certain version, which then breaks everything else that relies on the latest), and all becomes sub-optimal if you deviate a bit from the expectations. Now this is fine for some people but you can't blame the ones that have different environments. This is why most people doing actual embedded cringe when someone suggests using Python, because the experience is not usually pleasant.
ObscureScience 7y ago
The fact glibc is the "standard" does make musl users required to work a bit harder. I would say the proper solution regarding Alpine would be to simply package more python libraries in the official repos and install them from there (pandas is in edge at the moment). Another solution could be to build in one docker image and use that as a pip repo for the development one.
Havoc 7y ago
I understand the need for it, but at least at my amateur level I dislike it. I can afford a decent VPS - I don't care about the extra 100MB. I do care if the CLI command I want doesn't work because it got stripped out for the sake of efficiency. Now I'm spending time very inefficiently adding back the stuff that is there by default on ubuntu? If I wanted maximum efficiency I'd be coding in ASM not deploying dockers.
seglberg 7y ago
Not sure I've seen this mentioned anywhere before, but what we tend to do is build all of our dependencies in CI by setting `_PYTHON_HOST_PLATFORM=alpine-x86_64` and uploading them to an internal index. This way when we go to build our final app images, there is no need to compile any python packages or install a toolchain. Our builds are generally under a minute or two as its just downloading and installing packages via Poetry/Pip.
xvilka 7y ago
About the runtime errors - I confirm, experienced periodic crashes on the same data that works fine in Ubuntu-based docker. I use OCaml though, but the point is still valid.
burnJS 7y ago
We're using an Alpine based image while prototyping an application at work right now. I really dislike the APK, it is so limited compared to what comes with APT. I guess its a little smaller than the limited Ubuntu image available, but APK is just not any fun to work with. Just my two cents.
wicket 7y ago
A couple more nitpicks for the author if he/she is reading this... > Alpine doesn’t support wheels You mean wheels do not support musl, or to be more precise they don't conform to POSIX 2008 or C11 standards. > Alpine has a smaller default stack size for threads, which can lead to Python crashes. You're linking to a Python bug. That's right, Python. This is a bug in Python. Not Alpine, nor musl. The whole point of using Alpine-based Docker images is that musl provides a lean, efficient and standards-compliant implementation of libc which should in turn result in faster runtime execution of the software running in your container, and other secondary benefits such as smaller binaries and faster build times. If your software (in this case, Python) is glibc-dependent and is not standards-compliant, the bug is in your software, not musl. Don't expect it to work. The bottom line is that if you don't understand this, you shouldn't be using Alpine. Please don't trash-talk something you don't understand.
balloob 7y ago
Don't get scared by this article and not use Alpine. It's perfectly fine to run Python and it's what we do with Home Assistant too. Home Assistant maintains Alpine wheels for a ton of packages for their 5 supported platforms: https://wheels.home-assistant.io/ https://wheels.home-assistant.io/ Repository of our wheels builder can be found at https://github.com/home-assistant/hassio-wheels https://github.com/home-assistant/hassio-wheels
hbogert 7y ago
Beside image size and build time, runtime performance can be really bad with alpine's libc implementation (Musl). My colleague pinpointed that for example PowerDNS has a throughput of less than 50% compared to a PowerDNS running with glibc. We should really make a blog.

306 points by itamarst 7y ago

Not really Alpine's problem that there aren't upstream wheels for your dependencies. I think you're being realistic with your suggestion but you're barking at the wrong tree.

hprotagonist 7y ago
musl seems very niche. That’s an ecosystem problem; that is on Alpine for swimming upstream.
wzdd 7y ago
Multi-stage container builds are also not exactly unusual. You would generally not want an entire C build system sitting around in your container.
- hinkley 7y ago
  Isn’t this also a problem of requiring the compiler toolchain at install time? I think we will eventually reach consensus that package managers requiring a compiler is a misfeature. I think I’d be just fine having two base images, one for runtime, and a superset image for Continuous Integration that layers the compiler toolchain on top. But Node has this same behavior and it drives me nuts.
  - viraptor 7y ago
    > I think we will eventually reach consensus that package managers requiring a compiler is a misfeature What's your alternative solution? For simple native libraries there's always libffi, but things that need numerical processing will want to actually plug into the underlying value representation directly to not sacrifice speed.
  - bmurphy1976 7y ago
    >Isn’t this also a problem of requiring the compiler toolchain at install time? This isn't a new problem if you use docker. To address it you have to either use multistage builds or more traditionally you install the dev tools, build and install your artifacts, then remove your dev tools all in a single step. It's annoying, but it's a common pattern.
dkarp 7y ago
I Agree, since this is the case for all distros using musl. This was a decision made by python to not support musl in the wheel manylinux standard: https://github.com/pypa/manylinux/blob/46241e9debbaf4b045c888af55ec80311bf27d99/pep-513.rst#platform-detection-for-installers https://github.com/pypa/manylinux/blob/46241e9debbaf4b045c88... If you need more than a few packages installed using pip, then Alpine simply is not a good choice for python.

nwmcsween 7y ago

What exactly is pip erroring out on with the wheel install? if it's looking for glibc /lib/ld-$foo.so you could symlink musl and see if it works and if python touches some glibc specific junk just install gcompat - https://pkgs.alpinelinux.org/package/edge/community/x86/gcompat https://pkgs.alpinelinux.org/package/edge/community/x86/gcom...

itamarst 7y ago
On Alpine it doesn't even download the wheel at all, it just downloads the source code.
- nwmcsween 7y ago
  Could you RUN apk add --no-cache gcompat then see what pip does? If pip is checking os-release for determining the libc then it is utterly broken.

favmutation 7y ago

From my own (limited) experience, a 25-minute build time seems pretty unusual for an Alpine image pulling in Python dependencies. The lack of wheel support on Alpine definitely seems like a compelling argument against using it for Python apps, but that build time indicates to me that some confounding factor might be at play here. Maybe matplotlib and/or pandas are just way bigger than the libraries I've pulled into Alpine images before? Curious to hear what others think.

pletnes 7y ago
Pandas and numpy (which all things data science depend on) require compilation of C/C++ code which comes in addition to the download/install steps. Wheel files can contain precompiled .so files.
- mwarkentin 7y ago
  Some of the libraries which can be very slow to install due to compilation that we've ended up building our own alpine-compatible wheels for: Pillow PyXB cffi cryptography gevent lxml msgpack-python psycopg2 pycrypto uWSGI

ShakataGaNai 7y ago

I do a lot of Python and a lot of Docker. Mostly python in Docker. I've used both Alpine and Ubuntu. There is a fair amount right about this article, and a lot wrong. First "the Dockerfiles in this article are not examples of best practices" Well, that's a big mistake. Of course if you don't follow best practices you won't get the best results. In these examples the author doesn't even follow the basic recommendations from the Docker Alpine image page. Ex, use "apk add --no-cache PACKAGE". When you're caching apt & apk, of course the image is going to be a ton larger. On the flip side he does basically exactly that to clean up ubuntus apt cache. The real article should have been "should you use alpine for every python/docker project?" and the answer is "No". If you're doing something complicated that requires a lot of system libs, like say machine learning or imagine manipulation - don't use Alpine. It's a pain. On the flip side if all you need is a small flask app, Alpine is a great solution. Also, build times and sizes don't matter too much in the grand scheme of things. Unless you're changing the Dockerfile regularly, it won't matter. Why? Because Docker caches each layer of the build. So if all you do is add your app code (which changes, and is added at the end of the Dockerfile) - sure the initial build might be 10 mn, but after that it'll be a few seconds. Docker pull caches just the same, so the initial pull might be large, but then after that it's just the new layers.

masklinn 7y ago
> Also, build times and sizes don't matter too much in the grand scheme of things. In which case... why do you bother with alpine in the first place?
- Rapzid 7y ago
  I don't. Never got on board with it and just stick to Ubuntu so I don't have to think about the differences. Never had a business driver come up for going with Alpine though.
- ShakataGaNai 7y ago
  > Too much There are cases in which it does matter. Just like anything else, it strongly depends on your use case. If you're into the microservices thing and rebuild your containers, or change requirements frequently - maybe container size (as they'll be pulled a lot, by a lot of different hosts) matter. Maybe you're making something for the public consumption and want to make sure it doesn't take up a huge amount of space. Maybe you're making an image for IOT/RPi type devices. You get the idea. Personally, I like using Alpine where possible because it's got less stuff. Less software means less things that could potentially have a security issue needing fixing/patching/updating later. However my default container for anything else is "miniubuntu" build as it's got all the basics, it's 85mb in size, and I can install all the things I need for the more complicated projects.

dkarp 7y ago

The build issue is because musl wheels don't exist, so installing with pip takes forever. If you just need pandas, then you can install it with apk along with many other packages and it'll be much quicker: https://pkgs.alpinelinux.org/packages?page=1&branch=edge&name=py3-%2A https://pkgs.alpinelinux.org/packages?page=1&branch=edge&nam... Alpine are doing what they can, by maintaining common python packages on apk. It is pip that is tied to glibc

avian 7y ago
Unfortunately in many (most?) cases I've seen the prescribed deployment practice for Python software is to use virtualenv (or some variant of it). Python dependencies are to be handled exclusively via pip (requirements.txt, install_requires in setup.py, etc.) and using distro-packaged Python modules is frowned upon because they are considered to be outdated. I'm not saying that's the best practice. Personally I like to use distro-packaged stuff as much as possible because it's less volatile and tends to go through some more review than things on PyPI. But the virtualenv/pip exclusive method does seem to prevail in the field in my experience, so the advice to just use Alpine packages isn't useful in many cases.
- jontaydev 7y ago
  It is best practice, but you probably don’t need virtualenv if you’re deploying a single application in a docker container.
  - brummm 7y ago
    Messing with the system python without using some form of virtual environment is a recipe for disaster.
    - dkarp 7y ago
      It's a docker box, you're already inside some form of virtual environment
    - kevin_thibedeau 7y ago

axyjo 7y ago

Definitely ran into "Alpine Linux can cause unexpected runtime bugs" -- with strftime, libc and musl don't have 100% equivalence of the date/time formatting strings.

chungus_khan 7y ago
C library variations between UNIX systems are the eternal curse of this species apparently.

deathanatos 7y ago

> And the resulting image is 851MB. If you include the dev dependencies and compiler toolchains in the final image. You can (and should) cut all that out of the final image.

yjftsjthsd-h 7y ago
The author sort of mentions this: > The image size can be made smaller, for example with multi-stage builds, but that means even more work. ...which I really don't find compelling, honestly. If you want a smaller image, then yes, you need to do one of the major things that produces smaller images. Don't install a whole dev chain, leave it in the image, and then complain that your image is big.
- jrockway 7y ago
  I think the advice of "ship gcc and its dependencies to production to avoid writing 2 extra lines in your Dockerfile" is pretty awful advice. Less stuff in your containers means faster image pulls and less attack surface. Seems crazy not to do it. FROM clean-image, COPY --from=base-image ... I am guessing with Python the difficulty is in enumerating what files you need to copy, because there's more than one. But I am sure that can be figured out exactly once and you can save yourself some time and money on unnecessary data transfer.

danpalmer 7y ago

Lots of criticism of the dependency issues detailed here, but no one talking about the fact that musl looks to allocate roughly ~50% slower than glibc. For many applications, particularly those that aren't written with low-level performance details in mind, this is likely to cause significant slowdowns (obviously not the full 50% as many things will be CPU bound, but still likely noticeable). More generally, it seems that Alpine optimises for build/deployment, but glibc-based distros are going to be faster at runtime. While a fast build/deploy is nice, it's probably not worth it if it comes at a cost of significant runtime performance issues.

ShakataGaNai 7y ago
Like anything else, if you need to worry about speed - you should optimize. There are just as many people who will shit on Python for it's speed/scale capabilities as there are people who will point out "Well, it works for Instagram".
- chungus_khan 7y ago
  The reason musl is slower is largely because the project's goals are about being lightweight and correct first and foremost, and this can often come into conflict with performance optimization. Python isn't the issue here, but almost any Python program will be heavily affected by this because the Python interpreter does a lot of allocation.
- danpalmer 7y ago
  There are levels to caring about speed, and this could easily cause issues at a level above what they should, being a fairly low level detail (memory allocations). I work on a Python application where we don't care about memory allocations (we care about the number of database queries for example). Using musl may make memory allocations expensive enough (as they are 50% slower) that we may have to start paying attention to this in some areas of our code.

Deradon 7y ago

I'd structure the Dockerfile in another way with the "python:3.8-alpine" approach. E.g. something like: FROM python:3.8-alpine RUN apk add --no-cache --virtual build-dependencies gcc build-base freetype-dev libpng-dev openblas-dev && \ pip install --no-cache-dir matplotlib pandas && \ apk del build-dependencies This way you won't keep the build dependencies in the layer and the final image. Maybe not all of the packages are build-dependencies, but at least there is no need to keep the gcc and build-base package around. Long story short, wrap your "pip install", "bundle install", "yarn install" around some "apk add .. && apk del" within one RUN. Yes, it would still compile for quite some time, but the final image size is not "851MB" but "489MB". Still larger than the "363MB" of the "python-slim" version. Guess "pip install" will keep some build artifacts around?

Arnavion 7y ago
TIL about `--virtual`. This is very useful. >-t, --virtual NAME >Instead of adding all the packages to 'world', create a new virtual package with the listed dependencies and add that to 'world'; the actions of the command are easily reverted by deleting the virtual package
daze42 7y ago
Your guess is correct. Pip does keep a cache of packages around. It's easy to fix by setting the `--no-cache-dir` switch when installing: `pip install --no-cache-dir -r requirements.txt`
je42 7y ago
alternatively you can create a multi staged build. also you dont need clean up the builder, since the unused files are not used in the final image. also add a new file you can just append it to the builder. (not rebuild required of the previous installs required since you don't care about the size of the builder image ) FROM python:3.8-alpine as builder RUN apk add --virtual build-dependencies gcc build-base freetype-dev libpng-dev openblas-dev RUN pip install matplotlib pandas FROM python:3.8-alpine # a couple of copy commands to copy binaries and py files from builder to the final image. COPY --from=builder /usr/bin /usr/bin COPY --from=builder /usr/local/lib/ /usr/local/lib/

aeyes 7y ago

That happens as well on Debian if there are no wheels for your packages. It happens quite often when Python versions change. I just run a multistage build where I first install all the packages, building wheels were necessary. Then I copy them to a new clean image from the first container. Another benefit is that I don't need to have a compiler installed in my final image. The Dockerfile is much cleaner than having one huge RUN which installs and then uninstalls build dependencies.

verst 7y ago
This. The author should use a multi-stage build where one stage is used for building the missing wheels. Those should then be copied over to the final stage. No need to install any build tools in the final Alpine (or Debian) image. The Alpine image will definitely be much smaller.

raverbashing 7y ago

The issues in the article mostly apply to packages with (some) extensions. Matplotlib? Pandas? Those are not typical except for the datascience crowd. For your regular web stack? It works fine. And yes, installing the apk will save time in a lot of occasions.

dkarp 7y ago
FWIW, my current project is a pretty typical web stack (Flask/SQLAlchemy) and the build time was bad enough that I switched back to debian. In a web stack, you're likely to have a lot of external imports, so I'd say you're generally better off with debian. If you're doing data science and only importing Pandas then you'll be fine as you can switch to the apk.
- jabl 7y ago
  Glibc libm has been heavily tuned for both speed and correctness over many years. Not sure I'd trust musl to the same extent in that area. (Assuming your data sciency workload uses those functions; if not, this doesn't apply)
- raverbashing 7y ago
  That's weird, my guess is that it might be because of a DB driver or some other module that needs C In my use cases the image size download eclipsed the build time for the most part

dmuth 7y ago

Wow--that actually solves a mystery which I have been trying to figure out for a few months now! Awhile back, I was trying to Dockerize the Twint app, and ran into exactly the same problem. I ended up writing a "lite" version of the Dockerfile that would strike references to Pandas in the Python code: https://github.com/dmuth/splunk-twint/blob/master/Dockerfile-lite https://github.com/dmuth/splunk-twint/blob/master/Dockerfile... To this day, I'm amazed it worked. But it took build time from 15 minutes down to 30 seconds. I will have to revisit this, however, because I would prefer to have just one container that builds quickly instead of having to do a workaround like that.

StreamBright 7y ago

This article nicely summarizes the shortcomings of Python (depending on lots of C, C++ libs) and how it is being tide to GCC. If Alpine was a tier 1 platform for Python usage I am pretty sure there would be musl flavoured wheel packages. Comparing apples to oranges and concluding they are different.

LaserJesus 7y ago

What about https://github.com/sgerrand/alpine-pkg-glibc https://github.com/sgerrand/alpine-pkg-glibc? It provides a glibc apk for Alpine. The miniconda Dockerfile makes use of this before continuing to install miniconda. https://github.com/ContinuumIO/docker-images/blob/master/miniconda3/alpine/Dockerfile https://github.com/ContinuumIO/docker-images/blob/master/min.... Would this solve all of the issues raised in the article?

mychael 7y ago

There is so much misinformation in this article. Its hard to tell if its deliberate or not, so I'll assume the author is uninformed.

yjftsjthsd-h 7y ago
I tend to agree that the author is at best making some very specific judgement calls (particularly, I think that refusing to do multi-stage builds disqualifies any size-based comments), but your comment isn't very constructive if you're not actually addressing anything that they say.

atak1 7y ago

The article makes you click on another article for the answer of "well, what Python docker build should you use?" If you're not satisfied with Alpine and need slim builds, answer is python3.8:slim-buster (from other article). From empirical experience, slim-buster has been a good choice for our use case of ML services.

rcarmo 7y ago

I disagree with the author on many counts, largely because I maintain two stable, multi-platform Python images: - https://github.com/insightfulsystems/alpine-python https://github.com/insightfulsystems/alpine-python - https://github.com/rcarmo/ubuntu-python https://github.com/rcarmo/ubuntu-python The first uses stock Alpine packages, and the second builds Python from scratch (with some optimizations) atop Ubuntu LTS, and they serve two different use cases, but maintaining both made me learn a few things and there are a few factual errors in the article. For starters, yes, you can run manylinux wheels on Alpine. Here’s how to do it: https://github.com/insightfulsystems/alpine-python/blob/master/3.6/Dockerfile#L28 https://github.com/insightfulsystems/alpine-python/blob/mast... So no, you don’t need to recompile every single package for an Alpine Python runtime. As to runtime differences, yes, they exist, but are _extremely_ dependent on your use cases. I have not encountered any of the bugs - I did have a few crashes with GPU libraries and Tensorflow (which is why I also maintain the Ubuntu-based version), but the author points out third-party accounts, one of which seems to be attributable to locale settings (which you should always set anyway). Performance differences are negligible on Intel (at least for web apps using Sanic and asyncio - I don’t do much Django these days), and the inclusion of a link to buy a “production-ready template” mid-article is just... iffy. Give us data and working code - I’d like to see an objective benchmark, for instance, and might even set up one with my own images.

dkarp 7y ago
Thanks for sharing your manylinux wheels setup. Will the python packages just link the musl libc instead of glibc? If that's all that is needed then it makes me wonder why pip is only downloading wheels on glibc distros currently.
- mwarkentin 7y ago
  I confirmed I was able to install lxml from a wheel and parse a web page with it.
- rcarmo 7y ago
  In my experience the only things that broke were the CUDA/NVIDIA bindings that are “several turtles down” from Keras and the like, but, again, these images serve different purposes and I use the Alpine ones mostly for web services and vanilla sci-kit stuff.

yellowapple 7y ago

> Most or perhaps all of these problems have already been fixed, but no doubt there are more problems to discover. Random breakage of this sort is just one more thing to worry about. It's 2020, and yet we're still falling victim to textbook examples of FUD.

indygreg2 7y ago

The criticisms about musl's memory allocator being slower than glibc may be true. However, this specific issue can be resolved by telling the Python interpreter to use e.g. jemalloc as its allocator instead of the system allocator. Unfortunately, this requires building a custom binary linking jemalloc and calling into Python C APIs to configure the embedded interpreter and that's probably prohibitively too much effort for many users, who view the Python interpreter as a pre-built black box and therefore see musl's allocator as an unavoidable constant. Fortunately, tools like PyOxidizer exist to make this easier. (PyOxidizer supports using jemalloc as the allocator with a 1 line change and jemalloc can deliver even more performance than glib'c allocator.)

shawn-butler 7y ago
raw_allocator (string) Which memory allocator to use for the PYMEM_DOMAIN_RAW allocator. Values can be jemalloc, rust, or system. Important: the rust crate is not recommended because it introduces performance overhead. From the docucmentation

ameixaseca 7y ago

This post is actually reversing cause and effect: Python wheels are slower on anything that is not glibc or x86(-64). There's nothing wrong from Alpine side or anything that is not x86(-64) but with the mechanism used by Python. This is a problem I have with Python: everything is assumed to be a standard GNU/Linux/desktop environment, you are assumed to always have to have the latest versions of everything (or to have "up to" a certain version, which then breaks everything else that relies on the latest), and all becomes sub-optimal if you deviate a bit from the expectations. Now this is fine for some people but you can't blame the ones that have different environments. This is why most people doing actual embedded cringe when someone suggests using Python, because the experience is not usually pleasant.

ObscureScience 7y ago

The fact glibc is the "standard" does make musl users required to work a bit harder. I would say the proper solution regarding Alpine would be to simply package more python libraries in the official repos and install them from there (pandas is in edge at the moment). Another solution could be to build in one docker image and use that as a pip repo for the development one.

Havoc 7y ago

I understand the need for it, but at least at my amateur level I dislike it. I can afford a decent VPS - I don't care about the extra 100MB. I do care if the CLI command I want doesn't work because it got stripped out for the sake of efficiency. Now I'm spending time very inefficiently adding back the stuff that is there by default on ubuntu? If I wanted maximum efficiency I'd be coding in ASM not deploying dockers.

seglberg 7y ago

Not sure I've seen this mentioned anywhere before, but what we tend to do is build all of our dependencies in CI by setting `_PYTHON_HOST_PLATFORM=alpine-x86_64` and uploading them to an internal index. This way when we go to build our final app images, there is no need to compile any python packages or install a toolchain. Our builds are generally under a minute or two as its just downloading and installing packages via Poetry/Pip.

xvilka 7y ago

About the runtime errors - I confirm, experienced periodic crashes on the same data that works fine in Ubuntu-based docker. I use OCaml though, but the point is still valid.

burnJS 7y ago

We're using an Alpine based image while prototyping an application at work right now. I really dislike the APK, it is so limited compared to what comes with APT. I guess its a little smaller than the limited Ubuntu image available, but APK is just not any fun to work with. Just my two cents.

wicket 7y ago

A couple more nitpicks for the author if he/she is reading this... > Alpine doesn’t support wheels You mean wheels do not support musl, or to be more precise they don't conform to POSIX 2008 or C11 standards. > Alpine has a smaller default stack size for threads, which can lead to Python crashes. You're linking to a Python bug. That's right, Python. This is a bug in Python. Not Alpine, nor musl. The whole point of using Alpine-based Docker images is that musl provides a lean, efficient and standards-compliant implementation of libc which should in turn result in faster runtime execution of the software running in your container, and other secondary benefits such as smaller binaries and faster build times. If your software (in this case, Python) is glibc-dependent and is not standards-compliant, the bug is in your software, not musl. Don't expect it to work. The bottom line is that if you don't understand this, you shouldn't be using Alpine. Please don't trash-talk something you don't understand.

balloob 7y ago

Don't get scared by this article and not use Alpine. It's perfectly fine to run Python and it's what we do with Home Assistant too. Home Assistant maintains Alpine wheels for a ton of packages for their 5 supported platforms: https://wheels.home-assistant.io/ https://wheels.home-assistant.io/ Repository of our wheels builder can be found at https://github.com/home-assistant/hassio-wheels https://github.com/home-assistant/hassio-wheels

hbogert 7y ago

Beside image size and build time, runtime performance can be really bad with alpine's libc implementation (Musl). My colleague pinpointed that for example PowerDNS has a throughput of less than 50% compared to a PowerDNS running with glibc. We should really make a blog.