6 ms·

The build issue is because musl wheels don't exist, so installing with pip takes forever. If you just need pandas, then you can install it with apk along with

by dkarp 7y ago

The build issue is because musl wheels don't exist, so installing with pip takes forever. If you just need pandas, then you can install it with apk along with many other packages and it'll be much quicker: https://pkgs.alpinelinux.org/packages?page=1&branch=edge&name=py3-%2A https://pkgs.alpinelinux.org/packages?page=1&branch=edge&nam... Alpine are doing what they can, by maintaining common python packages on apk. It is pip that is tied to glibc

avian 7y ago
Unfortunately in many (most?) cases I've seen the prescribed deployment practice for Python software is to use virtualenv (or some variant of it). Python dependencies are to be handled exclusively via pip (requirements.txt, install_requires in setup.py, etc.) and using distro-packaged Python modules is frowned upon because they are considered to be outdated. I'm not saying that's the best practice. Personally I like to use distro-packaged stuff as much as possible because it's less volatile and tends to go through some more review than things on PyPI. But the virtualenv/pip exclusive method does seem to prevail in the field in my experience, so the advice to just use Alpine packages isn't useful in many cases.
- jontaydev 7y ago
  It is best practice, but you probably don’t need virtualenv if you’re deploying a single application in a docker container.
  - brummm 7y ago
    Messing with the system python without using some form of virtual environment is a recipe for disaster.
    - dkarp 7y ago
      It's a docker box, you're already inside some form of virtual environment
    - kevin_thibedeau 7y ago
      Debian distros have a separate dist-packages directory for system libs. You can do whatever you want in site-packages. It is effectively like a default virtualenv when operating in a container.
    - penagwin 7y ago
      In the docker container you ARE the system python version. In fact many docker best practices would say that preferably only 1 application is running, and that would be your application.
      - viraptor 7y ago
        It's still not safe. For example there was a conflict one day between awscli, urllib and some system tool (apt?) which led to a broken (apt fails) system if you just "pip install awscli" globally. Even in docker image, you still need virtualenv to be safe (or use only global distro packages).
        fjp 7y ago
        Could you elaborate on why? I worked at a co that almost exclusively ran Alpine containers with a single Python app. No use of virtualenv. Never experienced any hangups with this.
        viraptor 7y ago
        Looked up the issue and it was cloud-init, not apt that failed. Here are the details: https://github.com/aws/aws-cli/issues/3678 https://github.com/aws/aws-cli/issues/3678 In one sentence: "Installing the latest aws-cli on an image is preventing new AMIs based on that image from booting due to the above issue with urllib3." While this specific issue wouldn't affect docker images since you normally don't run cloud-init on them, it's just luck that it wasn't some other utility affected instead. Next time it can affect docker images too.
        kortilla 7y ago
        Why do you have cloud-init in a container?
        markbnj 7y ago
        Same here, and I'm curious to know why some think virtualenv is required in a container, as opposed to just starting with am image containing a clean python install and adding what you need.
    - pid_0 7y ago
      Not in docker. The entire point of docker is isolation so when you use something like python3.7-slim, its fine to use the system python. Thats the whole point of using the image. Now you still want to run and install pip packages as a non-root user of course, but you don't need a virtualenv in docker.
- dkarp 7y ago
  Absolutely agree. Which is why I don't use Alpine linux for python projects and wouldn't recommend anyone else does. In the article, they are only using pandas and not using a requirements.txt file. So it's a pretty perfect case to just use apk. In cases where you don't need many packages, you can still use Alpine if you would like without most of the downsides mentioned in the article.
  - swiley 7y ago
    I write a decent amount of python on alpine, in general I stick to the standard library and occasionally something like pyserial, sqlite, or numpy for specific things. IMO this attitude of pulling in dozens of packages with pip is pretty dangerous, I don’t care which language you’re using.
    - kryptn 7y ago
      What's the alternative?
      - swiley 7y ago
        Removing features, moving them into micro services, and sometimes writing a class or two yourself.
        kryptn 7y ago
        That just sounds like distributing the dependencies. You'll still install them all, just not in the same spot.
masklinn 7y ago
> If you just need pandas, then you can install it with apk along with many other packages and it'll be much quicker: https://pkgs.alpinelinux.org/packages?page=1&branch=edge&nam.. https://pkgs.alpinelinux.org/packages?page=1&branch=edge&nam.... And so again need to rely on something "non-standard". > Alpine are doing what they can, by maintaining common python packages on apk. It is pip that is tied to glibc The "manylinux" tag means glibc by necessity, because nobody has driven a PEP for tagging, detecting and managing musl-based wheels to completion[0]. The issue is not restricted to the choice of libc, or even linux alone, either. But again, that requires people actually put in the work to chip at and fix the issue[1]. [0] https://github.com/pypa/manylinux/issues/37 https://github.com/pypa/manylinux/issues/37 [1] https://github.com/pypa/packaging-problems/issues/69 https://github.com/pypa/packaging-problems/issues/69
- dkarp 7y ago
  Not everyone uses pip. Many data scientists use conda. Using you distribution's package manager is a legitimate solution. The point is though, that this is a python packaging issue. Not an Alpine issue and not an issue that Alpine can do much about. As you linked, pypa have a big task to make packages work on every possible platform. Unfortunately, this comes as an unexpected surprise for anyone who wants to try out Alpine - maybe pip should show a warning when wheels aren't available because of musl.
  - alexhutcheson 7y ago
    https://xkcd.com/1987/ https://xkcd.com/1987/
hackernudes 7y ago
For those like me who have no idea what "wheels" is in this context, I think it is referring to https://github.com/pypa/wheel https://github.com/pypa/wheel This looks like some sort of binary Python packaging thingy. I think @dkarp is saying that there exists pre-existing binary builds for glibc for many things, but not for musl libc so they are forced to be built explicitly when using pip to manage Python dependencies. I could not figure out where this repository of binary packages is or who maintains it. Any corrections would be appreciated.
- korijn 7y ago
  Pre-built binary distributions ("wheels") and source distributions (which sometimes need a compilation step) just live side by side on pypi.org, the public python package index.
- mumblemumble 7y ago
  You've got it. Wheels in Python are like gems in ruby. (Or eggs in [python minus a few years].) The official Python package index has binary images, but they're compiled against glibc, and therefore don't work with Alpine. So when you `pip install` a package with binary dependencies on Alpine, it needs to compile them. Which can lead to a bit of a trip down a rabbit hole. As others have mentioned, Alpine's repository does include Alpine-compatible binary versions of many popular Python packages. So you can often save yourself a lot of trouble by using apk instead of pip to get those.
  - oefrha 7y ago
    > The official Python package index has binary images, but they're compiled against glibc, and therefore don't work with Alpine. That’s a false statement (depending on interpretation). Whether a PyPI package has wheels is entirely at the discretion of the package maintainer, who is responsible for compiling all source and binary distributions; PyPI does not compile anything and only accepts uploads, verbatim. If the maintainer doesn’t upload wheels or doesn’t upload wheels for your platform, then no wheels for you (not from PyPI). In general compiling statically linked wheels (when you have C extensions and external dependencies) is a fairly involved process. At the moment there are six platform tags for Linux wheels supported by PyPI: manylinux{1,2010,2014}_{x86_64,i686}. Each is based on a CentOS release with very old glibc. See https://github.com/pypa/manylinux https://github.com/pypa/manylinux PEP 513 - maynylinux1 https://www.python.org/dev/peps/pep-0513/ https://www.python.org/dev/peps/pep-0513/ PEP 571 - manylinux2010 https://www.python.org/dev/peps/pep-0571/ https://www.python.org/dev/peps/pep-0571/ PEP 599 - manylinux2014 https://www.python.org/dev/peps/pep-0599/ https://www.python.org/dev/peps/pep-0599/

by dkarp 7y ago

avian 7y ago

Unfortunately in many (most?) cases I've seen the prescribed deployment practice for Python software is to use virtualenv (or some variant of it). Python dependencies are to be handled exclusively via pip (requirements.txt, install_requires in setup.py, etc.) and using distro-packaged Python modules is frowned upon because they are considered to be outdated. I'm not saying that's the best practice. Personally I like to use distro-packaged stuff as much as possible because it's less volatile and tends to go through some more review than things on PyPI. But the virtualenv/pip exclusive method does seem to prevail in the field in my experience, so the advice to just use Alpine packages isn't useful in many cases.

jontaydev 7y ago
It is best practice, but you probably don’t need virtualenv if you’re deploying a single application in a docker container.
- brummm 7y ago
  Messing with the system python without using some form of virtual environment is a recipe for disaster.
  - dkarp 7y ago
    It's a docker box, you're already inside some form of virtual environment
  - kevin_thibedeau 7y ago
    Debian distros have a separate dist-packages directory for system libs. You can do whatever you want in site-packages. It is effectively like a default virtualenv when operating in a container.
  - penagwin 7y ago
    In the docker container you ARE the system python version. In fact many docker best practices would say that preferably only 1 application is running, and that would be your application.
    - viraptor 7y ago
      It's still not safe. For example there was a conflict one day between awscli, urllib and some system tool (apt?) which led to a broken (apt fails) system if you just "pip install awscli" globally. Even in docker image, you still need virtualenv to be safe (or use only global distro packages).
      - fjp 7y ago
        Could you elaborate on why? I worked at a co that almost exclusively ran Alpine containers with a single Python app. No use of virtualenv. Never experienced any hangups with this.
        viraptor 7y ago
        Looked up the issue and it was cloud-init, not apt that failed. Here are the details: https://github.com/aws/aws-cli/issues/3678 https://github.com/aws/aws-cli/issues/3678 In one sentence: "Installing the latest aws-cli on an image is preventing new AMIs based on that image from booting due to the above issue with urllib3." While this specific issue wouldn't affect docker images since you normally don't run cloud-init on them, it's just luck that it wasn't some other utility affected instead. Next time it can affect docker images too.
        kortilla 7y ago
        Why do you have cloud-init in a container?
        markbnj 7y ago
        Same here, and I'm curious to know why some think virtualenv is required in a container, as opposed to just starting with am image containing a clean python install and adding what you need.
  - pid_0 7y ago
    Not in docker. The entire point of docker is isolation so when you use something like python3.7-slim, its fine to use the system python. Thats the whole point of using the image. Now you still want to run and install pip packages as a non-root user of course, but you don't need a virtualenv in docker.
dkarp 7y ago
Absolutely agree. Which is why I don't use Alpine linux for python projects and wouldn't recommend anyone else does. In the article, they are only using pandas and not using a requirements.txt file. So it's a pretty perfect case to just use apk. In cases where you don't need many packages, you can still use Alpine if you would like without most of the downsides mentioned in the article.
- swiley 7y ago
  I write a decent amount of python on alpine, in general I stick to the standard library and occasionally something like pyserial, sqlite, or numpy for specific things. IMO this attitude of pulling in dozens of packages with pip is pretty dangerous, I don’t care which language you’re using.
  - kryptn 7y ago
    What's the alternative?
    - swiley 7y ago
      Removing features, moving them into micro services, and sometimes writing a class or two yourself.
      - kryptn 7y ago
        That just sounds like distributing the dependencies. You'll still install them all, just not in the same spot.

masklinn 7y ago

> If you just need pandas, then you can install it with apk along with many other packages and it'll be much quicker: https://pkgs.alpinelinux.org/packages?page=1&branch=edge&nam.. https://pkgs.alpinelinux.org/packages?page=1&branch=edge&nam.... And so again need to rely on something "non-standard". > Alpine are doing what they can, by maintaining common python packages on apk. It is pip that is tied to glibc The "manylinux" tag means glibc by necessity, because nobody has driven a PEP for tagging, detecting and managing musl-based wheels to completion[0]. The issue is not restricted to the choice of libc, or even linux alone, either. But again, that requires people actually put in the work to chip at and fix the issue[1]. [0] https://github.com/pypa/manylinux/issues/37 https://github.com/pypa/manylinux/issues/37 [1] https://github.com/pypa/packaging-problems/issues/69 https://github.com/pypa/packaging-problems/issues/69

dkarp 7y ago
Not everyone uses pip. Many data scientists use conda. Using you distribution's package manager is a legitimate solution. The point is though, that this is a python packaging issue. Not an Alpine issue and not an issue that Alpine can do much about. As you linked, pypa have a big task to make packages work on every possible platform. Unfortunately, this comes as an unexpected surprise for anyone who wants to try out Alpine - maybe pip should show a warning when wheels aren't available because of musl.
- alexhutcheson 7y ago
  https://xkcd.com/1987/ https://xkcd.com/1987/

hackernudes 7y ago

For those like me who have no idea what "wheels" is in this context, I think it is referring to https://github.com/pypa/wheel https://github.com/pypa/wheel This looks like some sort of binary Python packaging thingy. I think @dkarp is saying that there exists pre-existing binary builds for glibc for many things, but not for musl libc so they are forced to be built explicitly when using pip to manage Python dependencies. I could not figure out where this repository of binary packages is or who maintains it. Any corrections would be appreciated.

korijn 7y ago
Pre-built binary distributions ("wheels") and source distributions (which sometimes need a compilation step) just live side by side on pypi.org, the public python package index.
mumblemumble 7y ago
You've got it. Wheels in Python are like gems in ruby. (Or eggs in [python minus a few years].) The official Python package index has binary images, but they're compiled against glibc, and therefore don't work with Alpine. So when you `pip install` a package with binary dependencies on Alpine, it needs to compile them. Which can lead to a bit of a trip down a rabbit hole. As others have mentioned, Alpine's repository does include Alpine-compatible binary versions of many popular Python packages. So you can often save yourself a lot of trouble by using apk instead of pip to get those.
- oefrha 7y ago
  > The official Python package index has binary images, but they're compiled against glibc, and therefore don't work with Alpine. That’s a false statement (depending on interpretation). Whether a PyPI package has wheels is entirely at the discretion of the package maintainer, who is responsible for compiling all source and binary distributions; PyPI does not compile anything and only accepts uploads, verbatim. If the maintainer doesn’t upload wheels or doesn’t upload wheels for your platform, then no wheels for you (not from PyPI). In general compiling statically linked wheels (when you have C extensions and external dependencies) is a fairly involved process. At the moment there are six platform tags for Linux wheels supported by PyPI: manylinux{1,2010,2014}_{x86_64,i686}. Each is based on a CentOS release with very old glibc. See https://github.com/pypa/manylinux https://github.com/pypa/manylinux PEP 513 - maynylinux1 https://www.python.org/dev/peps/pep-0513/ https://www.python.org/dev/peps/pep-0513/ PEP 571 - manylinux2010 https://www.python.org/dev/peps/pep-0571/ https://www.python.org/dev/peps/pep-0571/ PEP 599 - manylinux2014 https://www.python.org/dev/peps/pep-0599/ https://www.python.org/dev/peps/pep-0599/