3 ms·

Dependency management can be a pain. Want to install numpy? You have to use alpine's version because compiling it from pip requires glibc.

rkeene2 7y ago
Can you do something like `pip install --no-binary :all:` (or equivalent pip.conf entry) ?
- greenshackle2 7y ago
  You normally will not get wheels with compiled binaries on alpine anyway because it is not compatible with any of the platform tags (manylinux* is glibc only). I think they're saying numpy won't compile with musl off the shelf, you need patches for alpine. Pip just won't help you there.
ddevault 7y ago
Using anything other than distro packages is a pain. pip is an anti-pattern and one giant security risk.
- fortytw2 7y ago
  by doing this, do you suggest also tying your dependencies versions to those packages with the distro version you're running?
  - ddevault 7y ago
    Yes, or just shadow them with newer versions and run your own repo.
- takeda 7y ago
  Disagree, although pip should assume --user option by default which installs the packages in user directory instead of system directory (some distros (was it Debian?) modify it to work that way). The best way of using Python though, is to create a virtualenv (since Python 3, venv module is built in which makes it straight forward, python -m venv <directory>) and install packages inside of it. This gives you control to use exact versions of packages you need in your applications and makes your application not tied to the OS, so system upgrades and Python version updates are much easier to perform.
  - dnautics 7y ago
    > This gives you control to use exact versions of packages you need in your applications and makes your application not tied to the OS Is this true for drivers, too? like the coupling of cuda versions to tensorflow versions?
    - takeda 7y ago
      Unfortunately that's one weakness, but unless you package your application using system packager (which comes with its own set of issues[1]) If you have any dependencies that depend on system libraries there are two options: - the libraries can be compiled statically, they call them manylinux wheels, this generally works well, except if your dependencies are overlapping with python or other packages' dependencies. Most commonly this happen if package depends on openssl. If the compiled-in openssl is different than the version on the system in certain circumstances python might crash. This is for example why psycopg2 was initially distributed as manylinux but now they opt for the second method (they still provide psycopg2-binary but they discourage its use; most time it works fine but it is a problem on certain distros) - make python create bindings on installation, this makes installation process do small compilation to create bindings between system library and python, this makes the packages robust, but tied to the OS. In my experience this is not an issue, but it can be annoying, because you still don't have 100% control of dependencies. This issue is what made me investigate Nix[2] package manager, because it gives you full control of all dependencies down to libc making everything fully reproducible, so you can control the exact version of python to use and all system and python dependencies. [1] in one of my previous jobs on a team that I joined they were running their applications on CentOS 5 that was already EOL, because rebuilding RPMs to CentOS 7 (at the time the newest version) was a lot of work, another issue was that they were bound to python package version that came with the system, they could create their own RPMs, but no one did it because maintaining that was adding more work. I spent time converting the python code to be packaged using setuptools. Once that was done switching the OS was trivial. We finally also could use the latest versions of many of our dependencies. [2] This is IMO good article which describes tooling that makes Nix much more enjoyable to use: https://christine.website/blog/how-i-start-nix-2020-03-08 https://christine.website/blog/how-i-start-nix-2020-03-08
greenshackle2 7y ago
Also, annoyingly, there is still no platform tag for python wheels for alpine/musl, so you cannot get binary wheels for it from pypi.

rkeene2 7y ago

Can you do something like `pip install --no-binary :all:` (or equivalent pip.conf entry) ?

greenshackle2 7y ago
You normally will not get wheels with compiled binaries on alpine anyway because it is not compatible with any of the platform tags (manylinux* is glibc only). I think they're saying numpy won't compile with musl off the shelf, you need patches for alpine. Pip just won't help you there.

ddevault 7y ago

Using anything other than distro packages is a pain. pip is an anti-pattern and one giant security risk.

fortytw2 7y ago
by doing this, do you suggest also tying your dependencies versions to those packages with the distro version you're running?
- ddevault 7y ago
  Yes, or just shadow them with newer versions and run your own repo.
takeda 7y ago
Disagree, although pip should assume --user option by default which installs the packages in user directory instead of system directory (some distros (was it Debian?) modify it to work that way). The best way of using Python though, is to create a virtualenv (since Python 3, venv module is built in which makes it straight forward, python -m venv <directory>) and install packages inside of it. This gives you control to use exact versions of packages you need in your applications and makes your application not tied to the OS, so system upgrades and Python version updates are much easier to perform.
- dnautics 7y ago
  > This gives you control to use exact versions of packages you need in your applications and makes your application not tied to the OS Is this true for drivers, too? like the coupling of cuda versions to tensorflow versions?
  - takeda 7y ago
    Unfortunately that's one weakness, but unless you package your application using system packager (which comes with its own set of issues[1]) If you have any dependencies that depend on system libraries there are two options: - the libraries can be compiled statically, they call them manylinux wheels, this generally works well, except if your dependencies are overlapping with python or other packages' dependencies. Most commonly this happen if package depends on openssl. If the compiled-in openssl is different than the version on the system in certain circumstances python might crash. This is for example why psycopg2 was initially distributed as manylinux but now they opt for the second method (they still provide psycopg2-binary but they discourage its use; most time it works fine but it is a problem on certain distros) - make python create bindings on installation, this makes installation process do small compilation to create bindings between system library and python, this makes the packages robust, but tied to the OS. In my experience this is not an issue, but it can be annoying, because you still don't have 100% control of dependencies. This issue is what made me investigate Nix[2] package manager, because it gives you full control of all dependencies down to libc making everything fully reproducible, so you can control the exact version of python to use and all system and python dependencies. [1] in one of my previous jobs on a team that I joined they were running their applications on CentOS 5 that was already EOL, because rebuilding RPMs to CentOS 7 (at the time the newest version) was a lot of work, another issue was that they were bound to python package version that came with the system, they could create their own RPMs, but no one did it because maintaining that was adding more work. I spent time converting the python code to be packaged using setuptools. Once that was done switching the OS was trivial. We finally also could use the latest versions of many of our dependencies. [2] This is IMO good article which describes tooling that makes Nix much more enjoyable to use: https://christine.website/blog/how-i-start-nix-2020-03-08 https://christine.website/blog/how-i-start-nix-2020-03-08

greenshackle2 7y ago

Also, annoyingly, there is still no platform tag for python wheels for alpine/musl, so you cannot get binary wheels for it from pypi.