8 ms·

How Python dependency management can be improved at this point? Should we hold our breath for Python 4 to solve it?

ddavis 6y ago
The current paradigm feels OK to me. Unfortunately there are a lot of old crappy "intro to Python" blog posts and tutorials still telling people to use 'sudo pip install' to install packages on their system Python. pip/setuptools do not support that workflow. I think it's slowly getting more widely recognized.
- dheera 6y ago
  It seems to me like Python 3.8 has broken a LOT of stuff, violating semantic versioning. Like, for example, RealSense cameras don't work anymore.
  - nuclear_eclipse 6y ago
    Python has never used semantic versioning. There is a long history of small, breaking changes between "minor" releases of Python. Eg, Python 3.7 made `async` and `await` global keywords, breaking any case where old code used `async` as a variable name.
    - rumanator 6y ago
      To be fair, Python's minor versions have been handled as semver's major versions since ever.
    - jnxx 6y ago
      By the way, is there any list of such small breaking changes in Python?
  - eesmith 6y ago
    If I understand https://github.com/IntelRealSense/librealsense/issues/6296 https://github.com/IntelRealSense/librealsense/issues/6296 and https://github.com/IntelRealSense/librealsense/issues/5777#issuecomment-582480988 https://github.com/IntelRealSense/librealsense/issues/5777#i... correctly, what happened is the RealSense developers don't provide a pre-compiled version for 3.8 on PyPI. Python is "broken" in that the C API does not maintain backwards compatibility. (See https://www.python.org/dev/peps/pep-0620/#the-c-api-blocks-cpython-evolutions https://www.python.org/dev/peps/pep-0620/#the-c-api-blocks-c... for a proposed solution.) RealSense is "broken" in that they don't provide builds for all the versions of Python their users use. But that's because "Officially, Librealsense is only currently supported up to Ubuntu 18.04 LTS" Perhaps also your process is "broken" for depending only on pre-compiled builds; the published solution is to build from source. https://github.com/IntelRealSense/librealsense/issues/6296#issuecomment-644291191 https://github.com/IntelRealSense/librealsense/issues/6296#i... says it's possible. Or "broken" for using an unsupported environment and expecting it will work. Software in general is broken. It's amazing that it works as well as it does.
    - dheera 6y ago
      > the published solution is to build from source I tried that, it didn't work, and they keep closing issues about it instead of addressing it. They are a company with a 200B market cap, and it's already 5 months since the release of 20.04 and they still can't get their drivers straight. People like me want to use their hardware. I don't have time to be debugging their drivers -- I have other work to do. I don't think it's unreasonable ask for them to support the latest LTS release of the most common Linux distribution in a timely manner with pre-compiled binaries that just work, in return for the $179 that I and tens of thousands of others paid for the device. This isn't some open source community software that I'd be glad to pitch in some time and help debug. This is a driver for a device that I paid a for-profit company for and I expect timely support especially for an exceedingly common OS that I use.
      - eesmith 6y ago
        You wrote "Python 3.8 has broken a LOT of stuff", as if it were only Python's problem. Clearly other things are broken too.
- neckardt 6y ago
  I'm one of those people using 'sudo pip install'. What should I be using instead?
  - CapriciousCptl 6y ago
    Python -m venv in your project directory. Then source ./bin/activate. Pip install from the project directory as your regular user (no sudo). You can also set up venv (virtual environments) in home or somewhere else instead of for each project. This is called python virtual environments and very easy to adopt.
    - ydnaclementine 6y ago
      Ruby has this figured out well by comparison. Install the bundler gem (which you're going to be installing anyway to get dependencies), cd into the project, and prefix your commands with `bundle exec` to use project gems Remembering to start/stop virtualenv and remembering the names of your virtual envs is one too many steps
      - easton 6y ago
        Or just use poetry/pipenv shell (or poetry/pipenv run for one-off commands). Both provide a good wrapper, although poetry seems a bit more stable these days.
      - thewrinklyninja 6y ago
        Perl has copied this as well with Carton. Works really well and as standard keeps the modules in the project dir. https://metacpan.org/pod/Carton https://metacpan.org/pod/Carton
      - viraptor 6y ago
        If you prefer "bundle exec foo", then "bin/python -mfoo" is pretty much the equivalent and doesn't require running activate before.
    - kevin_thibedeau 6y ago
corty 6y ago
Just don't use pip, use apt. What isn't packaged isn't worth it anyways. Non-distro package managers, a dozen per language are a disease.
- Joker_vD 6y ago
  I think you meant to write "just use pre-built Docker images".
- mistrial9 6y ago
  the reality is that there are binary needs at the OS link level, and then there is pure-python or ML with drivers or whatnot, per user.. its not actually easy to divide which is which ..
- im3w1l 6y ago
  If a python tool wants to be in apt, then it can only use apt dependencies. So that's something to think about.
- jnwatson 6y ago
  apt is completely inappropriate unless you're using 3 year old dependencies. Ubuntu 20.04 (released in April) doesn't have Python 3.8, which was released in October 2019.
  - pritambaral 6y ago
    This is patently false; the default python3 on Ubuntu 20.04[1] is 3.8.2. A separate package python3.8 is also available in built-in repositories for all supported versions of Ubuntu going back all the way to 18.04[2]. 1: https://packages.ubuntu.com/focal/python3 https://packages.ubuntu.com/focal/python3 2: https://packages.ubuntu.com/search?keywords=python3.8 https://packages.ubuntu.com/search?keywords=python3.8
  - avhon1 6y ago
    Apt is perfectly able to deliver recent versions. Here is Debian's official Python3 package, currently version 3.8.2-3: https://packages.debian.org/bullseye/python3 https://packages.debian.org/bullseye/python3
akersten 6y ago
Honestly, just throw everything in a venv and forget anything about needing to manage platform versions again. Unless, of course, virtualenv can't be installed on the host in the first place due to setuptools/pip issues :)
- abiogenesis 6y ago
  Python comes with built-in venv support since 3.3: https://docs.python.org/3/library/venv.html https://docs.python.org/3/library/venv.html
  - AlphaSite 6y ago
    Deviant packaging breaks/removes this.
    - erfgh 6y ago
      That's why it's deviant.
      - Conan_Kudo 6y ago
        I assume the poster meant "Debian packaging", but yes. Debian and Ubuntu remove venv support along with distutils from the standard library by default.
        regularfry 6y ago
        This is annoying, but it does encourage the principle of "the system python is for the system, and only incidentally for the users of that system". I'm almost always in a non-system Python for anything interesting, unless I'm writing something to integrate with the OS.
        Conan_Kudo 6y ago
        It breaks interrogation of the Python interpreter, since those functions are part of distutils. It also breaks creating virtualenvs if it relies on the venv module.
pdonis 6y ago
I have never really understood how Python dependency management got broken in the first place. The basic model is the same as the basic Unix shell model: you have a path, which is just a list of places where executable thingies can be found, and whatever wants to execute the thingies looks in each place in the list, in order, until it finds the thingie it wants, or it bails out with an error if it's gone through the entire list and can't find what it's looking for. In a Unix shell, the list is $PATH (or maybe $LD_LIBRARY_PATH, since most Python dependencies are libraries); in Python, it's sys.path. Installation of an executable thingie then is simply a matter of "make sure this ends up in one of the places listed in the path". What is so hard about this?
- misnome 6y ago
  What about different packages with different version dependencies?
  - brendawalsh 6y ago
    So, my most recent success has been using pyenv and pipenv. I install pyenv first, then install any specific point releases I need, then switch to the python version I want to use, and then use pipenv to install dependencies or run commands in the virtual environment.
    - zelphirkalt 6y ago
      I think lately pipenv also interacts with pyenv to do this automatically.
  - pdonis 6y ago
    Version numbering and labeling of the packages--the same way Unix solves the same problem with shared libraries. A program that wants a specific version of a library looks for it by specific version.

The current paradigm feels OK to me. Unfortunately there are a lot of old crappy "intro to Python" blog posts and tutorials still telling people to use 'sudo pip install' to install packages on their system Python. pip/setuptools do not support that workflow. I think it's slowly getting more widely recognized.

dheera 6y ago
It seems to me like Python 3.8 has broken a LOT of stuff, violating semantic versioning. Like, for example, RealSense cameras don't work anymore.
- nuclear_eclipse 6y ago
  Python has never used semantic versioning. There is a long history of small, breaking changes between "minor" releases of Python. Eg, Python 3.7 made `async` and `await` global keywords, breaking any case where old code used `async` as a variable name.
  - rumanator 6y ago
    To be fair, Python's minor versions have been handled as semver's major versions since ever.
  - jnxx 6y ago
    By the way, is there any list of such small breaking changes in Python?
- eesmith 6y ago
  If I understand https://github.com/IntelRealSense/librealsense/issues/6296 https://github.com/IntelRealSense/librealsense/issues/6296 and https://github.com/IntelRealSense/librealsense/issues/5777#issuecomment-582480988 https://github.com/IntelRealSense/librealsense/issues/5777#i... correctly, what happened is the RealSense developers don't provide a pre-compiled version for 3.8 on PyPI. Python is "broken" in that the C API does not maintain backwards compatibility. (See https://www.python.org/dev/peps/pep-0620/#the-c-api-blocks-cpython-evolutions https://www.python.org/dev/peps/pep-0620/#the-c-api-blocks-c... for a proposed solution.) RealSense is "broken" in that they don't provide builds for all the versions of Python their users use. But that's because "Officially, Librealsense is only currently supported up to Ubuntu 18.04 LTS" Perhaps also your process is "broken" for depending only on pre-compiled builds; the published solution is to build from source. https://github.com/IntelRealSense/librealsense/issues/6296#issuecomment-644291191 https://github.com/IntelRealSense/librealsense/issues/6296#i... says it's possible. Or "broken" for using an unsupported environment and expecting it will work. Software in general is broken. It's amazing that it works as well as it does.
  - dheera 6y ago
    > the published solution is to build from source I tried that, it didn't work, and they keep closing issues about it instead of addressing it. They are a company with a 200B market cap, and it's already 5 months since the release of 20.04 and they still can't get their drivers straight. People like me want to use their hardware. I don't have time to be debugging their drivers -- I have other work to do. I don't think it's unreasonable ask for them to support the latest LTS release of the most common Linux distribution in a timely manner with pre-compiled binaries that just work, in return for the $179 that I and tens of thousands of others paid for the device. This isn't some open source community software that I'd be glad to pitch in some time and help debug. This is a driver for a device that I paid a for-profit company for and I expect timely support especially for an exceedingly common OS that I use.
    - eesmith 6y ago
      You wrote "Python 3.8 has broken a LOT of stuff", as if it were only Python's problem. Clearly other things are broken too.
neckardt 6y ago
I'm one of those people using 'sudo pip install'. What should I be using instead?
- CapriciousCptl 6y ago
  Python -m venv in your project directory. Then source ./bin/activate. Pip install from the project directory as your regular user (no sudo). You can also set up venv (virtual environments) in home or somewhere else instead of for each project. This is called python virtual environments and very easy to adopt.
  - ydnaclementine 6y ago
    Ruby has this figured out well by comparison. Install the bundler gem (which you're going to be installing anyway to get dependencies), cd into the project, and prefix your commands with `bundle exec` to use project gems Remembering to start/stop virtualenv and remembering the names of your virtual envs is one too many steps
    - easton 6y ago
      Or just use poetry/pipenv shell (or poetry/pipenv run for one-off commands). Both provide a good wrapper, although poetry seems a bit more stable these days.
    - thewrinklyninja 6y ago
      Perl has copied this as well with Carton. Works really well and as standard keeps the modules in the project dir. https://metacpan.org/pod/Carton https://metacpan.org/pod/Carton
    - viraptor 6y ago
      If you prefer "bundle exec foo", then "bin/python -mfoo" is pretty much the equivalent and doesn't require running activate before.
  - kevin_thibedeau 6y ago

corty 6y ago

Just don't use pip, use apt. What isn't packaged isn't worth it anyways. Non-distro package managers, a dozen per language are a disease.

Joker_vD 6y ago
I think you meant to write "just use pre-built Docker images".
mistrial9 6y ago
the reality is that there are binary needs at the OS link level, and then there is pure-python or ML with drivers or whatnot, per user.. its not actually easy to divide which is which ..
im3w1l 6y ago
If a python tool wants to be in apt, then it can only use apt dependencies. So that's something to think about.
jnwatson 6y ago
apt is completely inappropriate unless you're using 3 year old dependencies. Ubuntu 20.04 (released in April) doesn't have Python 3.8, which was released in October 2019.
- pritambaral 6y ago
  This is patently false; the default python3 on Ubuntu 20.04[1] is 3.8.2. A separate package python3.8 is also available in built-in repositories for all supported versions of Ubuntu going back all the way to 18.04[2]. 1: https://packages.ubuntu.com/focal/python3 https://packages.ubuntu.com/focal/python3 2: https://packages.ubuntu.com/search?keywords=python3.8 https://packages.ubuntu.com/search?keywords=python3.8
- avhon1 6y ago
  Apt is perfectly able to deliver recent versions. Here is Debian's official Python3 package, currently version 3.8.2-3: https://packages.debian.org/bullseye/python3 https://packages.debian.org/bullseye/python3

akersten 6y ago

Honestly, just throw everything in a venv and forget anything about needing to manage platform versions again. Unless, of course, virtualenv can't be installed on the host in the first place due to setuptools/pip issues :)

abiogenesis 6y ago
Python comes with built-in venv support since 3.3: https://docs.python.org/3/library/venv.html https://docs.python.org/3/library/venv.html
- AlphaSite 6y ago
  Deviant packaging breaks/removes this.
  - erfgh 6y ago
    That's why it's deviant.
    - Conan_Kudo 6y ago
      I assume the poster meant "Debian packaging", but yes. Debian and Ubuntu remove venv support along with distutils from the standard library by default.
      - regularfry 6y ago
        This is annoying, but it does encourage the principle of "the system python is for the system, and only incidentally for the users of that system". I'm almost always in a non-system Python for anything interesting, unless I'm writing something to integrate with the OS.
        Conan_Kudo 6y ago
        It breaks interrogation of the Python interpreter, since those functions are part of distutils. It also breaks creating virtualenvs if it relies on the venv module.

pdonis 6y ago

I have never really understood how Python dependency management got broken in the first place. The basic model is the same as the basic Unix shell model: you have a path, which is just a list of places where executable thingies can be found, and whatever wants to execute the thingies looks in each place in the list, in order, until it finds the thingie it wants, or it bails out with an error if it's gone through the entire list and can't find what it's looking for. In a Unix shell, the list is $PATH (or maybe $LD_LIBRARY_PATH, since most Python dependencies are libraries); in Python, it's sys.path. Installation of an executable thingie then is simply a matter of "make sure this ends up in one of the places listed in the path". What is so hard about this?

misnome 6y ago
What about different packages with different version dependencies?
- brendawalsh 6y ago
  So, my most recent success has been using pyenv and pipenv. I install pyenv first, then install any specific point releases I need, then switch to the python version I want to use, and then use pipenv to install dependencies or run commands in the virtual environment.
  - zelphirkalt 6y ago
    I think lately pipenv also interacts with pyenv to do this automatically.
- pdonis 6y ago
  Version numbering and labeling of the packages--the same way Unix solves the same problem with shared libraries. A program that wants a specific version of a library looks for it by specific version.