5 ms·

This is my decision flow and I rarely have an issue: Are you the end user of the Python code? Yes -> Is available in your distro? Yes->

by fer 5y ago

This is my decision flow and I rarely have an issue: Are you the end user of the Python code? Yes -> Is available in your distro? Yes->Use package manager No->Use pip install in user mode No -> Create virtualenv with the Python version you want (including pypy!) and do your pip thing there Some extreme use cases may benefit from anaconda, but personally I've never needed to use it. My only pain point is dealing with legacy code that relies on PYTHONPATH. Nothing good ever starts setting PYTHONPATH.

nerdponx 5y ago
> Create virtualenv with the Python version you want (including pypy!) and do your pip thing there You might benefit from using Pipx in this case: https://pypa.github.io/pipx/ https://pypa.github.io/pipx/ Pipx is good for the case of "I want to run a standalone Python application that is available through Pip, but not my system's package repo." This is a more common case than you might think. It's a sensible alternative to `pip install --user`, and having self-contained deps for tool is a bit like `npm install --global` or even `volta install`.
- WhyNotHugo 5y ago
  Yup, this kinda works. It doesn't address the greater issue tho: that it's getting harder and harder for distributions to package things right, and provider packages for their users (evidenced by the fact that you need a second package manager just for python stuff).
  - nerdponx 5y ago
    How is Pip any different than Cpanm, NPM, Gem, Luarocks, Nimble, Go's thing, Cargo, whatever JVM people use, whatever Haskell people use, etc. in that regard? Distros have a hard job, but at the same time programming language tooling devs have more "customers" than just distro maintainers.
isolli 5y ago
Hm, why? I'm a happy user of PYTHONPATH!
- doubleunplussed 5y ago
  It's completely global, shared by all Python interpreters of all versions. I set PYTHONPATH, but the code in that directory is solely small debugging utils of mine that I want available in every Python interpreter, and I make sure not to put anything more complex in there.
  - cturner 5y ago
    "It's completely global" It doesn't have to be. You can have a launcher for a project that sets PYTHONPATH just when you launch that project. What is bad practice is to be setting PYTHONPATH in your .bashrc, and for the reason you give - that makes it global across python launches.
    - doubleunplussed 5y ago
      That'll prevent it leaking to most things, but not to subprocesses of your application. For example your application might interact with command-line tools written in Python, and unless you delete PYTHONPATH from the environment variables prior to launching any subprocesses, they'll inherit it. This could lead to subtle and confusing breakage.
  - cozzyd 5y ago
    So is PATH and LD_LIBRARY_PATH. You just change those as you need to...
  - isolli 5y ago
    Alright. I actually set it in a Docker container. It works well there.
- Uberphallus 5y ago
  The only justified situation I can find is when you are working on two (or more) independent components at the same time. My pain point in particular with PYTHONPATH (or playing with sys.path) is that people tend to use it with the only purpose of making import lines shorter, which brings naming collisions of all sorts when you aren't creative enough.
WhyNotHugo 5y ago
> Use pip install in user mode This is a great recipe for disaster. Whatever you install in user mode will shadow anything installed system-wide, so when you try to run some system-wide project, it may now fail. I'm also not a fan of how it drops scripts into `./.local/bin`, since that's where I keep my own script, and is version controlled. The installation will also be frozen and never get updated -- unless you remember to do it manually. Finally, and worst of all, this leaves you in a dead end if your packages have conflicting dependencies, which is too often the case in Python-land.
- pfranz 5y ago
  So you're suggesting always using virtualenv? I used to just use pip to install to the system. Months/years later I would try to untangle the mess of packages I was just playing with, what the OS wanted/needed, I got those conflicting dependencies you mention, etc. I usually ended up reinstalling the OS. At the time I may not have been as knowledgeable about where the OS package manager keeps packages vs pip--but the whole thing wasn't very user-friendly either. For years I've been installing into user knowing I can just blow it away. I've dabbled with virtualenv, but it's such a pain to set up and activate. If I have a few projects with similar libraries it's more of a pain to set them all up and switch around. If I end up using a script for something important, I just spend the extra time at that point to "package" it.
  - pwang 5y ago
    This is one of the reasons people use Anaconda/miniconda for non-data science work: conda environments are self-contained Python installs, so if you conda/pip install packages into those environments, they will not break each other. This design requirement arose from the specific needs of numerical computing (which always drags in a ton of system-level C/C++/FORTRAN dependencies), but is a generically useful design construct. Anaconda is a distro, and conda is a package manager, that works across OS platforms and hardware architectures, and installs cleanly into userland without requiring admin privileges. The only way we achieve this difficult goal is by creating a distro and build system that creates "portable" packages that can be relocated/relinked at install-time. Ultimately, Python's challenges in this department come from the fact that it has such great integration with low-level C/C++ libraries. This gives it super powers as duct tape/glue language, but it also drags it down into the packaging tech debt of C/C++. Hmm... maybe I should write that blog post: "Python Packaging Isn't The Problem; C/C++ Is." :-)

by fer 5y ago

nerdponx 5y ago

> Create virtualenv with the Python version you want (including pypy!) and do your pip thing there You might benefit from using Pipx in this case: https://pypa.github.io/pipx/ https://pypa.github.io/pipx/ Pipx is good for the case of "I want to run a standalone Python application that is available through Pip, but not my system's package repo." This is a more common case than you might think. It's a sensible alternative to `pip install --user`, and having self-contained deps for tool is a bit like `npm install --global` or even `volta install`.

WhyNotHugo 5y ago
Yup, this kinda works. It doesn't address the greater issue tho: that it's getting harder and harder for distributions to package things right, and provider packages for their users (evidenced by the fact that you need a second package manager just for python stuff).
- nerdponx 5y ago
  How is Pip any different than Cpanm, NPM, Gem, Luarocks, Nimble, Go's thing, Cargo, whatever JVM people use, whatever Haskell people use, etc. in that regard? Distros have a hard job, but at the same time programming language tooling devs have more "customers" than just distro maintainers.

isolli 5y ago

Hm, why? I'm a happy user of PYTHONPATH!

doubleunplussed 5y ago
It's completely global, shared by all Python interpreters of all versions. I set PYTHONPATH, but the code in that directory is solely small debugging utils of mine that I want available in every Python interpreter, and I make sure not to put anything more complex in there.
- cturner 5y ago
  "It's completely global" It doesn't have to be. You can have a launcher for a project that sets PYTHONPATH just when you launch that project. What is bad practice is to be setting PYTHONPATH in your .bashrc, and for the reason you give - that makes it global across python launches.
  - doubleunplussed 5y ago
    That'll prevent it leaking to most things, but not to subprocesses of your application. For example your application might interact with command-line tools written in Python, and unless you delete PYTHONPATH from the environment variables prior to launching any subprocesses, they'll inherit it. This could lead to subtle and confusing breakage.
- cozzyd 5y ago
  So is PATH and LD_LIBRARY_PATH. You just change those as you need to...
- isolli 5y ago
  Alright. I actually set it in a Docker container. It works well there.
Uberphallus 5y ago
The only justified situation I can find is when you are working on two (or more) independent components at the same time. My pain point in particular with PYTHONPATH (or playing with sys.path) is that people tend to use it with the only purpose of making import lines shorter, which brings naming collisions of all sorts when you aren't creative enough.

WhyNotHugo 5y ago

> Use pip install in user mode This is a great recipe for disaster. Whatever you install in user mode will shadow anything installed system-wide, so when you try to run some system-wide project, it may now fail. I'm also not a fan of how it drops scripts into `./.local/bin`, since that's where I keep my own script, and is version controlled. The installation will also be frozen and never get updated -- unless you remember to do it manually. Finally, and worst of all, this leaves you in a dead end if your packages have conflicting dependencies, which is too often the case in Python-land.

pfranz 5y ago
So you're suggesting always using virtualenv? I used to just use pip to install to the system. Months/years later I would try to untangle the mess of packages I was just playing with, what the OS wanted/needed, I got those conflicting dependencies you mention, etc. I usually ended up reinstalling the OS. At the time I may not have been as knowledgeable about where the OS package manager keeps packages vs pip--but the whole thing wasn't very user-friendly either. For years I've been installing into user knowing I can just blow it away. I've dabbled with virtualenv, but it's such a pain to set up and activate. If I have a few projects with similar libraries it's more of a pain to set them all up and switch around. If I end up using a script for something important, I just spend the extra time at that point to "package" it.
- pwang 5y ago
  This is one of the reasons people use Anaconda/miniconda for non-data science work: conda environments are self-contained Python installs, so if you conda/pip install packages into those environments, they will not break each other. This design requirement arose from the specific needs of numerical computing (which always drags in a ton of system-level C/C++/FORTRAN dependencies), but is a generically useful design construct. Anaconda is a distro, and conda is a package manager, that works across OS platforms and hardware architectures, and installs cleanly into userland without requiring admin privileges. The only way we achieve this difficult goal is by creating a distro and build system that creates "portable" packages that can be relocated/relinked at install-time. Ultimately, Python's challenges in this department come from the fact that it has such great integration with low-level C/C++ libraries. This gives it super powers as duct tape/glue language, but it also drags it down into the packaging tech debt of C/C++. Hmm... maybe I should write that blog post: "Python Packaging Isn't The Problem; C/C++ Is." :-)