4 ms·

I’m not a Python dev. I do need to occasionally run things written in Python. I made the mistake of trying to get pip + Conda + pyenv (or whatever) to install a

I’m not a Python dev. I do need to occasionally run things written in Python. I made the mistake of trying to get pip + Conda + pyenv (or whatever) to install a fairly simple tool. I have no idea how the dev got their setup working, but it was totally and utterly unreproducable, even after they sat down on my computer for several hours. In that amount of time, I could have probably rewritten it in PHP (that’s actually exactly what I ended up doing while they attempted to get a Docker container running). Needless to say, I will only use the distro package manager these days. I know the versions are (probably) compatible, maintainers will usually backport security patches, etc. You get none of that using whatever flavor of the week python package manager.

rolandog 5y ago
I had to learn this by FUBAR'ing my system a long time ago, but my setup process for working with a python package from PyPI (i.e. not installable by or updated enough in OS's package manager) nowadays is: - sudo apt install pip3 - pip3 install --user --upgrade pipenv (In workdir): - pipenv install --three package - pipenv run package --option Works like a charm and doesn't mess with my system.
noodlesUK 5y ago
I think this actually gets to the crux of the matter. The existing python dependency management tools (especially the new shiny ones like poetry) are very much designed by python devs, for python devs. What you're describing is a totally different use case, which is running released software in a sensible way. The distro package managers are probably the best place for that, but bridging the gap between them and the python ecosystem is an obvious challenge.
nerdponx 5y ago
This is especially frustrating to read because one of the main selling points of Conda is reproducibility. In data science teams, I've found it indispensable for making sure people can all run each others' project code. So for anyone reading this in the future: don't try to use Pyenv to install Conda. Pyenv tries to set up shims for every binary in the Conda env, which will likely break your PATH. Pyenv supports installing Conda because Anaconda used to be "just" a Python distribution. They can otherwise coexist without trouble on the same system.
- pwang 5y ago
  Thanks for the feedback... I had no idea that pyenv was hosing conda envs. Do you have more details on this brokenness?
  - nerdponx 5y ago
    My understanding of the problem is that Pyenv attempts to detect the contents of "/bin" relative to the top level of every Python installation that it manages. It does this so that it can set up its shim to handle any executable that gets installed in any Pyenv-managed environment. This is how Pyenv creates the "foobar is not available in your current environment, but is available in x.y.z" message. It's also a much more reliable solution than trying to explicitly whitelist every possible script that might get installed. The problem is that this was only designed to work for Python executables and scripts installed by Pip. Conda environments can contain a lot more than that; it's not hard to end up with an entire C compiler toolchain in there (possibly even both GCC and LLVM) or even Coreutils. If Pyenv detects `bin/gcc` in a Conda env, it will set up a system-wide shim for GCC, which no longer passes the `gcc` command along to the OS, but intercepts it, only to inform you that no such command exists in the current env! So it's not that Pyenv hoses Conda envs. It's that Pyenv can hose PATH if you have it manage a Conda installation, and if that Conda installation ends up with non-Python stuff in `/bin`. Obviously I don't know what exactly was broken when you tried to set up that application. But this particular adverse interaction bit me at work a few years ago, and ever since then I have insisted that Pyenv should never manage a Conda installation. I think that's a reasonable policy anyway, in light of the facts that: 1) Conda isn't really a "Python distribution" anymore. 2) The Pyenv installer just runs the opaque Conda installer script and there's basically no way to control the version that gets installed. 3) They are different tools that serve different purposes and it doesn't make sense to have one manage the other anyway. 4) You probably shouldn't use the Python that's installed in the base Conda environment anyway. You need that to run Conda itself, and you want to keep the list of requirements small to make sure that updates can progress cleanly. It's basically the same as any Linux package manager like APT. Except of course, those tools don't generally support "environments" other than chroot.
formerly_proven 5y ago
Both Conda and pyenv are for managing (their own) Python installations to run different Python versions in parallel, it doesn't really compute to try and use both at the same time.
1337shadow 5y ago
> I have no idea how the dev got their setup working Tip: in such situations I start looking at the CI scripts in their repositories. Not ideal but gets me through!
- pdimitar 5y ago
  Same BTW. And I started having a Docker image for each Python script that I need to run (I'm not a Python dev). Took me a while but I've mostly tamed the beast. And at least I can run the occasional script that has no alternatives.
vultour 5y ago
It's not really the fault of Python that somebody fucked their environment beyond belief. I've always used pip and almost never had a problem. A "fairly simple" application should not require the tools you mentioned.
- pdimitar 5y ago
  That, my friend, is called a filter bubble. And your comment is not productive btw. I for one never "fucked my environment", I only ever used distro packages and I still couldn't make a simple script even start without it spitting cryptic error messages about modules or paths. I'm not a Python dev; I just needed to run a tool that didn't have any other alternatives. Meanwhile, I can download scripts written in a range of other languages and just fire 1-2 commands and the thing will work. With Python? Almost never.

rolandog 5y ago

I had to learn this by FUBAR'ing my system a long time ago, but my setup process for working with a python package from PyPI (i.e. not installable by or updated enough in OS's package manager) nowadays is: - sudo apt install pip3 - pip3 install --user --upgrade pipenv (In workdir): - pipenv install --three package - pipenv run package --option Works like a charm and doesn't mess with my system.

noodlesUK 5y ago

I think this actually gets to the crux of the matter. The existing python dependency management tools (especially the new shiny ones like poetry) are very much designed by python devs, for python devs. What you're describing is a totally different use case, which is running released software in a sensible way. The distro package managers are probably the best place for that, but bridging the gap between them and the python ecosystem is an obvious challenge.

nerdponx 5y ago

This is especially frustrating to read because one of the main selling points of Conda is reproducibility. In data science teams, I've found it indispensable for making sure people can all run each others' project code. So for anyone reading this in the future: don't try to use Pyenv to install Conda. Pyenv tries to set up shims for every binary in the Conda env, which will likely break your PATH. Pyenv supports installing Conda because Anaconda used to be "just" a Python distribution. They can otherwise coexist without trouble on the same system.

pwang 5y ago
Thanks for the feedback... I had no idea that pyenv was hosing conda envs. Do you have more details on this brokenness?
- nerdponx 5y ago
  My understanding of the problem is that Pyenv attempts to detect the contents of "/bin" relative to the top level of every Python installation that it manages. It does this so that it can set up its shim to handle any executable that gets installed in any Pyenv-managed environment. This is how Pyenv creates the "foobar is not available in your current environment, but is available in x.y.z" message. It's also a much more reliable solution than trying to explicitly whitelist every possible script that might get installed. The problem is that this was only designed to work for Python executables and scripts installed by Pip. Conda environments can contain a lot more than that; it's not hard to end up with an entire C compiler toolchain in there (possibly even both GCC and LLVM) or even Coreutils. If Pyenv detects `bin/gcc` in a Conda env, it will set up a system-wide shim for GCC, which no longer passes the `gcc` command along to the OS, but intercepts it, only to inform you that no such command exists in the current env! So it's not that Pyenv hoses Conda envs. It's that Pyenv can hose PATH if you have it manage a Conda installation, and if that Conda installation ends up with non-Python stuff in `/bin`. Obviously I don't know what exactly was broken when you tried to set up that application. But this particular adverse interaction bit me at work a few years ago, and ever since then I have insisted that Pyenv should never manage a Conda installation. I think that's a reasonable policy anyway, in light of the facts that: 1) Conda isn't really a "Python distribution" anymore. 2) The Pyenv installer just runs the opaque Conda installer script and there's basically no way to control the version that gets installed. 3) They are different tools that serve different purposes and it doesn't make sense to have one manage the other anyway. 4) You probably shouldn't use the Python that's installed in the base Conda environment anyway. You need that to run Conda itself, and you want to keep the list of requirements small to make sure that updates can progress cleanly. It's basically the same as any Linux package manager like APT. Except of course, those tools don't generally support "environments" other than chroot.

formerly_proven 5y ago

Both Conda and pyenv are for managing (their own) Python installations to run different Python versions in parallel, it doesn't really compute to try and use both at the same time.

1337shadow 5y ago

> I have no idea how the dev got their setup working Tip: in such situations I start looking at the CI scripts in their repositories. Not ideal but gets me through!

pdimitar 5y ago
Same BTW. And I started having a Docker image for each Python script that I need to run (I'm not a Python dev). Took me a while but I've mostly tamed the beast. And at least I can run the occasional script that has no alternatives.

vultour 5y ago

It's not really the fault of Python that somebody fucked their environment beyond belief. I've always used pip and almost never had a problem. A "fairly simple" application should not require the tools you mentioned.

pdimitar 5y ago
That, my friend, is called a filter bubble. And your comment is not productive btw. I for one never "fucked my environment", I only ever used distro packages and I still couldn't make a simple script even start without it spitting cryptic error messages about modules or paths. I'm not a Python dev; I just needed to run a tool that didn't have any other alternatives. Meanwhile, I can download scripts written in a range of other languages and just fire 1-2 commands and the thing will work. With Python? Almost never.