8 ms·

I am not one to install python packages using my distro's package manager, but I totally agree with the sentiment that we need a more standard build/dependency

I am not one to install python packages using my distro's package manager, but I totally agree with the sentiment that we need a more standard build/dependency management system in python. I like poetry, and I think most people are heading that way, but it doesn't seem to play super nice with pyenv (which is a critical tool) a lot of the time, and I think that a first party endorsement of the "one true build system" a la golang or rust would be a huge step.

hkt 5y ago
Golang is a great model to follow. I think the fundamental thing that python (and ruby, when I still used it) has taught me is to run a mile if a language is without a robust dependency management system. The pain is just not worth it, even for a nice language with an otherwise robust ecosystem.
- yxhuvud 5y ago
  Huh? The package dependency system in Ruby is a breeze, and has been a solved problem since 2007 or whenever Bundler was made.
- noodlesUK 5y ago
  With golang, it's not just the dependency management side, gofmt (and various bits of other tooling) is also an incredible blessing. Yes, prescriptivism can sometimes feel weird to us computer nerds, but sometimes it's just much better to have a canonical way of doing common tasks. It means onboarding people is much easier, and infrastructure/code/whatever is more reusable.
- goodpoint 5y ago
  Go is terrible for distributions to package due to the poor library versioning. Also, static linking makes security updates really expensive to build at scale.
withinboredom 5y ago
I’m not a Python dev. I do need to occasionally run things written in Python. I made the mistake of trying to get pip + Conda + pyenv (or whatever) to install a fairly simple tool. I have no idea how the dev got their setup working, but it was totally and utterly unreproducable, even after they sat down on my computer for several hours. In that amount of time, I could have probably rewritten it in PHP (that’s actually exactly what I ended up doing while they attempted to get a Docker container running). Needless to say, I will only use the distro package manager these days. I know the versions are (probably) compatible, maintainers will usually backport security patches, etc. You get none of that using whatever flavor of the week python package manager.
- rolandog 5y ago
  I had to learn this by FUBAR'ing my system a long time ago, but my setup process for working with a python package from PyPI (i.e. not installable by or updated enough in OS's package manager) nowadays is: - sudo apt install pip3 - pip3 install --user --upgrade pipenv (In workdir): - pipenv install --three package - pipenv run package --option Works like a charm and doesn't mess with my system.
- noodlesUK 5y ago
  I think this actually gets to the crux of the matter. The existing python dependency management tools (especially the new shiny ones like poetry) are very much designed by python devs, for python devs. What you're describing is a totally different use case, which is running released software in a sensible way. The distro package managers are probably the best place for that, but bridging the gap between them and the python ecosystem is an obvious challenge.
- nerdponx 5y ago
  This is especially frustrating to read because one of the main selling points of Conda is reproducibility. In data science teams, I've found it indispensable for making sure people can all run each others' project code. So for anyone reading this in the future: don't try to use Pyenv to install Conda. Pyenv tries to set up shims for every binary in the Conda env, which will likely break your PATH. Pyenv supports installing Conda because Anaconda used to be "just" a Python distribution. They can otherwise coexist without trouble on the same system.
atoav 5y ago
I landed with poetry as well. The issue with python dependency managment in my book is, that it is incredibly (and needlessly) hard to learn how to do it properly. It took me years to figure out how things should work and there are still issues now and then that cost me a few hours to figure out and restore things. Meanwhile when I use Rust all these things are taken care of in cargo. It is part of the language. There is one right way to do things and the way is supported by comfortable tooling, that works so well that you literally don't even consider thinking about anything else. The way python does dependencies is totally unpythonic. The fact that it is 2021 and this isn't fixed or at least the number one priority of things that need fixing sheds a dark shadow on the whole language – a language that I like to use. Poetry is good. But it isn't as good as cargo, because it also has to deal with all the legacy cruft. To run code developed with poetry on a non-poetry system you have to figure out all the ways of dealing with envs, paths and such. Issues like these get me a little fired up, because the collective brainpower wasted on something that should have been elegantly solved in one place is gigantic.
- jhrmnn 5y ago
  To run code developed with Poetry, you shouldn’t even know it’s developed with Poetry. It should be released as a source distribution or a wheel on PyPI or as a conda package if it contains nontrivial binary extensions. These distributions should be then either packaged by the OS or they should be installed with all their dependencies into a separate virtual environment in /opt.
  - atoav 5y ago
    Sure, if I release the code I tend to do just that. However a lot of the code I have been writing was to be used on servers where installing poetry was not an option. That means there was no straightforward, well documented or otherwise easy way to copy the project onto the server an "just have it work". And this was not due to me not taking care of my dev environment. Now I know how to do this, so this is not a problem. My complaint was mostly, that this was a waste of time.
    - 5y ago
BerislavLopac 5y ago
> I like poetry, and I think most people are heading that way, but it doesn't seem to play super nice with pyenv Do you have an example of a situation when the two tools had issues? I've been using both for years, and don't remember having any problems.
- noodlesUK 5y ago
  Yes. I tried to use Gitpod (a cloud development env platform) which ships with pyenv. Poetry would very stubbornly only use the system python and completely ignore whatever the pyenv shim was pointing to.

hkt 5y ago

Golang is a great model to follow. I think the fundamental thing that python (and ruby, when I still used it) has taught me is to run a mile if a language is without a robust dependency management system. The pain is just not worth it, even for a nice language with an otherwise robust ecosystem.

yxhuvud 5y ago
Huh? The package dependency system in Ruby is a breeze, and has been a solved problem since 2007 or whenever Bundler was made.
noodlesUK 5y ago
With golang, it's not just the dependency management side, gofmt (and various bits of other tooling) is also an incredible blessing. Yes, prescriptivism can sometimes feel weird to us computer nerds, but sometimes it's just much better to have a canonical way of doing common tasks. It means onboarding people is much easier, and infrastructure/code/whatever is more reusable.
goodpoint 5y ago
Go is terrible for distributions to package due to the poor library versioning. Also, static linking makes security updates really expensive to build at scale.

withinboredom 5y ago

I’m not a Python dev. I do need to occasionally run things written in Python. I made the mistake of trying to get pip + Conda + pyenv (or whatever) to install a fairly simple tool. I have no idea how the dev got their setup working, but it was totally and utterly unreproducable, even after they sat down on my computer for several hours. In that amount of time, I could have probably rewritten it in PHP (that’s actually exactly what I ended up doing while they attempted to get a Docker container running). Needless to say, I will only use the distro package manager these days. I know the versions are (probably) compatible, maintainers will usually backport security patches, etc. You get none of that using whatever flavor of the week python package manager.

rolandog 5y ago
I had to learn this by FUBAR'ing my system a long time ago, but my setup process for working with a python package from PyPI (i.e. not installable by or updated enough in OS's package manager) nowadays is: - sudo apt install pip3 - pip3 install --user --upgrade pipenv (In workdir): - pipenv install --three package - pipenv run package --option Works like a charm and doesn't mess with my system.
noodlesUK 5y ago
I think this actually gets to the crux of the matter. The existing python dependency management tools (especially the new shiny ones like poetry) are very much designed by python devs, for python devs. What you're describing is a totally different use case, which is running released software in a sensible way. The distro package managers are probably the best place for that, but bridging the gap between them and the python ecosystem is an obvious challenge.
nerdponx 5y ago
This is especially frustrating to read because one of the main selling points of Conda is reproducibility. In data science teams, I've found it indispensable for making sure people can all run each others' project code. So for anyone reading this in the future: don't try to use Pyenv to install Conda. Pyenv tries to set up shims for every binary in the Conda env, which will likely break your PATH. Pyenv supports installing Conda because Anaconda used to be "just" a Python distribution. They can otherwise coexist without trouble on the same system.

atoav 5y ago

I landed with poetry as well. The issue with python dependency managment in my book is, that it is incredibly (and needlessly) hard to learn how to do it properly. It took me years to figure out how things should work and there are still issues now and then that cost me a few hours to figure out and restore things. Meanwhile when I use Rust all these things are taken care of in cargo. It is part of the language. There is one right way to do things and the way is supported by comfortable tooling, that works so well that you literally don't even consider thinking about anything else. The way python does dependencies is totally unpythonic. The fact that it is 2021 and this isn't fixed or at least the number one priority of things that need fixing sheds a dark shadow on the whole language – a language that I like to use. Poetry is good. But it isn't as good as cargo, because it also has to deal with all the legacy cruft. To run code developed with poetry on a non-poetry system you have to figure out all the ways of dealing with envs, paths and such. Issues like these get me a little fired up, because the collective brainpower wasted on something that should have been elegantly solved in one place is gigantic.

jhrmnn 5y ago
To run code developed with Poetry, you shouldn’t even know it’s developed with Poetry. It should be released as a source distribution or a wheel on PyPI or as a conda package if it contains nontrivial binary extensions. These distributions should be then either packaged by the OS or they should be installed with all their dependencies into a separate virtual environment in /opt.
- atoav 5y ago
  Sure, if I release the code I tend to do just that. However a lot of the code I have been writing was to be used on servers where installing poetry was not an option. That means there was no straightforward, well documented or otherwise easy way to copy the project onto the server an "just have it work". And this was not due to me not taking care of my dev environment. Now I know how to do this, so this is not a problem. My complaint was mostly, that this was a waste of time.
  - 5y ago

BerislavLopac 5y ago

> I like poetry, and I think most people are heading that way, but it doesn't seem to play super nice with pyenv Do you have an example of a situation when the two tools had issues? I've been using both for years, and don't remember having any problems.

noodlesUK 5y ago
Yes. I tried to use Gitpod (a cloud development env platform) which ships with pyenv. Poetry would very stubbornly only use the system python and completely ignore whatever the pyenv shim was pointing to.