5 ms·

It's starting to look like the huge embarrassment that is python packaging and environment management. I work with the language every day, and I still have no i

It's starting to look like the huge embarrassment that is python packaging and environment management. I work with the language every day, and I still have no idea what am I supposed to use between conda, pip, setuptools/setup.py, pyproject, meta.yml, poetry, pipenv, etc... Conda is so slow that I sometimes wonder if we are being trolled by some cruel God of programming. Pip is faster, but version resolution is iffy. Hell, even just assigning versions to python packages is nothing short of ridiculous. Do you use version.txt in the root folder and set it manually? Do you have it set from SCM? Which of the half a dozen packages do you use to have it set from SCM? setuptools_scm? Versioneer? There are a set of tools in the python ecosystem that have basically no equal in any other language and these tools and the surrounding mindshare make python irreplaceable in the near term. The language itself is easy to learn and powerful enough to be able to do data analysis with ease. Good python code is easy on the eyes, which I personally consider an important aspect. Outside of these tools, core parts of the ecosystem are basically an XKCD joke. I tried switching to Julia as I find both the language and the ecosystem are vastly superior in their foundations. Unfortunately the maturity is not there yet, and neither is the mindshare. If I had to bet my career on adopting the language in a business setting, I'd not be prepared to do so. Which is a shame, because the situation turns into a Catch-22.

alexgrover 5y ago
Starting to? The Python ecosystem has been pretty behind other languages for more like 10 years now. I'm still forced to use it for work but it's becoming more and more likely that "no python" will be a hard requirement for my next job search... What tools do you find irreplaceable?
Demiurge 5y ago
There are actually clear answers to all your questions. All the tools you listed do different things, except maybe poetry and pipenv, so you can pick whichever one you like, you're not supposed to do anything. You can have choice, illusion of free will, etc... As to module version, there is a standard on how to define it in __version__: https://www.python.org/dev/peps/pep-0008/#module-level-dunder-names https://www.python.org/dev/peps/pep-0008/#module-level-dunde...
- whimsicalism 5y ago
  > There are actually clear answers to all your questions. Strongly disagree. Conda, pipenv, pyenv, venv, poetry are all trying to solve the same problem (although conda tries to solve some other problems too). Choice is not always good, this is why we have standards. I would recommend pyenv. I understand why people are attracted to poetry, but pyenv arguably offers all of the same benefits that poetry has as well, and has better adoption.
  - joshuamorton 5y ago
    > Strongly disagree. Conda, pipenv, pyenv, venv, poetry are all trying to solve the same problem (although conda tries to solve some other problems too). Not really, some of these manage the issue of multiple system pythons (pyenv), while some manage isolated envs for particular projects (venv), and some try to be wholistic python project and dependency managers (pipenv, poetry, arguably venv + pip freeze, but that's not "wholistic"). Conda sort of tries to be all of the above as well as a bunch of other things (high performance options etc.)
    - whimsicalism 5y ago
      Pyenv also has isolated envs for specific projects, poetry also tries to solve that problem. There is a ton of overlap, denying that is disingenuous.
      - ErikBjare 5y ago
        While there is overlap, there's also a ton of differences (pyproject for dependencies, dependency resolution, version locks). I use both pyenv and poetry, and basically nothing else apart from the occasional call to pip. I cannot imagine dropping poetry and only use pyenv, but I can imagine dropping pyenv (have been looking at asdf recently...)
lacker 5y ago
I feel like the problem is that pip is clearly the most popular, but philosophically the pip developers do not want to build a solution to all packaging and environment management into pip. If I were pip dictator, I would try to make pip the one tool to handle all python packaging and environment management. In particular, that means pip would handle the management of different Python versions, different Python environments, native dependencies, running tests, making builds perfectly reproducible, releasing new versions of libraries, and creating new projects. "Creating new projects" seems like it is not a big deal, but in practice I think that if there were simply a "pip new" command that set up a new project using the best practices advocated by the pip team, it would go a long way toward standardizing the ecosystem here.
- scrollaway 5y ago
  You're kind of describing poetry. I think Python has a real future with solved packaging if Poetry can get the backing it needs at the language level.
N1H1L 5y ago
Try mamba instead of conda [0]. It's really fast. The first time you have to run it as: conda install -c conda-forge mamba From then on, you replace conda with mamba. For example, if you are installing dask-cuda from the rapidsai channel you run it as: mamba install -c rapidsai dask-cuda At this point, mamba is just so much better and faster, that it's the first package I install in an Anaconda environment. [0] https://github.com/mamba-org/mamba https://github.com/mamba-org/mamba
- DickieStarshine 5y ago
  That's exactly the problem: instead of using conda you should instead use one of the myriad of others! It's balkanisation.
- short_sells_poo 5y ago
  Unfortunately mamba doesn't support working behind proxies (so it's a non-starter in most corporate environments) and it's anecdotal but I've had it fail and crash much more so than conda does. It is faster indeed, but there's the other aspect as another poster points out: there's now Mamba and Conda, so instead of having one standard, we now have two. Plus, mamba can't be used for creating and managing environments, so you still have to do 'coda env create ...' for example.
mixmastamyk 5y ago
Just use setup.py and __version__, ignore fashion until a winner emerges. Hasn’t failed me in ~twenty years.
- short_sells_poo 5y ago
  This doesn't allow you to create conda packages, so you have to install things via pip. This can then cause headaches as you now have two mechanisms by which package versions are managed. Or you give up conda entirely and the scientific python ecosystem that comes with it.
  - mixmastamyk 5y ago
    Yes, most libraries have wheels, many others can be compiled, or installed from packages online.

alexgrover 5y ago

Starting to? The Python ecosystem has been pretty behind other languages for more like 10 years now. I'm still forced to use it for work but it's becoming more and more likely that "no python" will be a hard requirement for my next job search... What tools do you find irreplaceable?

Demiurge 5y ago

There are actually clear answers to all your questions. All the tools you listed do different things, except maybe poetry and pipenv, so you can pick whichever one you like, you're not supposed to do anything. You can have choice, illusion of free will, etc... As to module version, there is a standard on how to define it in __version__: https://www.python.org/dev/peps/pep-0008/#module-level-dunder-names https://www.python.org/dev/peps/pep-0008/#module-level-dunde...

whimsicalism 5y ago
> There are actually clear answers to all your questions. Strongly disagree. Conda, pipenv, pyenv, venv, poetry are all trying to solve the same problem (although conda tries to solve some other problems too). Choice is not always good, this is why we have standards. I would recommend pyenv. I understand why people are attracted to poetry, but pyenv arguably offers all of the same benefits that poetry has as well, and has better adoption.
- joshuamorton 5y ago
  > Strongly disagree. Conda, pipenv, pyenv, venv, poetry are all trying to solve the same problem (although conda tries to solve some other problems too). Not really, some of these manage the issue of multiple system pythons (pyenv), while some manage isolated envs for particular projects (venv), and some try to be wholistic python project and dependency managers (pipenv, poetry, arguably venv + pip freeze, but that's not "wholistic"). Conda sort of tries to be all of the above as well as a bunch of other things (high performance options etc.)
  - whimsicalism 5y ago
    Pyenv also has isolated envs for specific projects, poetry also tries to solve that problem. There is a ton of overlap, denying that is disingenuous.
    - ErikBjare 5y ago
      While there is overlap, there's also a ton of differences (pyproject for dependencies, dependency resolution, version locks). I use both pyenv and poetry, and basically nothing else apart from the occasional call to pip. I cannot imagine dropping poetry and only use pyenv, but I can imagine dropping pyenv (have been looking at asdf recently...)

lacker 5y ago

I feel like the problem is that pip is clearly the most popular, but philosophically the pip developers do not want to build a solution to all packaging and environment management into pip. If I were pip dictator, I would try to make pip the one tool to handle all python packaging and environment management. In particular, that means pip would handle the management of different Python versions, different Python environments, native dependencies, running tests, making builds perfectly reproducible, releasing new versions of libraries, and creating new projects. "Creating new projects" seems like it is not a big deal, but in practice I think that if there were simply a "pip new" command that set up a new project using the best practices advocated by the pip team, it would go a long way toward standardizing the ecosystem here.

scrollaway 5y ago
You're kind of describing poetry. I think Python has a real future with solved packaging if Poetry can get the backing it needs at the language level.

N1H1L 5y ago

Try mamba instead of conda [0]. It's really fast. The first time you have to run it as: conda install -c conda-forge mamba From then on, you replace conda with mamba. For example, if you are installing dask-cuda from the rapidsai channel you run it as: mamba install -c rapidsai dask-cuda At this point, mamba is just so much better and faster, that it's the first package I install in an Anaconda environment. [0] https://github.com/mamba-org/mamba https://github.com/mamba-org/mamba

DickieStarshine 5y ago
That's exactly the problem: instead of using conda you should instead use one of the myriad of others! It's balkanisation.
short_sells_poo 5y ago
Unfortunately mamba doesn't support working behind proxies (so it's a non-starter in most corporate environments) and it's anecdotal but I've had it fail and crash much more so than conda does. It is faster indeed, but there's the other aspect as another poster points out: there's now Mamba and Conda, so instead of having one standard, we now have two. Plus, mamba can't be used for creating and managing environments, so you still have to do 'coda env create ...' for example.

mixmastamyk 5y ago

Just use setup.py and __version__, ignore fashion until a winner emerges. Hasn’t failed me in ~twenty years.

short_sells_poo 5y ago
This doesn't allow you to create conda packages, so you have to install things via pip. This can then cause headaches as you now have two mechanisms by which package versions are managed. Or you give up conda entirely and the scientific python ecosystem that comes with it.
- mixmastamyk 5y ago
  Yes, most libraries have wheels, many others can be compiled, or installed from packages online.