3 ms·

There’s a chain of stuffs happening regarding packaging and the first step in the chain using PyPI limits what one can do to fix it. Conda starts with a differe

There’s a chain of stuffs happening regarding packaging and the first step in the chain using PyPI limits what one can do to fix it. Conda starts with a different design in how to build a package and also how to bundle it. It specifically handles compiled code much better, because in that non pure Python case, compiler is needed at some point, and that is going to cause problems if you have no control over it. Conda essentially achieve reproducibility, with the necessary compiler considered as part of the dependencies. While in a certain sense of reproducibility can be achieved with PyPI packages, my understanding is that, eg with manylinux, it takes a hope for the best approach and can break in exotic cases, which should never happen with conda packages. > The genesis of Conda came after Guido van Rossum was invited to speak at the inaugural PyData meetup in 2012; in a Q&A on the subject of packaging difficulties, he told us that when it comes to packaging, "it really sounds like your needs are so unusual compared to the larger Python community that you're just better off building your own" From https://jakevdp.github.io/blog/2016/08/25/conda-myths-and-misconceptions/ https://jakevdp.github.io/blog/2016/08/25/conda-myths-and-mi...

zahlman 2y ago
> From [link] This was already in my research pile. "Myth #2" there is what I meant about "not treating Python as anything special". The GvR quote is infamous, and seen as short-sighted, but it made sense in context. The story is often told (though the link doesn't do this) as if the "main" Python community intended to ostracize the PyData folks. But 12 years later there is still much important dialog with them, and NumPy is one of the most commonly downloaded PyPI packages. My discussions on discuss.python.org exposed me to entire websites of analysis of the existing problems (like https://pypackaging-native.github.io/ https://pypackaging-native.github.io/) and people like Henry Schreiner and Ralf Gommers are driving much of the discussion. > in that non pure Python case, compiler is needed at some point, and that is going to cause problems if you have no control over it. Conda essentially achieve reproducibility, with the necessary compiler considered as part of the dependencies. This is one of the major differences, yes: Conda has a system for ensuring you have the right compiler. It's hoped that PEP 725 "Specifying external dependencies in pyproject.toml" (https://peps.python.org/pep-0725/ https://peps.python.org/pep-0725/) will be a first step towards resolving this in the Pip world, or perhaps towards better interoperation with Conda. (Of course, the hard part is actually using that metadata to install compilers.)
- KolenCh 2y ago
  Interesting. I’m looking forward to your future posts. One question I have in mind when dealing with PyPI vs conda is that why wouldn’t conda already solved all the problems with PyPI. My own answers to that includes 1) PyPI is still the de facto place to claim a namespace and it would be foolish to only release it in conda and not PyPI, and 2) there are many more packages available in PyPI and not conda, and the effort to just make all PyPI packages I used available in PyPI is too laborious. Another issue is that many packages available on PyPI are not packaged well, from version constraints to setup.py madness to licensing issues. People managing the conda-forge channel is more gate keeping about those issues. I wonder even if the tooling around PyPI is improved, the community culture is much more difficult to be changed.

zahlman 2y ago

> From [link] This was already in my research pile. "Myth #2" there is what I meant about "not treating Python as anything special". The GvR quote is infamous, and seen as short-sighted, but it made sense in context. The story is often told (though the link doesn't do this) as if the "main" Python community intended to ostracize the PyData folks. But 12 years later there is still much important dialog with them, and NumPy is one of the most commonly downloaded PyPI packages. My discussions on discuss.python.org exposed me to entire websites of analysis of the existing problems (like https://pypackaging-native.github.io/ https://pypackaging-native.github.io/) and people like Henry Schreiner and Ralf Gommers are driving much of the discussion. > in that non pure Python case, compiler is needed at some point, and that is going to cause problems if you have no control over it. Conda essentially achieve reproducibility, with the necessary compiler considered as part of the dependencies. This is one of the major differences, yes: Conda has a system for ensuring you have the right compiler. It's hoped that PEP 725 "Specifying external dependencies in pyproject.toml" (https://peps.python.org/pep-0725/ https://peps.python.org/pep-0725/) will be a first step towards resolving this in the Pip world, or perhaps towards better interoperation with Conda. (Of course, the hard part is actually using that metadata to install compilers.)

KolenCh 2y ago
Interesting. I’m looking forward to your future posts. One question I have in mind when dealing with PyPI vs conda is that why wouldn’t conda already solved all the problems with PyPI. My own answers to that includes 1) PyPI is still the de facto place to claim a namespace and it would be foolish to only release it in conda and not PyPI, and 2) there are many more packages available in PyPI and not conda, and the effort to just make all PyPI packages I used available in PyPI is too laborious. Another issue is that many packages available on PyPI are not packaged well, from version constraints to setup.py madness to licensing issues. People managing the conda-forge channel is more gate keeping about those issues. I wonder even if the tooling around PyPI is improved, the community culture is much more difficult to be changed.