3 ms·

Ok, thank you. It’s a pity for me as I think having a resource that explain the big picture of both would be great. But I don’t have enough of a big picture to

Ok, thank you. It’s a pity for me as I think having a resource that explain the big picture of both would be great. But I don’t have enough of a big picture to write about it yet. The thing about conda is that it is not just a tool, it is a completely different ecosystem to handle what you describe in your post. You touched on the topic of building & packaging, distribution, environment management, etc. And all these are different in conda land. To me the answer to the problems you mentioned in the post is conda. But I think it is commonly misunderstood and some people hate to use it. Not to mention needing to maintain publishing in both PyPI and a conda channel is laborious.

zahlman 2y ago
>it is a completely different ecosystem... all these are different in conda land. This matches my understanding. But I also understand that Conda doesn't treat Python as anything special; it just provides its own kind of environments for multi-language computing. Many people do swear by Conda and I should look into it eventually. (In some far future, maybe Paper can work in a Conda environment and avoid the problems that Pip encounters there.) If I'm going to fix problems (and educate about them) on the Python-first side, I need to have priorities, though.
- KolenCh 2y ago
  There’s a chain of stuffs happening regarding packaging and the first step in the chain using PyPI limits what one can do to fix it. Conda starts with a different design in how to build a package and also how to bundle it. It specifically handles compiled code much better, because in that non pure Python case, compiler is needed at some point, and that is going to cause problems if you have no control over it. Conda essentially achieve reproducibility, with the necessary compiler considered as part of the dependencies. While in a certain sense of reproducibility can be achieved with PyPI packages, my understanding is that, eg with manylinux, it takes a hope for the best approach and can break in exotic cases, which should never happen with conda packages. > The genesis of Conda came after Guido van Rossum was invited to speak at the inaugural PyData meetup in 2012; in a Q&A on the subject of packaging difficulties, he told us that when it comes to packaging, "it really sounds like your needs are so unusual compared to the larger Python community that you're just better off building your own" From https://jakevdp.github.io/blog/2016/08/25/conda-myths-and-misconceptions/ https://jakevdp.github.io/blog/2016/08/25/conda-myths-and-mi...
  - zahlman 2y ago
    > From [link] This was already in my research pile. "Myth #2" there is what I meant about "not treating Python as anything special". The GvR quote is infamous, and seen as short-sighted, but it made sense in context. The story is often told (though the link doesn't do this) as if the "main" Python community intended to ostracize the PyData folks. But 12 years later there is still much important dialog with them, and NumPy is one of the most commonly downloaded PyPI packages. My discussions on discuss.python.org exposed me to entire websites of analysis of the existing problems (like https://pypackaging-native.github.io/ https://pypackaging-native.github.io/) and people like Henry Schreiner and Ralf Gommers are driving much of the discussion. > in that non pure Python case, compiler is needed at some point, and that is going to cause problems if you have no control over it. Conda essentially achieve reproducibility, with the necessary compiler considered as part of the dependencies. This is one of the major differences, yes: Conda has a system for ensuring you have the right compiler. It's hoped that PEP 725 "Specifying external dependencies in pyproject.toml" (https://peps.python.org/pep-0725/ https://peps.python.org/pep-0725/) will be a first step towards resolving this in the Pip world, or perhaps towards better interoperation with Conda. (Of course, the hard part is actually using that metadata to install compilers.)
    - KolenCh 2y ago
      Interesting. I’m looking forward to your future posts. One question I have in mind when dealing with PyPI vs conda is that why wouldn’t conda already solved all the problems with PyPI. My own answers to that includes 1) PyPI is still the de facto place to claim a namespace and it would be foolish to only release it in conda and not PyPI, and 2) there are many more packages available in PyPI and not conda, and the effort to just make all PyPI packages I used available in PyPI is too laborious. Another issue is that many packages available on PyPI are not packaged well, from version constraints to setup.py madness to licensing issues. People managing the conda-forge channel is more gate keeping about those issues. I wonder even if the tooling around PyPI is improved, the community culture is much more difficult to be changed.

zahlman 2y ago

>it is a completely different ecosystem... all these are different in conda land. This matches my understanding. But I also understand that Conda doesn't treat Python as anything special; it just provides its own kind of environments for multi-language computing. Many people do swear by Conda and I should look into it eventually. (In some far future, maybe Paper can work in a Conda environment and avoid the problems that Pip encounters there.) If I'm going to fix problems (and educate about them) on the Python-first side, I need to have priorities, though.

KolenCh 2y ago
There’s a chain of stuffs happening regarding packaging and the first step in the chain using PyPI limits what one can do to fix it. Conda starts with a different design in how to build a package and also how to bundle it. It specifically handles compiled code much better, because in that non pure Python case, compiler is needed at some point, and that is going to cause problems if you have no control over it. Conda essentially achieve reproducibility, with the necessary compiler considered as part of the dependencies. While in a certain sense of reproducibility can be achieved with PyPI packages, my understanding is that, eg with manylinux, it takes a hope for the best approach and can break in exotic cases, which should never happen with conda packages. > The genesis of Conda came after Guido van Rossum was invited to speak at the inaugural PyData meetup in 2012; in a Q&A on the subject of packaging difficulties, he told us that when it comes to packaging, "it really sounds like your needs are so unusual compared to the larger Python community that you're just better off building your own" From https://jakevdp.github.io/blog/2016/08/25/conda-myths-and-misconceptions/ https://jakevdp.github.io/blog/2016/08/25/conda-myths-and-mi...
- zahlman 2y ago
  > From [link] This was already in my research pile. "Myth #2" there is what I meant about "not treating Python as anything special". The GvR quote is infamous, and seen as short-sighted, but it made sense in context. The story is often told (though the link doesn't do this) as if the "main" Python community intended to ostracize the PyData folks. But 12 years later there is still much important dialog with them, and NumPy is one of the most commonly downloaded PyPI packages. My discussions on discuss.python.org exposed me to entire websites of analysis of the existing problems (like https://pypackaging-native.github.io/ https://pypackaging-native.github.io/) and people like Henry Schreiner and Ralf Gommers are driving much of the discussion. > in that non pure Python case, compiler is needed at some point, and that is going to cause problems if you have no control over it. Conda essentially achieve reproducibility, with the necessary compiler considered as part of the dependencies. This is one of the major differences, yes: Conda has a system for ensuring you have the right compiler. It's hoped that PEP 725 "Specifying external dependencies in pyproject.toml" (https://peps.python.org/pep-0725/ https://peps.python.org/pep-0725/) will be a first step towards resolving this in the Pip world, or perhaps towards better interoperation with Conda. (Of course, the hard part is actually using that metadata to install compilers.)
  - KolenCh 2y ago
    Interesting. I’m looking forward to your future posts. One question I have in mind when dealing with PyPI vs conda is that why wouldn’t conda already solved all the problems with PyPI. My own answers to that includes 1) PyPI is still the de facto place to claim a namespace and it would be foolish to only release it in conda and not PyPI, and 2) there are many more packages available in PyPI and not conda, and the effort to just make all PyPI packages I used available in PyPI is too laborious. Another issue is that many packages available on PyPI are not packaged well, from version constraints to setup.py madness to licensing issues. People managing the conda-forge channel is more gate keeping about those issues. I wonder even if the tooling around PyPI is improved, the community culture is much more difficult to be changed.