6 ms·

How does it compare to conda/mamba + env.yaml?

by jmakov 5y ago

prpl 5y ago
Likely inferior since it’s python only. You can get pretty far with python first but in most sciences that’s not going to get you all sorts of community standard things that are C/C++/fortran
- mwt 5y ago
  Likely even a little bit more narrow than Python-only since it seems to be Poetry-only. Most Python projects probably work okay with Poetry, but it's still a relatively new tool compared to the other options.
  - travisd 5y ago
    I don’t think this is much of a concern here. Poetry is still not super compatible with a bunch of tooling, but that only matters when consuming Poetry projects. When you just install a dependency into a poetry project, it gets installed into a venv - nothing particularly exotic about it.
    - JPKab 5y ago
      Correct. They are commenting on Poetry without knowing anything about it, other than it being new and unfamiliar. Always amazed at the arrogance behind that behavior.
    - physicsguy 5y ago
      It matters a lot, because C, C++ and Fortran dependencies make up a large number of Python package dependencies. It’s not really a Poetry problem though, but a PyPi problem. Wheels improved the situation but aren’t a panacea.
  - JPKab 5y ago
    "Likely": Translation: I've never used poetry, don't know anything about it, and I'm going to talk out of my rear end and assume that it doesn't use conventional, existing venv/pip under the hood. There isn't a single python project out there that can't be worked on with poetry. It creates a venv. Poetry is doing a lot of things to manage automatically activating and deactivating it, etc, but it's just a venv.
    - Mehdi2277 5y ago
      False. Poetry implements it’s own dependency resolver. It does not use pip’s resolver. For most packages this is fine. For a couple packages it’s resolver does not work. Pytorch is one major ml library that is incompatible with poetry as poetry has a non-standard interpretation of local package version tags. https://github.com/python-poetry/poetry/issues/4231 https://github.com/python-poetry/poetry/issues/4231 Is one issue. If you even transitively depend on pytorch poetry will break and you will need hacks to make it work or just give up.
      - fault1 5y ago
        yes, alas major exceptions like pytorch make this old xkcd still ring true today: https://xkcd.com/1987/ https://xkcd.com/1987/ also, a lot of python packages don't seem to follow the idea of semvar very well. Especially when you compare it to other communities like Go.
        MaxBarraclough 5y ago
        The xkcd hover-over text sounds almost like it's describing containers: > The Python environmental protection agency wants to seal it in a cement chamber [...]
    - misnome 5y ago
      ISTR poetry really didn't like the black "eternal beta" versioning scheme for a while and installing it didn't work properly. Also, it really doesn't work super well if you need to work on multiple development projects simultaneously, and when "just change the way you structure your entire project" is poetry's answer to this, the solution is "okay we won't use poetry".
- JPKab 5y ago
  Huh? Have you ever used poetry before? Are you implying that it can't be used to install Python dependencies that have c/fortran like pandas? If that's what you're saying that is not correct.
  - fault1 5y ago
    I think he was saying that conda/mamba are much more language agnostic than poetry. It handles a lot of packages that have /nothing/ to do with python, and it handles them a reasonable way. Poetry does seem much more tied to python (for better or worse). Keep in mind at least in mamba's case, not being tied to the python interpreter is very much a feature, not a bug: https://medium.com/@QuantStack/open-software-packaging-for-science-61cecee7fc23 https://medium.com/@QuantStack/open-software-packaging-for-s...
    - mark_l_watson 5y ago
      I agree. I know people who avoid conda because of long install/update times, but I am more than willing to occasionally have to go outside for a walk while packages I rely on get properly managed and installed. BTW, sorry for being off topic, but Lex Fridman has two great recent long form interviews with Anaconda founders Peter Wang and Travis Oliphant. Good discussions!
      - misnome 5y ago
        We have found that mamba (fast alternate front end instead of conda) and micromamba entirely fixes this issue for us. IMO “conda install” is entirely unfit for purpose and probably puts lots of people off of the ecosystem (15 minutes for megabytes of error messages often in the case of unresolved dependency trees). Our CI was often spending more time running conda than running our test suite. I assume there is some political difference or acrimony which means the actual sources about core conda are entirely silent on the issue - I’ve privately replied to people expressing concern over this on the conda mailing list several times and people are always relieved to know there is a solution. Seriously, it’s the only way we can tolerate using conda as an ecosystem any more, and only ever feel this pain of 15-20 minute feedback cycles preparing packages with conda-build (for which boa is a promising new replacement we haven’t moved to yet)

How does it compare to conda/mamba + env.yaml?

by jmakov 5y ago

How does it compare to conda/mamba + env.yaml?

prpl 5y ago

Likely inferior since it’s python only. You can get pretty far with python first but in most sciences that’s not going to get you all sorts of community standard things that are C/C++/fortran

mwt 5y ago
Likely even a little bit more narrow than Python-only since it seems to be Poetry-only. Most Python projects probably work okay with Poetry, but it's still a relatively new tool compared to the other options.
- travisd 5y ago
  I don’t think this is much of a concern here. Poetry is still not super compatible with a bunch of tooling, but that only matters when consuming Poetry projects. When you just install a dependency into a poetry project, it gets installed into a venv - nothing particularly exotic about it.
  - JPKab 5y ago
    Correct. They are commenting on Poetry without knowing anything about it, other than it being new and unfamiliar. Always amazed at the arrogance behind that behavior.
  - physicsguy 5y ago
    It matters a lot, because C, C++ and Fortran dependencies make up a large number of Python package dependencies. It’s not really a Poetry problem though, but a PyPi problem. Wheels improved the situation but aren’t a panacea.
- JPKab 5y ago
  "Likely": Translation: I've never used poetry, don't know anything about it, and I'm going to talk out of my rear end and assume that it doesn't use conventional, existing venv/pip under the hood. There isn't a single python project out there that can't be worked on with poetry. It creates a venv. Poetry is doing a lot of things to manage automatically activating and deactivating it, etc, but it's just a venv.
  - Mehdi2277 5y ago
    False. Poetry implements it’s own dependency resolver. It does not use pip’s resolver. For most packages this is fine. For a couple packages it’s resolver does not work. Pytorch is one major ml library that is incompatible with poetry as poetry has a non-standard interpretation of local package version tags. https://github.com/python-poetry/poetry/issues/4231 https://github.com/python-poetry/poetry/issues/4231 Is one issue. If you even transitively depend on pytorch poetry will break and you will need hacks to make it work or just give up.
    - fault1 5y ago
      yes, alas major exceptions like pytorch make this old xkcd still ring true today: https://xkcd.com/1987/ https://xkcd.com/1987/ also, a lot of python packages don't seem to follow the idea of semvar very well. Especially when you compare it to other communities like Go.
      - MaxBarraclough 5y ago
        The xkcd hover-over text sounds almost like it's describing containers: > The Python environmental protection agency wants to seal it in a cement chamber [...]
  - misnome 5y ago
    ISTR poetry really didn't like the black "eternal beta" versioning scheme for a while and installing it didn't work properly. Also, it really doesn't work super well if you need to work on multiple development projects simultaneously, and when "just change the way you structure your entire project" is poetry's answer to this, the solution is "okay we won't use poetry".
JPKab 5y ago
Huh? Have you ever used poetry before? Are you implying that it can't be used to install Python dependencies that have c/fortran like pandas? If that's what you're saying that is not correct.
- fault1 5y ago
  I think he was saying that conda/mamba are much more language agnostic than poetry. It handles a lot of packages that have /nothing/ to do with python, and it handles them a reasonable way. Poetry does seem much more tied to python (for better or worse). Keep in mind at least in mamba's case, not being tied to the python interpreter is very much a feature, not a bug: https://medium.com/@QuantStack/open-software-packaging-for-science-61cecee7fc23 https://medium.com/@QuantStack/open-software-packaging-for-s...
  - mark_l_watson 5y ago
    I agree. I know people who avoid conda because of long install/update times, but I am more than willing to occasionally have to go outside for a walk while packages I rely on get properly managed and installed. BTW, sorry for being off topic, but Lex Fridman has two great recent long form interviews with Anaconda founders Peter Wang and Travis Oliphant. Good discussions!
    - misnome 5y ago
      We have found that mamba (fast alternate front end instead of conda) and micromamba entirely fixes this issue for us. IMO “conda install” is entirely unfit for purpose and probably puts lots of people off of the ecosystem (15 minutes for megabytes of error messages often in the case of unresolved dependency trees). Our CI was often spending more time running conda than running our test suite. I assume there is some political difference or acrimony which means the actual sources about core conda are entirely silent on the issue - I’ve privately replied to people expressing concern over this on the conda mailing list several times and people are always relieved to know there is a solution. Seriously, it’s the only way we can tolerate using conda as an ecosystem any more, and only ever feel this pain of 15-20 minute feedback cycles preparing packages with conda-build (for which boa is a promising new replacement we haven’t moved to yet)