3 ms·

As a Python infra person, I would refine that to: If you're building an application, use Poetry. If you're building a library, use Flit. Use PEP621 metadata in

by jreese 5y ago

As a Python infra person, I would refine that to: If you're building an application, use Poetry. If you're building a library, use Flit. Use PEP621 metadata in pyproject.toml regardless. Poetry is much more focused on managing dependencies for applications than dealing with libraries that have to be used by other libraries or applications. See this deep discussion for some timely/relevant examples: https://iscinumpy.dev/post/bound-version-constraints/ https://iscinumpy.dev/post/bound-version-constraints/

kbumsik 5y ago
Can we mix Poetry and PEP621 though? I tried to use project.dependencies and project.optional-dependencies fields in pyproject.toml defined by PEP621. However Poetry seems to ignore it and it tries to stick with its own tools.poetry section.
- jreese 5y ago
  Ah, you're right. I thought Poetry had adopted PEP 621 already, but it looks like there's still a debate about if/how they want to do that. A bit ironic considering the author of Poetry was listed as an author on PEP 621, and that was accepted just over a year ago.
  - kbumsik 5y ago
    It went final this March though. So Poetry might needs more time to think about the transitioning plan. Python packaging is again in a chaos since 2021 because of the rise of standardized PEPs.
    - spurgu 5y ago
      I know you didn't mean it that way, but "since 2021"..? :D I was working actively with Python (and loving it) between 2012-2015 (during which the 2 vs 3 transition/conflicts were raging) and the couple of times I've tried to get back into the groove since then (in side projects) I keep hitting walls and getting confused about how to even get the basic things running. I've heard good things about Poetry but never tried it. PDM being closer to npm/yarn sounds like an attractive option (since they're very familar), just to tinker around (not to build a complex application with specific requirements). Will give it a shot!
eesmith 5y ago
I've seen others make the same reference, but I've not understood it. My package is a Python module and a set of command-line tools based on that module. Is my package a library? Or an application? Because it seems to be "both". It also includes several C extensions, using either the Python/C API manually or Cython. I would love to switch to something more modern than setup.py, but none of these new back-ends seem to support this. For example, I just checked - flit explicitly focuses on "pure Python packages with no build steps" "and leaves the hard things up to other tools".
- jreese 5y ago
  Do you expect other people to add your project as a dependency of their projects, and call into your API? Then it's a library. If the only expected users of your API is your own project, it's an application. And yes, if you need to build native extensions, setuptools is still your only real option. It's not great, and results in hundreds of packages that all find their own definition of insanity. But given just how crazy the C/C++ toolchain ecosystem is, especially when you have to deal with Windows, macOS, and the diversity of Linux, no one else has been will to tackle that problem because setuptools already exists.
  - eesmith 5y ago
    Following that definition, the project I mentioned is a library. But quoting PEP 517 "Much experience suggests that large successful projects often originate as quick hacks". I have another project which started as a command-line tool only, meant for the end-user to build, maintain, search, and update a data set. It was not designed to be called by another program, neither through import nor popen. Following the guideline, I should use poetry. It then acquired a limited Python API so search functionality could be incorporated in a web app. One of my users paid for that feature. I don't know if others use it. This turned my package into a library. Does that mean I should switch from poetry to flit? There seem to be several people who have tackled that problem, including emscons, PyQt-builder, and mesonpep517, which all defer to an existing third-party build system. I don't use or have experience with any of the underlying build systems, and version numbers like 0.28.0 and 0.2 make me wary. Are there others? Finding such modules on PyPI is not easy! A search for "517" is almost useless. Could there be a new trove category for such packages?
saila 5y ago
> Poetry is much more focused on managing dependencies for applications than dealing with libraries The linked article is quite long. Do you have a quick example? I've built lots of apps and libraries using Poetry and haven't run into any dependency issues, and you can specify versions constraints in the same way as Flit, so it's hard to see how they'd be different in that regard. Edit: I think I see what the issue is--Poetry sets upper version bounds by default via its ^X.Y.Z and ~X.Y.Z syntax and advocates for always specifying upper bounds in their docs. So, it's not that Poetry can't be used for a library, but you'll have to know that you can, and perhaps should, use different version syntax in some cases (i.e., the standard >=X.Y syntax).

by jreese 5y ago

kbumsik 5y ago

Can we mix Poetry and PEP621 though? I tried to use project.dependencies and project.optional-dependencies fields in pyproject.toml defined by PEP621. However Poetry seems to ignore it and it tries to stick with its own tools.poetry section.

jreese 5y ago
Ah, you're right. I thought Poetry had adopted PEP 621 already, but it looks like there's still a debate about if/how they want to do that. A bit ironic considering the author of Poetry was listed as an author on PEP 621, and that was accepted just over a year ago.
- kbumsik 5y ago
  It went final this March though. So Poetry might needs more time to think about the transitioning plan. Python packaging is again in a chaos since 2021 because of the rise of standardized PEPs.
  - spurgu 5y ago
    I know you didn't mean it that way, but "since 2021"..? :D I was working actively with Python (and loving it) between 2012-2015 (during which the 2 vs 3 transition/conflicts were raging) and the couple of times I've tried to get back into the groove since then (in side projects) I keep hitting walls and getting confused about how to even get the basic things running. I've heard good things about Poetry but never tried it. PDM being closer to npm/yarn sounds like an attractive option (since they're very familar), just to tinker around (not to build a complex application with specific requirements). Will give it a shot!

eesmith 5y ago

I've seen others make the same reference, but I've not understood it. My package is a Python module and a set of command-line tools based on that module. Is my package a library? Or an application? Because it seems to be "both". It also includes several C extensions, using either the Python/C API manually or Cython. I would love to switch to something more modern than setup.py, but none of these new back-ends seem to support this. For example, I just checked - flit explicitly focuses on "pure Python packages with no build steps" "and leaves the hard things up to other tools".

jreese 5y ago
Do you expect other people to add your project as a dependency of their projects, and call into your API? Then it's a library. If the only expected users of your API is your own project, it's an application. And yes, if you need to build native extensions, setuptools is still your only real option. It's not great, and results in hundreds of packages that all find their own definition of insanity. But given just how crazy the C/C++ toolchain ecosystem is, especially when you have to deal with Windows, macOS, and the diversity of Linux, no one else has been will to tackle that problem because setuptools already exists.
- eesmith 5y ago
  Following that definition, the project I mentioned is a library. But quoting PEP 517 "Much experience suggests that large successful projects often originate as quick hacks". I have another project which started as a command-line tool only, meant for the end-user to build, maintain, search, and update a data set. It was not designed to be called by another program, neither through import nor popen. Following the guideline, I should use poetry. It then acquired a limited Python API so search functionality could be incorporated in a web app. One of my users paid for that feature. I don't know if others use it. This turned my package into a library. Does that mean I should switch from poetry to flit? There seem to be several people who have tackled that problem, including emscons, PyQt-builder, and mesonpep517, which all defer to an existing third-party build system. I don't use or have experience with any of the underlying build systems, and version numbers like 0.28.0 and 0.2 make me wary. Are there others? Finding such modules on PyPI is not easy! A search for "517" is almost useless. Could there be a new trove category for such packages?

saila 5y ago

> Poetry is much more focused on managing dependencies for applications than dealing with libraries The linked article is quite long. Do you have a quick example? I've built lots of apps and libraries using Poetry and haven't run into any dependency issues, and you can specify versions constraints in the same way as Flit, so it's hard to see how they'd be different in that regard. Edit: I think I see what the issue is--Poetry sets upper version bounds by default via its ^X.Y.Z and ~X.Y.Z syntax and advocates for always specifying upper bounds in their docs. So, it's not that Poetry can't be used for a library, but you'll have to know that you can, and perhaps should, use different version syntax in some cases (i.e., the standard >=X.Y syntax).