9 ms·

Underappreciated challenges with Python packaging

120 points by groodt 4y ago

CJefferson 4y ago
I've been planning on packaging a python package recently, and the internet is annoyingly full of guides which are, I think, out of date. They at least suggest quite different things. I just have a single python file, meant to be treated as an executable (no package at present). There are a whole bunch of tests, but that's obviously separate. Any suggestions on modern best practices welcome!
- woodruffw 4y ago
  If it's pure Python, the only packaging file you need is `pyproject.toml`. You can fill that file with packaging metadata per PEP 518 and PEP 621, including using modern build tooling like flit[1] for the build backend and build[2] for the frontend. With that, you entire package build (for all distribution types) should be reducible to `python -m build`. Here's an example of a full project doing everything with just `pyproject.toml`[3] (FD: my project). [1]: https://github.com/pypa/flit https://github.com/pypa/flit [2]: https://github.com/pypa/build https://github.com/pypa/build [3]: https://github.com/pypa/pip-audit https://github.com/pypa/pip-audit
  - quietbritishjim 4y ago
    > including using modern build tooling like flit[1] for the build backend and build[2] for the frontend. Or you can use setuptools, which is the package that enables old setup.py builds, as the backend with pyproject.toml. This has the advantage of being mature, unlikely to be abandoned, and possibly some familiarity if you've used it before. Even then, you can use build as the front end build tool.
    - woodruffw 4y ago
      Yes, setuptools is also perfectly fine. It had some rough edges around PEP 621 support for a while, but those were mostly smoothed out in 2021. (I'll note that maturity is not strong evidence here: distutils is very mature, but is in the process of being deprecated and removed from Python entirely. I don't think that's likely to happen to setuptools, but the fact that behavioral PEPs now exist for all of these tools means that the decline/abandonment of any poses much less of an ecosystem risk.)
      - quietbritishjim 4y ago
        I suppose I meant maturity as in: actively maintained and recommended for use for a long period, which doesn't apply to distutils. And I should've been more upfront about the real reason for suggesting setuptools: there seem to be a number of build tools that support pyproject.toml, including flit, poetry and setuptools (and I'm sure I've seen at least one other). For me, at least, when I was making a small library recently, it was an overwhelming choice for a part of my project that feels like just admin rather than core business logic. I came close to giving up and just using setup.py with `setup()`. At least setuptools with pyproject.toml is a choice that feels safe; it may not be the best, but it will certainly be good enough that I'm unlikely to regret it later, so I didn't need to spend a lot of time looking at the detailed pros and cons of all the choices.
        woodruffw 4y ago
        That's very reasonable! I don't mean to disparage that decision at all: setuptools is rock solid and a very safe choice.
  - kevin_thibedeau 4y ago
    Good luck finding clear documentation on how to write the pyproject file. For something that is supposed to be the way forward there hasn't been much effort to make it easy to implement. There's also annoyances like the inability to install a script in the search path without implementing it as a module. Something setup.py doesn't require.
    - woodruffw 4y ago
      > Good luck finding clear documentation on how to write the pyproject file. For something that is supposed to be the way forward there hasn't been much effort to make it easy to implement. PEP 621, which I mentioned, covers the format of `pyproject.toml` in detail. I also linked an example which, to the best of my knowledge, covers all current best practices for that file. > There's also annoyances like the inability to install a script in the search path without implementing it as a module. Something setup.py doesn't require. I'm not sure I'm following. A module in Python is just a Python file, so your script is a module. Are you saying that you can't distribute single-module packages with pyproject.toml? Because I don't think that's true.
      - kevin_thibedeau 4y ago
        The PEP in no way explains how to write a usable pyproject for ordinary projects. It's basically just targeted at people developing installers. I meant package. A directory with an __init__.py. You can't install standalone script.py (or a generated wrapper) as /usr/local/bin/script with a pyproject.
        woodruffw 4y ago
        > The PEP in no way explains how to write a usable pyproject for ordinary projects. It's basically just targeted at people developing installers. Did you look at it[1]? > I meant package. A directory with an __init__.py. You can't install standalone script.py (or a generated wrapper) as /usr/local/bin/script with a pyproject. I still don't think I understand what your expectation is here: a `pyproject.toml` is just a metadata specification. The only difference between it and `setup.py` is that the latter is arbitrary code. There's an old, long deprecated way to use `setup.py`, namely `setup.py install`. But that's been discouraged in favor of `pip install` for years, which behaves precisely the same way with `pyproject.toml`. If you want to install a script into `/usr/local/bin`, `pip install` with a package specified in `pyproject.toml` will work just fine. [1]: https://peps.python.org/pep-0621/#example https://peps.python.org/pep-0621/#example
- ensignavenger 4y ago
  If you are wanting to release it to pypi as a python package, I would personally use Poetry. But your case- a single pure Python package, is a simple case that won't have many problems like are brought up in the article, whatever tool you use. If you want a stand alone executable, I haven't found a good, single, cross platform tool for that yet... seems like there is a separate tool for each platform.
  - quietbritishjim 4y ago
    Nuitka works on Windows, Linux and Mac https://nuitka.net/ https://nuitka.net/
  - selcuka 4y ago
    > If you want a stand alone executable, I haven't found a good, single, cross platform tool for that yet. PyInstaller is cross platform, and arguably good.
- mixmastamyk 4y ago
  Keep it simple and fashion-proof. Been using setup.py for a one-script package for one or two? decades: from setuptools import setup setup( name = 'foobar', scripts = ['foo'], # install a script from current fldr # ... ) A few years ago I had to start using twine to register and upload it to pypi.
  - tomrod 4y ago
    Is there a place you recommend where I can learn more about this?
    - mixmastamyk 4y ago
      It’s documented, try google, stackoverflow, and reading other packages’ setup.py for tricks. However the scripts=[], keyword is the key to the case above.
  - talideon 4y ago
    I wouldn't recommend that at all these days. setup.py definitely is not future-proof: it's undergoing deprecation. Better would be a basic pyproject.toml file along the lines of the following: [build-system] requires = ["setuptools"] build-backend = "setuptools.build_meta" [project] name = "foobar" version = "0.0.1" dependencies = [ "...", ] [project.scripts] foo = "foobar:main" See: https://setuptools.pypa.io/en/latest/userguide/quickstart.html https://setuptools.pypa.io/en/latest/userguide/quickstart.ht...
    - eesmith 4y ago
      > it's undergoing deprecation In the "we would rather people not use this but it's going to stay around for a long time" sense. I strongly doubt it will disappear within the next decade or two. There's a long tail of setup.py-based tools. Last I checked pyproject.toml only supports the simplest of Python/C extensions. Anything fancy, like --with/--without compilation flags to enable/disable optional support, compiler-specific flags (in my case, to add OpenMP), compile-time code generation (like using yacc/lex), etc. requires a setup.py and a bunch of hacking.
- japanuspus 4y ago
  As detailed in the other answers, there are two parts to this: 1) Creating a python package from your project (and possibly share this on pypi), and 2) Making this package available as an end-user application. For step 2 you can use nuitka or similar, but if your audience is somewhat developer-oriented, you can also propose for them to use pipx: https://github.com/pypa/pipx https://github.com/pypa/pipx.
- tpoacher 4y ago
  The approach I prefer is to not mess with setuptools etc at all in the first place, and simply make a nice executable package. e.g. https://github.com/tpapastylianou/self-contained-runnable-python-package-template https://github.com/tpapastylianou/self-contained-runnable-py...
miohtama 4y ago
Python joke: Sdist is only one letter away from sadist.
irskep 4y ago
This is an incredible example of organizing information well and making a case to a wide audience. It's difficult enough to shave all the yaks necessary to get a high-level view of all issues related to a problem, and to express all those problems in good writing is an additional tough challenge. These folks have done an amazing job at both. Shoutout to Material for MkDocs enabling the swanky theme and Markdown extensions. https://squidfunk.github.io/mkdocs-material/ https://squidfunk.github.io/mkdocs-material/
optimalsolver 4y ago
Python packaging is a solved problem: https://python-poetry.org/ https://python-poetry.org/
- ris 4y ago
  Oh really how does it solve binary dependencies?
  - cozzyd 4y ago
    Based on OP's handle, I would guess optimally
- blackcat201 4y ago
  Recently just migrate a project from pypoetry away to the traditional setup method. Poetry works great for simple package, but once you started to add in complexities, it just falls apart due to everything was abstract away and simplified into config files and command line.
  - draw_down 4y ago
    [dead]
  - _frkl 4y ago
    I had the same experience. I think setuptools nowadays is quite good, esp. In combination with setup tools_scm.
ris 4y ago
This is exactly why I avoid pip et al whenever I can and just use Nix.
- physPop 4y ago
  Do you know of any up to date blogs/howtos/guides on nix+python where the python project contains modules that need to be compiled (eg Cython, pybind, etc)? I've found the basic info at https://nixos.wiki/wiki/Packaging/Python https://nixos.wiki/wiki/Packaging/Python but it doesn't really go in depth for more complex use cases than having a setup.py...
  - williamvds 4y ago
    Have you checked out the manual? https://nixos.org/manual/nixpkgs/stable/#python https://nixos.org/manual/nixpkgs/stable/#python Particularly section 17.27.1.2. Developing with Python. Combine that with a generic guide on using Flakes and that should get you started
krick 4y ago
Is there any consensus on how to deal with packaging and environments in Python by now? Can you suggest me some tutorial for that? I've been out of the loop for a long time, and would like to get an update on how things are in Python in 2023, but I'm not sure if there even is a consensus — what I can find by googling seems to be several kinda competing approaches. This seems surprising, because most "modern" languages seem to have a very well defined set of practices to deal with all of that stuff. Some languages already come with their built-in stuff (Go, Rust), others simply have well-known solutions (like, technically there still exist PEAR and PECL for PHP, but everyone just knows how to use composer, which solves both packaging and dependency-management problems, and it's also pretty clear what problems it doesn't solve). For Python there seems to be like a dozen of tools and I'm not sure which are outdated and not used by anyone, which are useless fancy wrappers (not used by anyone) and what is the actual go-to tool (if there is any) for all common cases. Dependency-management, version locking, shipping an executable, environment separation for some local scripts, should I even ever use pip install globally, etc.
- saila 4y ago
  I don't think there's a consensus, but there are some good modern options. I think poetry is a good choice, and it seems to be fairly popular. I use it for all my Python projects and haven't found a compelling reason to switch to another option in the past few years. As to the question of whether you should ever use pip to install packages globally, the answer is almost always no. For command line tools, the best option IMO is pipx. The second best option is pip install --user. If you're developing a library or application, you should always isolate it in a virtualenv, which is something poetry will handle for you when you run `poetry install`.
- idoubtit 4y ago
  From the sibling thread about packaging and deploying a single script, there was no consensus. There was disagreement on the best way to package, and doubts about the mid term future of some suggested solutions. The following alternatives were suggested: - package with a `pyproject.toml` file configured to use modern tooling - package with a `pyproject.toml` file configured to use traditional `setup.py` tooling - package with traditional `setup.py` tooling - package with poetry - package with whatever, deploy with nuikta or pipx - skip the packaging and deploy with Pyinstaller - skip the packaging and deploy with nikta Note that, unless the Python world has radically changed while I was looking away, the packaging does not ensure a simple way to deploy the package and its single script. I remember vividly `pipenv` crashing on me, so switching to venv+pip (or was it virtualenv+pip?) then setting up a bash wrapper to call the Python script with the right venv...
egberts1 4y ago
Obligatory Xkcd comic. https://pypi.org/project/xkcd2347/ https://pypi.org/project/xkcd2347/
- gdprrrr 4y ago
  Also https://xkcd.com/1987/ https://xkcd.com/1987/
deniska 4y ago
With modern tooling packaging pure python code to be used by other python developers is a relatively painless process. The main problem with python packaging is that it's often C/C++ packaging in disguise, among multiple OSes and CPU architectures, and that's far from being solved. Building such python wheel is essentially like building a "portable" (aka one you don't need to properly install into the system) linux/windows/macos application. That comes with a variety of caveats and requires some specialized knowledge one wouldn't pick up playing around with just python alone.
0x008 4y ago
Python packaging’s greatest challenge is 10 competing tools and standards.
- woodruffw 4y ago
  At this point, there are 10 competing tools but no longer so many competing standards: the standards for Python packaging from 2015 onwards are PEP 517 (build system standardization), PEP 518 (using pyproject.toml to configure the build system), and PEP 621 (storing project metadata, previously standardized, in pyproject.toml). These standards build on top of each other, meaning that they don't offer conflicting advice. The TL;DR for Python packaging in 2022 is that, unless you're building CPython extensions, you can do everything through pyproject.toml with PyPA-maintained tooling.
  - 0x008 4y ago
    The problem is, as long as old python versions continue to exist, the competing standards also will continue to exist. From a user experience stand point it is horrible. Depending on the python, pip, or Setup tools version on a system the Install command might do something drastically different in each case. Often it’s not even clear what’s happening under the hood.
- hot_gril 4y ago
  And 2 competing Python versions.
- cozzyd 4y ago
  Many of the tools don't attempt to handle the challenges mentioned here
- groodt 4y ago
  Python packaging gets a lot of criticism. It's a meme. The thing is, it's actually improved dramatically over the years and continues to improve. The problems it solves are very complex if one looks a little below the surface. It is solving different problems to the ecosystems that it's often compared to: golang, rust, java, js.
wiredfool 4y ago
Pillow has most of the issues that are listed in the article. (oddly enough for a graphics library, the GPU part is the only part that I don't think we've stumbled over at one point or another.) From a quality of life issue -- having the sdist install behind an opt-in flag by default for our package would be great. Unless you're a developer with a lot of -dev packages for imaging libraries already on your system, you're not going to be able to build from source. And even if the error that pops up is along the lines of The headers or library files could not be found for {str(err)}, a required dependency when compiling Pillow from source. Please see the install instructions at: https://pillow.readthedocs.io/en/latest/installation.html We still get people posting issues about pillow failing to install. Build farms would be nice. We've burned tons of time on it between travis and GH actions and @cgohlke single handedly making all of the windows builds for the entire scientific python community. Ultimately, something like the debian packaging system is probably the best open source model for this. (though the splitting of the python standard library so that virtual envs aren't in the base install is a pita). Unstable gets a reasonably current set of packages, and crucially all of the underlying library dependencies are compiled together. It's also not _that_ hard to rebuild individual packages from source, in an automated fashion. (This may be what Conda is doing, but I've never looked in detail at their system)
- david2ndaccount 4y ago
  Could you release the sdist as a separate package and only upload binary wheels for a normal install?
  - uranusjr 4y ago
    You could, and that’s what psycopg2 did for a while (for a different reason but effectively they separated source and binary releases as different projects). This solves an immediate problem but introduces new challenges, probably the most significant being Python does not allow OR dependencies (I want at least one of those) and splitting the package made dependant projects’ lives miserable.

120 points by groodt 4y ago

I've been planning on packaging a python package recently, and the internet is annoyingly full of guides which are, I think, out of date. They at least suggest quite different things. I just have a single python file, meant to be treated as an executable (no package at present). There are a whole bunch of tests, but that's obviously separate. Any suggestions on modern best practices welcome!

woodruffw 4y ago
If it's pure Python, the only packaging file you need is `pyproject.toml`. You can fill that file with packaging metadata per PEP 518 and PEP 621, including using modern build tooling like flit[1] for the build backend and build[2] for the frontend. With that, you entire package build (for all distribution types) should be reducible to `python -m build`. Here's an example of a full project doing everything with just `pyproject.toml`[3] (FD: my project). [1]: https://github.com/pypa/flit https://github.com/pypa/flit [2]: https://github.com/pypa/build https://github.com/pypa/build [3]: https://github.com/pypa/pip-audit https://github.com/pypa/pip-audit
- quietbritishjim 4y ago
  > including using modern build tooling like flit[1] for the build backend and build[2] for the frontend. Or you can use setuptools, which is the package that enables old setup.py builds, as the backend with pyproject.toml. This has the advantage of being mature, unlikely to be abandoned, and possibly some familiarity if you've used it before. Even then, you can use build as the front end build tool.
  - woodruffw 4y ago
    Yes, setuptools is also perfectly fine. It had some rough edges around PEP 621 support for a while, but those were mostly smoothed out in 2021. (I'll note that maturity is not strong evidence here: distutils is very mature, but is in the process of being deprecated and removed from Python entirely. I don't think that's likely to happen to setuptools, but the fact that behavioral PEPs now exist for all of these tools means that the decline/abandonment of any poses much less of an ecosystem risk.)
    - quietbritishjim 4y ago
      I suppose I meant maturity as in: actively maintained and recommended for use for a long period, which doesn't apply to distutils. And I should've been more upfront about the real reason for suggesting setuptools: there seem to be a number of build tools that support pyproject.toml, including flit, poetry and setuptools (and I'm sure I've seen at least one other). For me, at least, when I was making a small library recently, it was an overwhelming choice for a part of my project that feels like just admin rather than core business logic. I came close to giving up and just using setup.py with `setup()`. At least setuptools with pyproject.toml is a choice that feels safe; it may not be the best, but it will certainly be good enough that I'm unlikely to regret it later, so I didn't need to spend a lot of time looking at the detailed pros and cons of all the choices.
      - woodruffw 4y ago
        That's very reasonable! I don't mean to disparage that decision at all: setuptools is rock solid and a very safe choice.
- kevin_thibedeau 4y ago
  Good luck finding clear documentation on how to write the pyproject file. For something that is supposed to be the way forward there hasn't been much effort to make it easy to implement. There's also annoyances like the inability to install a script in the search path without implementing it as a module. Something setup.py doesn't require.
  - woodruffw 4y ago
    > Good luck finding clear documentation on how to write the pyproject file. For something that is supposed to be the way forward there hasn't been much effort to make it easy to implement. PEP 621, which I mentioned, covers the format of `pyproject.toml` in detail. I also linked an example which, to the best of my knowledge, covers all current best practices for that file. > There's also annoyances like the inability to install a script in the search path without implementing it as a module. Something setup.py doesn't require. I'm not sure I'm following. A module in Python is just a Python file, so your script is a module. Are you saying that you can't distribute single-module packages with pyproject.toml? Because I don't think that's true.
    - kevin_thibedeau 4y ago
      The PEP in no way explains how to write a usable pyproject for ordinary projects. It's basically just targeted at people developing installers. I meant package. A directory with an __init__.py. You can't install standalone script.py (or a generated wrapper) as /usr/local/bin/script with a pyproject.
      - woodruffw 4y ago
        > The PEP in no way explains how to write a usable pyproject for ordinary projects. It's basically just targeted at people developing installers. Did you look at it[1]? > I meant package. A directory with an __init__.py. You can't install standalone script.py (or a generated wrapper) as /usr/local/bin/script with a pyproject. I still don't think I understand what your expectation is here: a `pyproject.toml` is just a metadata specification. The only difference between it and `setup.py` is that the latter is arbitrary code. There's an old, long deprecated way to use `setup.py`, namely `setup.py install`. But that's been discouraged in favor of `pip install` for years, which behaves precisely the same way with `pyproject.toml`. If you want to install a script into `/usr/local/bin`, `pip install` with a package specified in `pyproject.toml` will work just fine. [1]: https://peps.python.org/pep-0621/#example https://peps.python.org/pep-0621/#example
ensignavenger 4y ago
If you are wanting to release it to pypi as a python package, I would personally use Poetry. But your case- a single pure Python package, is a simple case that won't have many problems like are brought up in the article, whatever tool you use. If you want a stand alone executable, I haven't found a good, single, cross platform tool for that yet... seems like there is a separate tool for each platform.
- quietbritishjim 4y ago
  Nuitka works on Windows, Linux and Mac https://nuitka.net/ https://nuitka.net/
- selcuka 4y ago
  > If you want a stand alone executable, I haven't found a good, single, cross platform tool for that yet. PyInstaller is cross platform, and arguably good.
mixmastamyk 4y ago
Keep it simple and fashion-proof. Been using setup.py for a one-script package for one or two? decades: from setuptools import setup setup( name = 'foobar', scripts = ['foo'], # install a script from current fldr # ... ) A few years ago I had to start using twine to register and upload it to pypi.
- tomrod 4y ago
  Is there a place you recommend where I can learn more about this?
  - mixmastamyk 4y ago
    It’s documented, try google, stackoverflow, and reading other packages’ setup.py for tricks. However the scripts=[], keyword is the key to the case above.
- talideon 4y ago
  I wouldn't recommend that at all these days. setup.py definitely is not future-proof: it's undergoing deprecation. Better would be a basic pyproject.toml file along the lines of the following: [build-system] requires = ["setuptools"] build-backend = "setuptools.build_meta" [project] name = "foobar" version = "0.0.1" dependencies = [ "...", ] [project.scripts] foo = "foobar:main" See: https://setuptools.pypa.io/en/latest/userguide/quickstart.html https://setuptools.pypa.io/en/latest/userguide/quickstart.ht...
  - eesmith 4y ago
    > it's undergoing deprecation In the "we would rather people not use this but it's going to stay around for a long time" sense. I strongly doubt it will disappear within the next decade or two. There's a long tail of setup.py-based tools. Last I checked pyproject.toml only supports the simplest of Python/C extensions. Anything fancy, like --with/--without compilation flags to enable/disable optional support, compiler-specific flags (in my case, to add OpenMP), compile-time code generation (like using yacc/lex), etc. requires a setup.py and a bunch of hacking.
japanuspus 4y ago
As detailed in the other answers, there are two parts to this: 1) Creating a python package from your project (and possibly share this on pypi), and 2) Making this package available as an end-user application. For step 2 you can use nuitka or similar, but if your audience is somewhat developer-oriented, you can also propose for them to use pipx: https://github.com/pypa/pipx https://github.com/pypa/pipx.
tpoacher 4y ago
The approach I prefer is to not mess with setuptools etc at all in the first place, and simply make a nice executable package. e.g. https://github.com/tpapastylianou/self-contained-runnable-python-package-template https://github.com/tpapastylianou/self-contained-runnable-py...

miohtama 4y ago

Python joke: Sdist is only one letter away from sadist.

irskep 4y ago

This is an incredible example of organizing information well and making a case to a wide audience. It's difficult enough to shave all the yaks necessary to get a high-level view of all issues related to a problem, and to express all those problems in good writing is an additional tough challenge. These folks have done an amazing job at both. Shoutout to Material for MkDocs enabling the swanky theme and Markdown extensions. https://squidfunk.github.io/mkdocs-material/ https://squidfunk.github.io/mkdocs-material/

optimalsolver 4y ago

Python packaging is a solved problem: https://python-poetry.org/ https://python-poetry.org/

ris 4y ago
Oh really how does it solve binary dependencies?
- cozzyd 4y ago
  Based on OP's handle, I would guess optimally
blackcat201 4y ago
Recently just migrate a project from pypoetry away to the traditional setup method. Poetry works great for simple package, but once you started to add in complexities, it just falls apart due to everything was abstract away and simplified into config files and command line.
- draw_down 4y ago
  [dead]
- _frkl 4y ago
  I had the same experience. I think setuptools nowadays is quite good, esp. In combination with setup tools_scm.

ris 4y ago

This is exactly why I avoid pip et al whenever I can and just use Nix.

physPop 4y ago
Do you know of any up to date blogs/howtos/guides on nix+python where the python project contains modules that need to be compiled (eg Cython, pybind, etc)? I've found the basic info at https://nixos.wiki/wiki/Packaging/Python https://nixos.wiki/wiki/Packaging/Python but it doesn't really go in depth for more complex use cases than having a setup.py...
- williamvds 4y ago
  Have you checked out the manual? https://nixos.org/manual/nixpkgs/stable/#python https://nixos.org/manual/nixpkgs/stable/#python Particularly section 17.27.1.2. Developing with Python. Combine that with a generic guide on using Flakes and that should get you started

krick 4y ago

Is there any consensus on how to deal with packaging and environments in Python by now? Can you suggest me some tutorial for that? I've been out of the loop for a long time, and would like to get an update on how things are in Python in 2023, but I'm not sure if there even is a consensus — what I can find by googling seems to be several kinda competing approaches. This seems surprising, because most "modern" languages seem to have a very well defined set of practices to deal with all of that stuff. Some languages already come with their built-in stuff (Go, Rust), others simply have well-known solutions (like, technically there still exist PEAR and PECL for PHP, but everyone just knows how to use composer, which solves both packaging and dependency-management problems, and it's also pretty clear what problems it doesn't solve). For Python there seems to be like a dozen of tools and I'm not sure which are outdated and not used by anyone, which are useless fancy wrappers (not used by anyone) and what is the actual go-to tool (if there is any) for all common cases. Dependency-management, version locking, shipping an executable, environment separation for some local scripts, should I even ever use pip install globally, etc.

saila 4y ago
I don't think there's a consensus, but there are some good modern options. I think poetry is a good choice, and it seems to be fairly popular. I use it for all my Python projects and haven't found a compelling reason to switch to another option in the past few years. As to the question of whether you should ever use pip to install packages globally, the answer is almost always no. For command line tools, the best option IMO is pipx. The second best option is pip install --user. If you're developing a library or application, you should always isolate it in a virtualenv, which is something poetry will handle for you when you run `poetry install`.
idoubtit 4y ago
From the sibling thread about packaging and deploying a single script, there was no consensus. There was disagreement on the best way to package, and doubts about the mid term future of some suggested solutions. The following alternatives were suggested: - package with a `pyproject.toml` file configured to use modern tooling - package with a `pyproject.toml` file configured to use traditional `setup.py` tooling - package with traditional `setup.py` tooling - package with poetry - package with whatever, deploy with nuikta or pipx - skip the packaging and deploy with Pyinstaller - skip the packaging and deploy with nikta Note that, unless the Python world has radically changed while I was looking away, the packaging does not ensure a simple way to deploy the package and its single script. I remember vividly `pipenv` crashing on me, so switching to venv+pip (or was it virtualenv+pip?) then setting up a bash wrapper to call the Python script with the right venv...

egberts1 4y ago

Obligatory Xkcd comic. https://pypi.org/project/xkcd2347/ https://pypi.org/project/xkcd2347/

gdprrrr 4y ago
Also https://xkcd.com/1987/ https://xkcd.com/1987/

deniska 4y ago

With modern tooling packaging pure python code to be used by other python developers is a relatively painless process. The main problem with python packaging is that it's often C/C++ packaging in disguise, among multiple OSes and CPU architectures, and that's far from being solved. Building such python wheel is essentially like building a "portable" (aka one you don't need to properly install into the system) linux/windows/macos application. That comes with a variety of caveats and requires some specialized knowledge one wouldn't pick up playing around with just python alone.

0x008 4y ago

Python packaging’s greatest challenge is 10 competing tools and standards.

woodruffw 4y ago
At this point, there are 10 competing tools but no longer so many competing standards: the standards for Python packaging from 2015 onwards are PEP 517 (build system standardization), PEP 518 (using pyproject.toml to configure the build system), and PEP 621 (storing project metadata, previously standardized, in pyproject.toml). These standards build on top of each other, meaning that they don't offer conflicting advice. The TL;DR for Python packaging in 2022 is that, unless you're building CPython extensions, you can do everything through pyproject.toml with PyPA-maintained tooling.
- 0x008 4y ago
  The problem is, as long as old python versions continue to exist, the competing standards also will continue to exist. From a user experience stand point it is horrible. Depending on the python, pip, or Setup tools version on a system the Install command might do something drastically different in each case. Often it’s not even clear what’s happening under the hood.
hot_gril 4y ago
And 2 competing Python versions.
cozzyd 4y ago
Many of the tools don't attempt to handle the challenges mentioned here
groodt 4y ago
Python packaging gets a lot of criticism. It's a meme. The thing is, it's actually improved dramatically over the years and continues to improve. The problems it solves are very complex if one looks a little below the surface. It is solving different problems to the ecosystems that it's often compared to: golang, rust, java, js.

wiredfool 4y ago

Pillow has most of the issues that are listed in the article. (oddly enough for a graphics library, the GPU part is the only part that I don't think we've stumbled over at one point or another.) From a quality of life issue -- having the sdist install behind an opt-in flag by default for our package would be great. Unless you're a developer with a lot of -dev packages for imaging libraries already on your system, you're not going to be able to build from source. And even if the error that pops up is along the lines of The headers or library files could not be found for {str(err)}, a required dependency when compiling Pillow from source. Please see the install instructions at: https://pillow.readthedocs.io/en/latest/installation.html We still get people posting issues about pillow failing to install. Build farms would be nice. We've burned tons of time on it between travis and GH actions and @cgohlke single handedly making all of the windows builds for the entire scientific python community. Ultimately, something like the debian packaging system is probably the best open source model for this. (though the splitting of the python standard library so that virtual envs aren't in the base install is a pita). Unstable gets a reasonably current set of packages, and crucially all of the underlying library dependencies are compiled together. It's also not _that_ hard to rebuild individual packages from source, in an automated fashion. (This may be what Conda is doing, but I've never looked in detail at their system)

david2ndaccount 4y ago
Could you release the sdist as a separate package and only upload binary wheels for a normal install?
- uranusjr 4y ago
  You could, and that’s what psycopg2 did for a while (for a different reason but effectively they separated source and binary releases as different projects). This solves an immediate problem but introduces new challenges, probably the most significant being Python does not allow OR dependencies (I want at least one of those) and splitting the package made dependant projects’ lives miserable.