4 ms·

I always struggle with these discussions. I have been using Python a long time and while there ahve definitely been a handful of gotchas over the years, it was

I always struggle with these discussions. I have been using Python a long time and while there ahve definitely been a handful of gotchas over the years, it was never an experience that held me up long enought to think twice about. I am sure there is a better way but what holds me up is how often it impacts other people but not myself. Perhaps my usage is not advanced enough but it always leaves me wondering. Edit: Just as an additional thought, is one of the main issues when distributing to client machines to what is essentially python executables and the clients could be using different OS? My mind jumps to probably not using Python at that point unless there was a specific dependency/library to using it.

bbojan 3y ago
Same here. I always see people complaining about Python packaging, but rarely run into issues myself. Maybe it's an OS issue? I use Linux. Some of the projects at work use gigabytes of computer vision dependencies and again I don't remember the last time I had any issues. Could some of the people who do have issues give an example or two of the problems they run into?
- oblio 3y ago
  You don't try to distribute Python applications to multiple OSes, I assume?
- epage 3y ago
  Part of the problem is the frustration is the curse of choice, the need to research what tool to use. Before the new crowd of tools, like poetry, the problem was also having to figure out how to chain all the scattered tools together. I likely still have some bias from this experience. I stopped doing python before getting to adopt any of these tools but when I researched Poetry for a company to use, a fatal flaw is that they copied too much from Rust/Cargo, Cargo was born with its community and they help shaped each other. The rust community has a strong adherance to semver. In Python, you have at mess of versioning schemes (e.g. CalVer) and low quality version requirements. You need a way to override version requirements but they refuse. Doing cross platform development? The tools that do locking today via `requirements.txt` generate platform-specific lockfiles.
- throwup238 3y ago
  > CUDA version on your system does not match the CUDA version torch was compiled with Bane of my existence.
  - gipp 3y ago
    I mostly agree with the GP comment, but CUDA support has definitely been the one exception that was just an absolutely infuriating experience.
  - Shish2k 3y ago
    Yes this - I’ve been using python for decades, and been mostly fine (dropping down to ~zero packaging & deployment issues since I started putting all my services into docker images). But AI stuff? Somehow that manages to be a massive pain in the ass every single time
- marbs 3y ago
  Here are the two main packaging issues I run into, specifically when using Poetry: 1) Lack of support for building extension modules (as mentioned by the article). There is a workaround using an undocumented feature [0], which I've tried, but ultimately decided it was not the right approach. I still use Poetry, but build the extension as a separate step in CI, rather than kludging it into Poetry. 2) Lack of support for offline installs [1], e.g. being able to download the dependencies, copy them to another machine, and perform the install from the downloaded dependencies (similar to using "pip --no-index --find-links=."). Again, you can work around this (by using "poetry export --with-credentials" and "pip download" for fetching the dependencies, then firing up pypiserver [2] to run a local PyPI server on the offline machine), but ideally this would all be a first class feature of Poetry, similar to how it is in pip. I don't have the capacity to create Pull Requests for addressing these issues with Poetry, and I'm very grateful for the maintainers and those who do contribute. Instead, on the linked issues I share my notes on the matter, in the hope that it may at least help others and potentially get us closer to a solution. Regardless, I'm sticking with Poetry for now. Though to be fair, the only other Python packaging tools I've used extensively are Pipenv and pip/setuptools. It's time consuming to thoroughly try out these other packaging tools, and is generally lower priority than developing features/fixing bugs, so it's helpful to read about the author's experience with these other tools, such as PDM and Hatch. [0] https://github.com/python-poetry/poetry/issues/2740 https://github.com/python-poetry/poetry/issues/2740 [1] https://github.com/python-poetry/poetry/issues/2184 https://github.com/python-poetry/poetry/issues/2184 [2] https://pypi.org/project/pypiserver/ https://pypi.org/project/pypiserver/
- coldtea 3y ago
  >I always see people complaining about Python packaging, but rarely run into issues myself. Maybe it's an OS issue? I use Linux. No, it's a "what you do with it issue". Not necessarily about the mere "number of dependencies" used, when e.g. someone just makes some conda env and is fine with it. Things like relocating, provisioning, reproducibility, version updating, cross platform, etc, all have their issues, and it gets worse when you need to build your own packages.
jonawesomegreen 3y ago
I've encountered frustration with Python packaging in two main areas: 1) Installing applications within Docker containers. While wheels have improved this situation, I was surprised that building a package for easy copying into a container and running without the need for installing build tools and extensions in the final container image was not straightforward, especially coming from other languages. 2) Distributing Python utilities to end users across various platforms in an easily installable manner without requiring them to follow lengthy instructions to set up all the dependencies has been another challenge. We use Poetry and have largely "solved" #1 with a somewhat complicated Docker build. It works well now, so no one has to think about it much. That made deploying Python server-side fairly easy. However, #2 has been much more of a challenge, and I wonder if that is where other folks in this thread are feeling the most pain.
- kingkongjaffa 3y ago
  Pyinstaller has been robust enough for us to deploy Python written windows executables without any issues.
- guappa 3y ago
  All of those problems can be easily solved by distribution packages.
  - ogrisel 3y ago
    Distribution packages are nice but: - they require different instructions for each platform: one for each distribution: it's not possible to give a one liner that will work for all supported OS/architecture combinations, - distributions are not all updated at the speed of upstream. Sometimes users do not want to wait several years to use a recently released feature.
coldtea 3y ago
>it was never an experience that held me up long enought to think twice about The pain is when you're doing more complex stuff than creating a local virtualenv with some basic deps though.
- infecto 3y ago
  That is why I was wondering. I have produced a lot of software using python, deployed it on different platforms, using containers, not using containers, bare metal, cloud, using complex dependecies, etc. I have not run into a lot of issues and most of them for me were mostly depedency verion related where different versions were required for subdependencies but even then it was not super complicate. I realize more complex scenarios exist but it surprises me that its such a hot topic always with Python.
volker48 3y ago
The only time I've ever experienced a memorable amount of pain was when I was working on a project that used one of the newer options like Poetry. Using a virtualenv, a requirements.txt file, and pip I have not had serious issues in 15 years of using Python.

bbojan 3y ago

Same here. I always see people complaining about Python packaging, but rarely run into issues myself. Maybe it's an OS issue? I use Linux. Some of the projects at work use gigabytes of computer vision dependencies and again I don't remember the last time I had any issues. Could some of the people who do have issues give an example or two of the problems they run into?

oblio 3y ago
You don't try to distribute Python applications to multiple OSes, I assume?
epage 3y ago
Part of the problem is the frustration is the curse of choice, the need to research what tool to use. Before the new crowd of tools, like poetry, the problem was also having to figure out how to chain all the scattered tools together. I likely still have some bias from this experience. I stopped doing python before getting to adopt any of these tools but when I researched Poetry for a company to use, a fatal flaw is that they copied too much from Rust/Cargo, Cargo was born with its community and they help shaped each other. The rust community has a strong adherance to semver. In Python, you have at mess of versioning schemes (e.g. CalVer) and low quality version requirements. You need a way to override version requirements but they refuse. Doing cross platform development? The tools that do locking today via `requirements.txt` generate platform-specific lockfiles.
throwup238 3y ago
> CUDA version on your system does not match the CUDA version torch was compiled with Bane of my existence.
- gipp 3y ago
  I mostly agree with the GP comment, but CUDA support has definitely been the one exception that was just an absolutely infuriating experience.
- Shish2k 3y ago
  Yes this - I’ve been using python for decades, and been mostly fine (dropping down to ~zero packaging & deployment issues since I started putting all my services into docker images). But AI stuff? Somehow that manages to be a massive pain in the ass every single time
marbs 3y ago
Here are the two main packaging issues I run into, specifically when using Poetry: 1) Lack of support for building extension modules (as mentioned by the article). There is a workaround using an undocumented feature [0], which I've tried, but ultimately decided it was not the right approach. I still use Poetry, but build the extension as a separate step in CI, rather than kludging it into Poetry. 2) Lack of support for offline installs [1], e.g. being able to download the dependencies, copy them to another machine, and perform the install from the downloaded dependencies (similar to using "pip --no-index --find-links=."). Again, you can work around this (by using "poetry export --with-credentials" and "pip download" for fetching the dependencies, then firing up pypiserver [2] to run a local PyPI server on the offline machine), but ideally this would all be a first class feature of Poetry, similar to how it is in pip. I don't have the capacity to create Pull Requests for addressing these issues with Poetry, and I'm very grateful for the maintainers and those who do contribute. Instead, on the linked issues I share my notes on the matter, in the hope that it may at least help others and potentially get us closer to a solution. Regardless, I'm sticking with Poetry for now. Though to be fair, the only other Python packaging tools I've used extensively are Pipenv and pip/setuptools. It's time consuming to thoroughly try out these other packaging tools, and is generally lower priority than developing features/fixing bugs, so it's helpful to read about the author's experience with these other tools, such as PDM and Hatch. [0] https://github.com/python-poetry/poetry/issues/2740 https://github.com/python-poetry/poetry/issues/2740 [1] https://github.com/python-poetry/poetry/issues/2184 https://github.com/python-poetry/poetry/issues/2184 [2] https://pypi.org/project/pypiserver/ https://pypi.org/project/pypiserver/
coldtea 3y ago
>I always see people complaining about Python packaging, but rarely run into issues myself. Maybe it's an OS issue? I use Linux. No, it's a "what you do with it issue". Not necessarily about the mere "number of dependencies" used, when e.g. someone just makes some conda env and is fine with it. Things like relocating, provisioning, reproducibility, version updating, cross platform, etc, all have their issues, and it gets worse when you need to build your own packages.

jonawesomegreen 3y ago

I've encountered frustration with Python packaging in two main areas: 1) Installing applications within Docker containers. While wheels have improved this situation, I was surprised that building a package for easy copying into a container and running without the need for installing build tools and extensions in the final container image was not straightforward, especially coming from other languages. 2) Distributing Python utilities to end users across various platforms in an easily installable manner without requiring them to follow lengthy instructions to set up all the dependencies has been another challenge. We use Poetry and have largely "solved" #1 with a somewhat complicated Docker build. It works well now, so no one has to think about it much. That made deploying Python server-side fairly easy. However, #2 has been much more of a challenge, and I wonder if that is where other folks in this thread are feeling the most pain.

kingkongjaffa 3y ago
Pyinstaller has been robust enough for us to deploy Python written windows executables without any issues.
guappa 3y ago
All of those problems can be easily solved by distribution packages.
- ogrisel 3y ago
  Distribution packages are nice but: - they require different instructions for each platform: one for each distribution: it's not possible to give a one liner that will work for all supported OS/architecture combinations, - distributions are not all updated at the speed of upstream. Sometimes users do not want to wait several years to use a recently released feature.

coldtea 3y ago

>it was never an experience that held me up long enought to think twice about The pain is when you're doing more complex stuff than creating a local virtualenv with some basic deps though.

infecto 3y ago
That is why I was wondering. I have produced a lot of software using python, deployed it on different platforms, using containers, not using containers, bare metal, cloud, using complex dependecies, etc. I have not run into a lot of issues and most of them for me were mostly depedency verion related where different versions were required for subdependencies but even then it was not super complicate. I realize more complex scenarios exist but it surprises me that its such a hot topic always with Python.

volker48 3y ago

The only time I've ever experienced a memorable amount of pain was when I was working on a project that used one of the newer options like Poetry. Using a virtualenv, a requirements.txt file, and pip I have not had serious issues in 15 years of using Python.