6 ms·

So, what is "the tool for beginners fighting with python packaging problems"? That is, the pattern seems to be that someone mentions a solution, then a zillion

So, what is "the tool for beginners fighting with python packaging problems"? That is, the pattern seems to be that someone mentions a solution, then a zillion responses as to why it sucks. Is there any tool or pair that sucks least for most cases and beginners? I get that every case is different, but perhaps there are some useful starting points?

IshKebab 3y ago
There isn't one. Python distribution and packaging is just fundamentally horribly broken. I think his point was that you shouldn't pretend to users that just switching to pyenv is the solution.
- notatallshaw 3y ago
  > Python distribution and packaging is just fundamentally horribly broken It's clearly not because most people successfully use it fine. The problem of distribution and packaging is often a matter of user expectations vs. the actual problem. The user expectations is that Python is a high level language and will run the same across different machines regardless of OS and Hardware. The actual problem is Python is a glue language often depending on lots of libraries that are sensitive to how they were compiled and what hardware they are targeting. So people can end up moving the the glue of the project first and then expect everything else to just work. Things are getting better (e.g. https://lukeplant.me.uk/blog/posts/python-packaging-must-be-getting-better-a-datapoint/ https://lukeplant.me.uk/blog/posts/python-packaging-must-be-...), a lot of people have put a lot of work into standards and improving the ecosystem, but it is a hard problem that most other popular languages don't interface with nearly as much.
  - Capricorn2481 3y ago
    I don't disagree with what you're saying but that article seems odd. It's just a story of someone installing something once that didn't break immediately. Not only is it anecdotal, it doesn't even seem to confirm whether the pip story is getting better or if they just got lucky.
    - notatallshaw 3y ago
      I agree, but there is no large scale study on this. As someone who has managed Python distributions in a large company and who triages issues on the pip github issue page that my anecdotal experience is things are getting better. The only hard statistic I can point to is the number of top packages on PyPI that offer wheels has substantially gone up, and is close to 100% in the top 500.
  - hot_gril 3y ago
    They use it fine by using Docker. So many major Python repos on GitHub come with a Dockerfile compared to, say, NodeJS. It's unfortunate, but having dealt with Python packages before, I don't blame them.
    - notatallshaw 3y ago
      Lots of analysts, data scientists, traders, engineers, etc., use Python third party packages successfully, and have never touched, or maybe even head of, Docker. And yeah, in general there are significantly less NodeJS third party packages interfacing with packages that directly depend on OSes and hardware. Python has many third party packages that are older than NodeJS that depend on foreign function interfaces, win32 com apis, directly talking to graphics shaders, etc.
      - hot_gril 3y ago
        Python packages are painful even if nothing native is involved. There are NodeJS packages that rely on native code too, difference is the packaging system is a lot simpler to use, though it's starting to get worse with the `import` vs `require` nonsense and Typescript.
  - IshKebab 3y ago
    > It's clearly not because most people successfully use it fine. Well, no because it's perfectly possible to successfully use a horribly broken system. I use Python "successfully", it just meant I have spend probably literal weeks of my life fighting pip and virtualenv and relative imports and finding I need flags like `--config-settings editable_mode=compat`. > The problem of distribution and packaging is often a matter of user expectations vs. the actual problem. Ha yes, I expect it to work reliably and simply and it doesn't! > The actual problem is Python is a glue language often depending on lots of libraries that are sensitive to how they were compiled and what hardware they are targeting. That's completely irrelevant to the kind of problems I was talking about. I run into issues with Python failing to compile C libraries relatively rarely! Even compiling Python itself seems to work quite well (maybe not surprising since that's one of the only ways to get a new version on Linux). It's all the packaging infrastructure that's a mess. Pip, virtualenv, setuptools, and also the module import system is a total disaster. You don't see questions like this for Go: https://stackoverflow.com/questions/14132789/relative-imports-for-the-billionth-time https://stackoverflow.com/questions/14132789/relative-import...
    - notatallshaw 3y ago
      > It's all the packaging infrastructure that's a mess. Pip, virtualenv, setuptools, and also the module import system is a total disaster. You don't see questions like this for Go: > https://stackoverflow.com/questions/14132789/relative-imports-for-the-billionth-time https://stackoverflow.com/questions/14132789/relative-import... I think you'll find the further you delve into it the less the problems are distinct, a lot of the issues with the module import system, pip, virtualenv, setuptools, etc. is because they are designed with having to support a vast range of things, from being depended on to interact with system libraries to downloading sdists and compiling arbitrary languages, etc. Though the specific example you linked was largely solved with Python 3, there was a lot of confusion during the 2 to 3 transition because people had to support both behaviors, but most people don't have to think about Python 2 any more.
      - IshKebab 3y ago
        > Though the specific example you linked was largely solved with Python 3 I can assure you it absolutely was not. > I think you'll find the further you delve into it the less the problems are distinct, a lot of the issues with the module import system, pip, virtualenv, setuptools, etc. is because they are designed with having to support a vast range of things, from being depended on to interact with system libraries to downloading sdists and compiling arbitrary languages, etc. Not really. There are plenty of systems that have to "support a vast range of things" that aren't this bad. In my opinion it's because the core Python devs didn't particularly care about the issue, never really tried to solve it, and as a result we have 10 incompatible half-baked third party solutions. It's similar to the situation with C/C++ - worse in some ways, better in others (at least there is a de facto package registry in Python). In some ways it's because both languages are very old and predate the idea that packaging should be easy and reliable. That's fine, but please don't pretend that it is easy and reliable now.
philwelch 3y ago
You could just give up and resort to using Docker.
- fishpen0 3y ago
  Honestly this. I've moved three companies now to docker based dev for teams on python. It's better for teams who want to customize how they in particular work without inadvertently also blocking themselves from quickly contributing to another team because that team uses a totally different python management scheme. So a team can be conda-in-docker and another team can be virtualenv-in-docker and another team can be raw-python-in-docker and that one group of weirdos can be jython or whatever other sad shit they cook up. The common interface is they all use `docker-compose up` and have their editors hooked into the containers.
- Zetobal 3y ago
  this is the way.
- BiteCode_dev 3y ago
  Using docker is not free, you are effectively trading one type of complexity for another. And of course you'll have to make sure people equally have good practices, since incorrectly using sudo pip install in docker and not a venv are very common. So again, one possible solution for a certain context, but I wouldn't sell that to most people. Certainly not to beginners. People writing their pythonanywhere website won't pop up a container, won't they ?
  - hot_gril 3y ago
    For a beginner trying to run someone else's Python project, it is free. You run one command and it works. There's tons more complexity under the hood, but you don't have to deal with it. This is why so many Python projects do resort to Docker. And btw, JS doesn't have this problem, they use NPM.
    - dragonwriter 3y ago
BiteCode_dev 3y ago
While there are no silver bullet, for beginners, I found that "Relieving your Python packaging pain" (https://www.bitecode.dev/p/relieving-your-python-packaging-pain https://www.bitecode.dev/p/relieving-your-python-packaging-p...) is the Pareto solution. That is, the solution that has the best ratio of effort, reward, but also the lower risk of failure. It's not no risk, but I've been helping beginners for 15 years with Python, and have tried everything you can think of. It does imply, on linux, to limit yourself to the choices of Python you can install. This constraint is, for most people, preferable than the alternative, even if it gets frustrating to our geeky soul.
hot_gril 3y ago
The silver bullet is Docker. It's not the best solution in every situation, but at least it'll work. For your own projects built from scratch (rather than big multi-dep projects off GitHub), there's a smaller learning curve if you go to the vanilla https://www.python.org/downloads/ https://www.python.org/downloads/ , install the latest, and use the included pip to install packages. That'll probably get you very far. It's not like the older days when you needed both Py3 and Py2. For experts working frequently in Python, tools like Pyenv can make more sense.

IshKebab 3y ago

There isn't one. Python distribution and packaging is just fundamentally horribly broken. I think his point was that you shouldn't pretend to users that just switching to pyenv is the solution.

notatallshaw 3y ago
> Python distribution and packaging is just fundamentally horribly broken It's clearly not because most people successfully use it fine. The problem of distribution and packaging is often a matter of user expectations vs. the actual problem. The user expectations is that Python is a high level language and will run the same across different machines regardless of OS and Hardware. The actual problem is Python is a glue language often depending on lots of libraries that are sensitive to how they were compiled and what hardware they are targeting. So people can end up moving the the glue of the project first and then expect everything else to just work. Things are getting better (e.g. https://lukeplant.me.uk/blog/posts/python-packaging-must-be-getting-better-a-datapoint/ https://lukeplant.me.uk/blog/posts/python-packaging-must-be-...), a lot of people have put a lot of work into standards and improving the ecosystem, but it is a hard problem that most other popular languages don't interface with nearly as much.
- Capricorn2481 3y ago
  I don't disagree with what you're saying but that article seems odd. It's just a story of someone installing something once that didn't break immediately. Not only is it anecdotal, it doesn't even seem to confirm whether the pip story is getting better or if they just got lucky.
  - notatallshaw 3y ago
    I agree, but there is no large scale study on this. As someone who has managed Python distributions in a large company and who triages issues on the pip github issue page that my anecdotal experience is things are getting better. The only hard statistic I can point to is the number of top packages on PyPI that offer wheels has substantially gone up, and is close to 100% in the top 500.
- hot_gril 3y ago
  They use it fine by using Docker. So many major Python repos on GitHub come with a Dockerfile compared to, say, NodeJS. It's unfortunate, but having dealt with Python packages before, I don't blame them.
  - notatallshaw 3y ago
    Lots of analysts, data scientists, traders, engineers, etc., use Python third party packages successfully, and have never touched, or maybe even head of, Docker. And yeah, in general there are significantly less NodeJS third party packages interfacing with packages that directly depend on OSes and hardware. Python has many third party packages that are older than NodeJS that depend on foreign function interfaces, win32 com apis, directly talking to graphics shaders, etc.
    - hot_gril 3y ago
      Python packages are painful even if nothing native is involved. There are NodeJS packages that rely on native code too, difference is the packaging system is a lot simpler to use, though it's starting to get worse with the `import` vs `require` nonsense and Typescript.
- IshKebab 3y ago
  > It's clearly not because most people successfully use it fine. Well, no because it's perfectly possible to successfully use a horribly broken system. I use Python "successfully", it just meant I have spend probably literal weeks of my life fighting pip and virtualenv and relative imports and finding I need flags like `--config-settings editable_mode=compat`. > The problem of distribution and packaging is often a matter of user expectations vs. the actual problem. Ha yes, I expect it to work reliably and simply and it doesn't! > The actual problem is Python is a glue language often depending on lots of libraries that are sensitive to how they were compiled and what hardware they are targeting. That's completely irrelevant to the kind of problems I was talking about. I run into issues with Python failing to compile C libraries relatively rarely! Even compiling Python itself seems to work quite well (maybe not surprising since that's one of the only ways to get a new version on Linux). It's all the packaging infrastructure that's a mess. Pip, virtualenv, setuptools, and also the module import system is a total disaster. You don't see questions like this for Go: https://stackoverflow.com/questions/14132789/relative-imports-for-the-billionth-time https://stackoverflow.com/questions/14132789/relative-import...
  - notatallshaw 3y ago
    > It's all the packaging infrastructure that's a mess. Pip, virtualenv, setuptools, and also the module import system is a total disaster. You don't see questions like this for Go: > https://stackoverflow.com/questions/14132789/relative-imports-for-the-billionth-time https://stackoverflow.com/questions/14132789/relative-import... I think you'll find the further you delve into it the less the problems are distinct, a lot of the issues with the module import system, pip, virtualenv, setuptools, etc. is because they are designed with having to support a vast range of things, from being depended on to interact with system libraries to downloading sdists and compiling arbitrary languages, etc. Though the specific example you linked was largely solved with Python 3, there was a lot of confusion during the 2 to 3 transition because people had to support both behaviors, but most people don't have to think about Python 2 any more.
    - IshKebab 3y ago
      > Though the specific example you linked was largely solved with Python 3 I can assure you it absolutely was not. > I think you'll find the further you delve into it the less the problems are distinct, a lot of the issues with the module import system, pip, virtualenv, setuptools, etc. is because they are designed with having to support a vast range of things, from being depended on to interact with system libraries to downloading sdists and compiling arbitrary languages, etc. Not really. There are plenty of systems that have to "support a vast range of things" that aren't this bad. In my opinion it's because the core Python devs didn't particularly care about the issue, never really tried to solve it, and as a result we have 10 incompatible half-baked third party solutions. It's similar to the situation with C/C++ - worse in some ways, better in others (at least there is a de facto package registry in Python). In some ways it's because both languages are very old and predate the idea that packaging should be easy and reliable. That's fine, but please don't pretend that it is easy and reliable now.

philwelch 3y ago

You could just give up and resort to using Docker.

fishpen0 3y ago
Honestly this. I've moved three companies now to docker based dev for teams on python. It's better for teams who want to customize how they in particular work without inadvertently also blocking themselves from quickly contributing to another team because that team uses a totally different python management scheme. So a team can be conda-in-docker and another team can be virtualenv-in-docker and another team can be raw-python-in-docker and that one group of weirdos can be jython or whatever other sad shit they cook up. The common interface is they all use `docker-compose up` and have their editors hooked into the containers.
Zetobal 3y ago
this is the way.
BiteCode_dev 3y ago
Using docker is not free, you are effectively trading one type of complexity for another. And of course you'll have to make sure people equally have good practices, since incorrectly using sudo pip install in docker and not a venv are very common. So again, one possible solution for a certain context, but I wouldn't sell that to most people. Certainly not to beginners. People writing their pythonanywhere website won't pop up a container, won't they ?
- hot_gril 3y ago
  For a beginner trying to run someone else's Python project, it is free. You run one command and it works. There's tons more complexity under the hood, but you don't have to deal with it. This is why so many Python projects do resort to Docker. And btw, JS doesn't have this problem, they use NPM.
  - dragonwriter 3y ago

BiteCode_dev 3y ago

While there are no silver bullet, for beginners, I found that "Relieving your Python packaging pain" (https://www.bitecode.dev/p/relieving-your-python-packaging-pain https://www.bitecode.dev/p/relieving-your-python-packaging-p...) is the Pareto solution. That is, the solution that has the best ratio of effort, reward, but also the lower risk of failure. It's not no risk, but I've been helping beginners for 15 years with Python, and have tried everything you can think of. It does imply, on linux, to limit yourself to the choices of Python you can install. This constraint is, for most people, preferable than the alternative, even if it gets frustrating to our geeky soul.

hot_gril 3y ago

The silver bullet is Docker. It's not the best solution in every situation, but at least it'll work. For your own projects built from scratch (rather than big multi-dep projects off GitHub), there's a smaller learning curve if you go to the vanilla https://www.python.org/downloads/ https://www.python.org/downloads/ , install the latest, and use the included pip to install packages. That'll probably get you very far. It's not like the older days when you needed both Py3 and Py2. For experts working frequently in Python, tools like Pyenv can make more sense.