4 ms·

What's the cliff notes of the right way / where's the signal in noise? I don't write Python, but Python is like Java these days, it finds you.

BiteCode_dev 4y ago
In short, a huge number of packaging problems come not from packaging, but from the way you install, configure and call python. Unfortunatly, everybody and their mother tell you how to do this, and you'll find 1000 articles to get you in trouble, but very few to tell you what you should actually do. The full story is pretty long and most people don't have the time and resources to learn everything about it. The easiest way is to follow a receipe for the least pain possible, which includes limiting yourself to a single path for bootstrapping python. Here is the one I currently recommend to my teams at the end of the comment: https://news.ycombinator.com/item?id=32805483#32807220 https://news.ycombinator.com/item?id=32805483#32807220 It can't solve as much as knowing the whole deal, but it will remove a lot of ways to shoot yourself in the foot, and work around a lot of the legacy issues. And above all, it's easy to keep around and to apply. I noticed that anything more complicated will lead to people, so I removed a lot of things from it years after years (like -m, pipx, etc). Of course, this will not solve the setup that are already in place, which you will likely have. I can't provide a generic way to solve that, there are too many possibilities. Hopefully the community will eventually solve most of this by providing better default in the future, but this is an excrutiatingly slow process.
- aragilar 4y ago
  My impression is Python packaging is stuck in a local maximum, and that any attempt to move out of it will be as disruptive as 2->3. Dropping support for conda et al will only split the ecosystem, hasten the rise of Julia, and make Python packaging more acrimonious. I like your simplified approach (and take a similar approach where I can), but much of the value of Python comes from its ability to be inserted without too much fuss (assuming you'd build things from source) into an existing system. I'd suggest while wheels were a good idea at the time, they've become a parallel ecosystem and source of division.
  - BiteCode_dev 4y ago
    Not at all. As I said, most packaging problems don't come from packaging itself, but for bootstrapping. Symptoms from a terrible bootstrapping situation creep up and they come out in packaging. Unfortunaly the solution is of the worst nature possible for a FOSS project, because not technical: it's policital. This is the kryptonite of volunteer efforts. The community has to get organised around one way to install, configure and run python. But it's really hard to do, and requires to synchronized a lot of different group of people, efforts, and make sure everyone are on the same page. E.G, of a single point: To chose which version of python to run, the installer for windows provides the "py" command, anaconda provides the conda prompt, but the windows store provides a suffixed command like most unix do. Most people are not aware of those differences, and when they attempt to follow a tutorial that has been written by somebody knowing only one of those combinations, the person will fail. Either with a "command not found", a broken install, "import error", a "syntax error" or installing things for one version of python and running the other version by mistake. This will lead the person to believe packaging is broken, while it's not. The way we install and run python is broken. But how do you solve this problem? By engaging in a long, frustrating and thankless talk with each of the parties involved. There will be debates, people will talk about taste, diversity, backward compatibility and so on. It will be a nightmare. The alternative would be for the core devs to come up with a single recommended setup for all plateforms, with a unified configuration, and promote that massively. This will cause another huge political problem: you now have 10000 of tutorials that are referencing one of the old ways, and you will spend the next 20 years explaining to people any previous configuration are not supported. There will be a lot of complains from devs, and pressures from companies. Either way, you end up with a load of work, and unhappy people, which nobody wants to sign for. It would be better on the long run, but nobody wants to sacrifice themself to be hated in the end. And that's just one single point. You still have to solve homebrew troubles, linux distro splitting python and limiting versions, bootstrapping tooling outside of venv, solving the uncanny valley of venv, and so on. Note that we haven't even touched packaging yet.

BiteCode_dev 4y ago

In short, a huge number of packaging problems come not from packaging, but from the way you install, configure and call python. Unfortunatly, everybody and their mother tell you how to do this, and you'll find 1000 articles to get you in trouble, but very few to tell you what you should actually do. The full story is pretty long and most people don't have the time and resources to learn everything about it. The easiest way is to follow a receipe for the least pain possible, which includes limiting yourself to a single path for bootstrapping python. Here is the one I currently recommend to my teams at the end of the comment: https://news.ycombinator.com/item?id=32805483#32807220 https://news.ycombinator.com/item?id=32805483#32807220 It can't solve as much as knowing the whole deal, but it will remove a lot of ways to shoot yourself in the foot, and work around a lot of the legacy issues. And above all, it's easy to keep around and to apply. I noticed that anything more complicated will lead to people, so I removed a lot of things from it years after years (like -m, pipx, etc). Of course, this will not solve the setup that are already in place, which you will likely have. I can't provide a generic way to solve that, there are too many possibilities. Hopefully the community will eventually solve most of this by providing better default in the future, but this is an excrutiatingly slow process.

aragilar 4y ago
My impression is Python packaging is stuck in a local maximum, and that any attempt to move out of it will be as disruptive as 2->3. Dropping support for conda et al will only split the ecosystem, hasten the rise of Julia, and make Python packaging more acrimonious. I like your simplified approach (and take a similar approach where I can), but much of the value of Python comes from its ability to be inserted without too much fuss (assuming you'd build things from source) into an existing system. I'd suggest while wheels were a good idea at the time, they've become a parallel ecosystem and source of division.
- BiteCode_dev 4y ago
  Not at all. As I said, most packaging problems don't come from packaging itself, but for bootstrapping. Symptoms from a terrible bootstrapping situation creep up and they come out in packaging. Unfortunaly the solution is of the worst nature possible for a FOSS project, because not technical: it's policital. This is the kryptonite of volunteer efforts. The community has to get organised around one way to install, configure and run python. But it's really hard to do, and requires to synchronized a lot of different group of people, efforts, and make sure everyone are on the same page. E.G, of a single point: To chose which version of python to run, the installer for windows provides the "py" command, anaconda provides the conda prompt, but the windows store provides a suffixed command like most unix do. Most people are not aware of those differences, and when they attempt to follow a tutorial that has been written by somebody knowing only one of those combinations, the person will fail. Either with a "command not found", a broken install, "import error", a "syntax error" or installing things for one version of python and running the other version by mistake. This will lead the person to believe packaging is broken, while it's not. The way we install and run python is broken. But how do you solve this problem? By engaging in a long, frustrating and thankless talk with each of the parties involved. There will be debates, people will talk about taste, diversity, backward compatibility and so on. It will be a nightmare. The alternative would be for the core devs to come up with a single recommended setup for all plateforms, with a unified configuration, and promote that massively. This will cause another huge political problem: you now have 10000 of tutorials that are referencing one of the old ways, and you will spend the next 20 years explaining to people any previous configuration are not supported. There will be a lot of complains from devs, and pressures from companies. Either way, you end up with a load of work, and unhappy people, which nobody wants to sign for. It would be better on the long run, but nobody wants to sacrifice themself to be hated in the end. And that's just one single point. You still have to solve homebrew troubles, linux distro splitting python and limiting versions, bootstrapping tooling outside of venv, solving the uncanny valley of venv, and so on. Note that we haven't even touched packaging yet.