16 ms·

Python desperately needs a better packaging solution than pip from Python Software Foundation. Stop all new features until this - the biggest pain point for Pyt

Python desperately needs a better packaging solution than pip from Python Software Foundation. Stop all new features until this - the biggest pain point for Python across all expertise levels from newbies to experienced Python programmers - is officially solved. No pipenv, no poetry, no conda, etc from third party devs who sometimes get tired of the pressure [1]. It is immense. Combine all this virtualenv stuff with great dependency management into one tool. In the spirit of Python, there should be only one way to package, manage dependencies and virtual environments. It should be super clean (see Go). Python needs a BDFL to enforce the hive mind. It is a fucking mess and it's unacceptable. Python is otherwise the most incredible language of our times. It is so far ahead of anything out there for general purpose use. Relevant xkcd: https://xkcd.com/1987/ https://xkcd.com/1987/ [1] https://github.com/pypa/pipenv/issues/4058 https://github.com/pypa/pipenv/issues/4058

nvrspyx 7y ago
I haven't looked much into it myself, but I've heard good things about Poetry [0]. 0: https://python-poetry.org/ https://python-poetry.org/
- rattray 7y ago
  This looks great. Can anyone who's used both comment on how poetry compares to yarn[0]? [0] https://yarnpkg.com/ https://yarnpkg.com/
  - franey 7y ago
    I think of Poetry as the Yarn of Python packaging for two main reasons (apart from package locking, which is the biggest similarity): the commands are more intuitive and the lockfile starts with a different letter from the dependency file. I don't know how many times I open package-lock.json instead of package.json because I tab-complete from my shell. Same with Pipfile and Pipfile.lock for anyone using Pipenv. I wrote this post[0] that compares Poetry and Pipenv if you're interested and talks about some of their differences. Poetry has both add and install commands, whereas Pipenv only has an install command, for instance. [0] https://johnfraney.ca/posts/2019/03/06/pipenv-poetry-benchmarks-ergonomics/ https://johnfraney.ca/posts/2019/03/06/pipenv-poetry-benchma...
    - rattray 7y ago
      Thanks, that's helpful and illustrative! Is poetry as good as yarn?
- joshvm 7y ago
  Poetry is good, but you should be aware that people with legacy versions of pip (I think before 19, see PEP 517) won't be able to pip install from your repository directly. That means anyone using pip installed with their package manager (up to the lastest version of Debian/Ubuntu in fact, which only provide v18). This doesn't affect installs from PyPI of course. So either you just accept this, or you also maintain a setup.py file as well. That's fine, but Poetry doesn't automatically bump version numbers in setup.py (as far as I know) so you need to do a couple of extra steps when you build. On the other hand, setting up a deploy key to PyPI and then using poetry build, poetry publish is very easy compared to using twine which I found often did daft things like upload mixed versions to PyPI unless the build directory was clean. Poetry seems to handle that a lot better.
  - kapilvt 7y ago
    having just converted a large project/monorepo to poetry, its pretty easy to generate setup.py/requirements from pyproject.toml https://github.com/cloud-custodian/cloud-custodian/blob/master/tools/dev/poetrypkg.py https://github.com/cloud-custodian/cloud-custodian/blob/mast... note most of this just working around poetry for multiple packages in a single repo, with interdependencies. flip side, our pip/setuptools users building trunk from ci are unaffected. that said there are open questions on poetry (governance/dev community), but the experience is overall pretty solid with a some minor rough edges, and the fact that it has a usable api makes those pretty minor. in contrast the only api on pip is cli arguments for subprocess/fork.
deleted 7y ago
[deleted]
dralley 7y ago
None of that is "necessary". Rust doesn't have a BDFL and they've gotten along fine. But I agree that the current situation is a mess. The "official" solutions are suboptimal and none of the unofficial solutions, which are pretty good, have enough clout to meaningfully simplify the ecosystem (held hostage by the majority of users with just pip). So the ecosystem is kind of frozen in place. PyPA doesn't seem to have the will/capacity to do anything more than very small, incremental changes.
- brettcannon 7y ago
  It's all about capacity, not will. The PyPA as a whole knows what is lacking, but when you're all just a bunch of volunteers it's hard to tackle massive projects that take potentially years to complete and then even longer to see gain traction in the community when you are doing it all in your spare time. And that's if you don't burn out before you finish (e.g. calling projects "garbage" is not motivating to the few people volunteering their time to try and fix the situation while keeping what is there afloat and running).
  - billfruit 7y ago
    Yes. I am not familiar with the python development process, but atleast have they an announced roadmap to move to something better that the present suboptimal methods.
Supermancho 7y ago
> Python is otherwise the most incredible language of our times By what metric(s)? It's based on statements, and not only on expressions. This is a fundamental issue that it has yet to grow out of.
- Supermancho 7y ago
  Downvotes because of what I think about the language. Zealotry at it's finest.
  - mixmastamyk 7y ago
    Welcome to HN. They aren't interested in your flippant opinions, man.
    - Supermancho 7y ago
      The opinions are serious on both sides. If I don't provide criticism, it's a strawman post. Everyone is wrong in criticism here, except when the specific topic is criticism and for that, we get tons of noise about Python. At the same time, it's not a open forum when one side is persecuted.
      - mixmastamyk 7y ago
        To expand, you didn't give enough information to decide whether your comment made sense or not. That makes it non-useful in the end. Such comments are downvoted here. I had to learn this myself, as I like to be sarcastic most of the time. If you don't have something substantial, cited, and/or supported to say, and it is also negative, better to not write anything.
no_wizard 7y ago
I agree. This really seems like high impact too. A first party, supported solution would solve this mess. Getting the Python dependency build story under control I would include in this category. In this department, I think Conda shows the way. They leverage Azure Pipelines to build much of the c dependencies ahead of time, and just send binaries down the pipe (at least, that's the high level overview as I understand it). I can't see a world where the Python Software Foundation can't get support for this, I think they have the contacts at the big companies to get it funded. The lead Python has on machine learning, data science, and scientific computing won't last forever (though, I think its safe to say no matter what, it ain't going anywhere.
- brettcannon 7y ago
  > I can't see a world where the Python Software Foundation can't get support for this, I think they have the contacts at the big companies to get it funded. You might be surprised at how hard it is to get money donated for this sort of thing. Large companies often have their own build infrastructure and thus build things for themselves from scratch, to the point of building their own build toolchains (e.g. look at Google and bazel, Facebook and buck). So the way they handle packaging means how the overall packaging ecosystem functions does not impact them like it does individuals or smaller companies. This is also why companies only started funding PyPI security work about two years ago; downloading source securely from PyPI is important to big companies as that's where they grab the source that they compile. But funding pip improvements only just happened in Decembe, starting with the new pip resolver work got money (https://pyfound.blogspot.com/2020/03/new-pip-resolver-to-roll-out-this-year.html https://pyfound.blogspot.com/2020/03/new-pip-resolver-to-rol...). And all that money came from Mozilla and the Chan Zuckerberg Initiative; two huge non-profits, not companies. Add on to the fact that the PSF is projected to have a $627,000 USD loss this year due to PyCon US 2020 being cancelled and it makes finding funding really hard (https://pyfound.blogspot.com/2020/03/psfs-projected-2020-financial-outcome.html https://pyfound.blogspot.com/2020/03/psfs-projected-2020-fin...). Python overall very much runs on volunteers, but the packaging system especially. The core team of Python itself might generously have a total of 3 devs/week being paid to work on CPython spread across all contributors (IOW it is nowhere near as impactful as 3 actual full-time devs who aren't context-switching constantly between "work work" and "Python work"). But for Python packaging in general? If you don't count conda I think packaging has less than even 3. But if anyone wants to donate to try and rectify this, please go to https://psfmember.org/civicrm/contribute/transact?reset=1&id=13 https://psfmember.org/civicrm/contribute/transact?reset=1&id... and donate to the packaging WG!
j88439h84 7y ago
Poetry is a modern packaging system for Python. It's really good. I hope everyone converges on it. http://python-poetry.org/ http://python-poetry.org/
- xzel 7y ago
  I just started up a project using this. It’s pretty good so far. The documentation on common tasks is a little bit lacking. VS Code integration is sort of broken in a few random ways (for example if I try and do a find all function definition it just hangs). I also wish there was a npm style script section in the definitions. I do think it really just works though and I think it’s a great solution compared to requirements.txt.
  - j88439h84 7y ago
    > I also wish there was a npm style script section in the definitions. Is that like [tool.poetry.scripts] ?
    - xzel 7y ago
      I don't think so. From their docs: > This section describe the scripts or executable that will be installed when installing the package I want something such that I can run `poetry run script_name` and it will run that command line in the poetry context. Right now I either rely on my bash history or use a bash script to mimic the behavior of npm / yarn scripts. I want to turn this `poetry run python project_name/main.py` into an alias like `poetry run main` or `poetry run dev`.
      - j88439h84 7y ago
        That's what tool.poetry.scripts does.
        xzel 7y ago
        O thats great. So just weirdly worded on the docs / I didn't understand. Thanks!
tych0 7y ago
FWIW, the issue you linked to was closed in favor of https://github.com/pypa/pipenv/issues/3369 https://github.com/pypa/pipenv/issues/3369. Looks like Canonical is paying him now.
- aprdm 7y ago
  I had some really bad experiences with pipenv. I don't think it is the way forward. It also seems like abandonware at the moment so I am glad there's at least work happening.
aprdm 7y ago
I totally agree with this. Don't forget all the bespoke systems that companies end up implementing because they either aren't aware of those other projects or they started doing it before some of those other projects existed or because they aren't a fit.
brettcannon 7y ago
I personally don't think that comic is fair to hold against pip or the packaging ecosystem. I did a separate post on deconstructing it at https://snarky.ca/deconstructing-xkcd-com-1987/ https://snarky.ca/deconstructing-xkcd-com-1987/.
harikb 7y ago
I teach Python to some middle school kids. Pip vs Pip3 and PYTHONPATH wasted entirety of two class sessions. By the way, Python should go beyond Go. It should single binary the someone can install without root privileges. The fact that I have hundreds of Chromebooks at my school and I can’t use any of them - school IT, correctly, doesn’t want me mucking with advanced settings.
- brettcannon 7y ago
  People would love to see that happen, but it requires time and effort from people to build such a solution. Some people are trying, e.g. https://pyoxidizer.readthedocs.io/ https://pyoxidizer.readthedocs.io/ and https://briefcase.readthedocs.io/ https://briefcase.readthedocs.io/ as well as stalwarts like PyInstaller and such. But as I said, it takes time and effort and there's only so much of that when it's being done by volunteers in their evenings and weekends. And as an aside, I would suggest teaching them `python -m pip` over `pip3`: https://snarky.ca/why-you-should-use-python-m-pip/ https://snarky.ca/why-you-should-use-python-m-pip/.
  - judge2020 7y ago
    Thanks for the links - pyoxidizer looks promising with some good docs and being built on rust (but it won't work with QT's PySide2 likely until the next major release[0]), while briefcase looks to have been tailored to "beeware" usage and is lacking any standalone usage information. Hoping both these work out, PyInstaller works but other options would be welcomed. 0: https://github.com/indygreg/PyOxidizer/issues/228#issuecomment-601953256 https://github.com/indygreg/PyOxidizer/issues/228#issuecomme...
  - j88439h84 7y ago
    I really hope the PSF can help the community move toward Poetry. It's a game changer relative to all the alternatives.
  - harikb 7y ago
    Thanks for the link. About the lack of resources problem though, I have a few points to make. I think one of the unfortunate things that happened is indeed the arrival of new languages. Go, Rust, Swift, whatever is taking away a bunch of senior developers to work on those. Even though Python is still "most wanted" and "most popular", imho that does not reflect the reality among senior developers who are capable of taking on large tasks like this. Sometimes the frustration expressed by package management has more to do with build dependencies rather than the distribution itself. Like a library that has a C dependency. In this context, the world of Go is equally messy, when CGO is involved. Luckily in their case, most people do rewrite libraries in Pure Go.
brettcannon 7y ago
Do note that Python the language is developed by a separate group as Python the packaging ecosystem. So even when Guido was a BDFL he completely stayed out of the packaging situation. And there is a packaging BDFL, but there's a severe lack of time and effort in the form of volunteers to tackle a lot of the projects, big or small. Plus rolling these things out takes years and it takes even longer to gain traction as people are often reluctant to change their build tools once things work. As an example of timelines, take what my blog post covers. PEP 518 and the concept of pyproject.toml was conceived in 2016, four years ago. Getting PEP 517 finalized and it all worked into pip didn't happen until pip 19 in January 2019, over a year ago (https://pip.pypa.io/en/stable/news/#id203 https://pip.pypa.io/en/stable/news/#id203). And a year later there's still enough of a need to educate folks that I wrote this blog post to help get the word out. IOW the timelines are huge and the pay-off is way down the line. And that's if you manage to be motivated enough to do the work. Burn-out is a huge problem in the packaging world. I mean reading that you think "it is a fucking mess and it's unacceptable" doesn't motivate me and others to keep trying to improve things if that's how us volunteers will be treated for years to come while we try to rectify the situation. Add on to the fact that we aren't getting paid for this and it makes the idea of pulling out my laptop tonight to work on some packaging stuff feel rather pointless. So please be understanding that people are working on things as best they can and that overt negativity about it all will work against you getting what you want.
- fermienrico 7y ago
  Definitely, it is always good to keep in the back of our minds that FOSS is developed by people in their free time and on a volunteering basis. No one owes us anything. No one should feel entitled. I am sorry if my complains come out as such. Python runs the world and it is because of the contributions of many amazing people.
  - idoubtit 7y ago
    > Definitely, it is always good to keep in the back of our minds that FOSS is developed by people in their free time and on a volunteering basis. Please don't over-generalize. Volunteering is not the only way FOSS is developed. E.g. the Linux kernel, Gitlab, Firefox, Golang main tools…
btashton 7y ago
How is go the poster child for clean dependency management? Every release since they started doing modules the defaults change and break some workflow. Add on top of that it activity pushed vendoring forever. I'm not super happy with package management under python but at least pip does not break every release.
earthboundkid 7y ago
I am an experienced Python programmer. I often have to collaborate with people who are not. There is no greater hell than talking them through installing a working Python environment. It’s impossible. Meanwhile, the pip project is full of developers who tell you that your problem is not a problem and you’re discouraging people by pointing out that they’re blame shifting: https://github.com/pypa/pip/issues/4995 https://github.com/pypa/pip/issues/4995 I honestly have zero faith that things will get better without a radical change in leadership. Pyproject.toml is a good step in the right direction, but it’s been around for years. They had time for adding the walrus operator, but not for replacing the setuptools, venv, pip Cerberus.
- j0057 7y ago
  I see pip's behaviour being called "bizarre" and "baffling", an expectation that the "fix" should be easy (it never is: you "simply" add a flag, now some code paths bifurcate, now you need more tests, documentation should be updated and reviewed, etc.), no pull request, and the pip developer is still patiently explaining why pip's behaviour is not meeting with expectations. But really, it takes a lot of entitlement to expect really anything whatsoever out of somebody else's free time.
  - twic 7y ago
    I see a weird, surprising, unhelpful design, and a lot of critical comments being deleted.
    - j0057 7y ago
      Weird, surprising and unhelpful are value judgements on your part. It could be a security feature to someone else! The hidden (not deleted) comments are hidden not because they contain criticism, but because of an unnecessarily harsh tone, which is unhelpful to anyone wanting to read up on the discussion at hand, and definitely not helpful for the motivation of anybody currently doing the unthankful work of maintaining pip. They do a very difficult job, where they have to deal with 20+ years of legacy, many different operating systems, many different Python distributions, many different skill levels in their user base, and even having to fight OSS fatigue due to unconstructive criticism. Pip is as good as it is due to their hard work, and we should be thankful to them.
pjmlp 7y ago
Python is a nice scripting and introduction to programming language, but "It is so far ahead of anything out there for general purpose use." it is not. Maybe when it has a JIT/AOT story that matches Common Lisp, Scheme, Julia, Smalltalk, JavsScript, or I am able to do refactorings with the easiness that it works on Smalltalk, I can re-consider my opinion.
- mixmastamyk 7y ago
  What the OP probably means is that it is hard to beat Python on its knockout combo of pragmatism, readability, stdlib, and third-party support. Does that mean its the best at everything? Of course not. It is second best at most of the important things however, a more unique position than it sounds.
  - pjmlp 7y ago
    That being the case, then Java and .NET come to mind.
    - mixmastamyk 7y ago
      Those are great when you need big or fast, but py is much better at the smaller, easier to write end.
      - pjmlp 7y ago
        Ah, so Python is for small and slow, doesn't seem to be the best in everything after all. Which brings us back to "nice scripting and introduction to programming language".
        wildfire 7y ago
        Yup. Clearly no one is going to be able to convince you otherwise, so is there much point in you continuing to read this thread?
fulafel 7y ago
Containers solve this for non-GUI apps, and pip still works for 90+% of use cases. Not sure if yet another solution would improve the situation, would just be more fragmentation and competition with more general (non-Python-specific) solutions.
codeduck 7y ago
> Python is otherwise the most incredible language of our times This is a strong claim. Ruby could arguably make the same claim. So could Elixir.
- BiteCode_dev 7y ago
  Given none of them are used outside of their specialty, no. For Python, it's the reversed: it's used everywhere, except for a few topics. It's the only scripting language like that.
  - ProZsolt 7y ago
    People have this bias that Ruby only used with Rails. Yes, Rails made it popular in the first place, but it has overgrown it. Other than data science/machine learning (because of the lack of libraries/frameworks) Ruby used in all over the place, just like Python. Ruby basically ruled the DevOps space before Go and still very relevant. Yes, Python is winning the popularity battle no doubt on that, but that nothing to do to the language or the surrounding tooling itself. It's like saying VHS is an incredible format compare to Betamax(Yes I'm old :D). I used Ruby (not Rails) and Python for many years by now. I think Ruby and its tooling are far better than Python's. But if anybody asked me which language should they learn as their first, it would be Python, just because there will be more jobs available.
avip 7y ago
python distribution is so annoying I always wrap it in docker and call it the day. I can never trust pip for on-premise installations of python code. Even the npm toolchain madness is somewhat more consolidated and organized.
collyw 7y ago
Yes the python environment space has exploded and now feels like JavaScript with multiple competing solutions and no obvious go-to one.
jakearmitage 7y ago
Wonderful language, crippling ecosystem.

nvrspyx 7y ago

I haven't looked much into it myself, but I've heard good things about Poetry [0]. 0: https://python-poetry.org/ https://python-poetry.org/

rattray 7y ago
This looks great. Can anyone who's used both comment on how poetry compares to yarn[0]? [0] https://yarnpkg.com/ https://yarnpkg.com/
- franey 7y ago
  I think of Poetry as the Yarn of Python packaging for two main reasons (apart from package locking, which is the biggest similarity): the commands are more intuitive and the lockfile starts with a different letter from the dependency file. I don't know how many times I open package-lock.json instead of package.json because I tab-complete from my shell. Same with Pipfile and Pipfile.lock for anyone using Pipenv. I wrote this post[0] that compares Poetry and Pipenv if you're interested and talks about some of their differences. Poetry has both add and install commands, whereas Pipenv only has an install command, for instance. [0] https://johnfraney.ca/posts/2019/03/06/pipenv-poetry-benchmarks-ergonomics/ https://johnfraney.ca/posts/2019/03/06/pipenv-poetry-benchma...
  - rattray 7y ago
    Thanks, that's helpful and illustrative! Is poetry as good as yarn?
joshvm 7y ago
Poetry is good, but you should be aware that people with legacy versions of pip (I think before 19, see PEP 517) won't be able to pip install from your repository directly. That means anyone using pip installed with their package manager (up to the lastest version of Debian/Ubuntu in fact, which only provide v18). This doesn't affect installs from PyPI of course. So either you just accept this, or you also maintain a setup.py file as well. That's fine, but Poetry doesn't automatically bump version numbers in setup.py (as far as I know) so you need to do a couple of extra steps when you build. On the other hand, setting up a deploy key to PyPI and then using poetry build, poetry publish is very easy compared to using twine which I found often did daft things like upload mixed versions to PyPI unless the build directory was clean. Poetry seems to handle that a lot better.
- kapilvt 7y ago
  having just converted a large project/monorepo to poetry, its pretty easy to generate setup.py/requirements from pyproject.toml https://github.com/cloud-custodian/cloud-custodian/blob/master/tools/dev/poetrypkg.py https://github.com/cloud-custodian/cloud-custodian/blob/mast... note most of this just working around poetry for multiple packages in a single repo, with interdependencies. flip side, our pip/setuptools users building trunk from ci are unaffected. that said there are open questions on poetry (governance/dev community), but the experience is overall pretty solid with a some minor rough edges, and the fact that it has a usable api makes those pretty minor. in contrast the only api on pip is cli arguments for subprocess/fork.

deleted 7y ago

[deleted]

dralley 7y ago

None of that is "necessary". Rust doesn't have a BDFL and they've gotten along fine. But I agree that the current situation is a mess. The "official" solutions are suboptimal and none of the unofficial solutions, which are pretty good, have enough clout to meaningfully simplify the ecosystem (held hostage by the majority of users with just pip). So the ecosystem is kind of frozen in place. PyPA doesn't seem to have the will/capacity to do anything more than very small, incremental changes.

brettcannon 7y ago
It's all about capacity, not will. The PyPA as a whole knows what is lacking, but when you're all just a bunch of volunteers it's hard to tackle massive projects that take potentially years to complete and then even longer to see gain traction in the community when you are doing it all in your spare time. And that's if you don't burn out before you finish (e.g. calling projects "garbage" is not motivating to the few people volunteering their time to try and fix the situation while keeping what is there afloat and running).
- billfruit 7y ago
  Yes. I am not familiar with the python development process, but atleast have they an announced roadmap to move to something better that the present suboptimal methods.

Supermancho 7y ago

> Python is otherwise the most incredible language of our times By what metric(s)? It's based on statements, and not only on expressions. This is a fundamental issue that it has yet to grow out of.

Supermancho 7y ago
Downvotes because of what I think about the language. Zealotry at it's finest.
- mixmastamyk 7y ago
  Welcome to HN. They aren't interested in your flippant opinions, man.
  - Supermancho 7y ago
    The opinions are serious on both sides. If I don't provide criticism, it's a strawman post. Everyone is wrong in criticism here, except when the specific topic is criticism and for that, we get tons of noise about Python. At the same time, it's not a open forum when one side is persecuted.
    - mixmastamyk 7y ago
      To expand, you didn't give enough information to decide whether your comment made sense or not. That makes it non-useful in the end. Such comments are downvoted here. I had to learn this myself, as I like to be sarcastic most of the time. If you don't have something substantial, cited, and/or supported to say, and it is also negative, better to not write anything.

no_wizard 7y ago

I agree. This really seems like high impact too. A first party, supported solution would solve this mess. Getting the Python dependency build story under control I would include in this category. In this department, I think Conda shows the way. They leverage Azure Pipelines to build much of the c dependencies ahead of time, and just send binaries down the pipe (at least, that's the high level overview as I understand it). I can't see a world where the Python Software Foundation can't get support for this, I think they have the contacts at the big companies to get it funded. The lead Python has on machine learning, data science, and scientific computing won't last forever (though, I think its safe to say no matter what, it ain't going anywhere.

brettcannon 7y ago
> I can't see a world where the Python Software Foundation can't get support for this, I think they have the contacts at the big companies to get it funded. You might be surprised at how hard it is to get money donated for this sort of thing. Large companies often have their own build infrastructure and thus build things for themselves from scratch, to the point of building their own build toolchains (e.g. look at Google and bazel, Facebook and buck). So the way they handle packaging means how the overall packaging ecosystem functions does not impact them like it does individuals or smaller companies. This is also why companies only started funding PyPI security work about two years ago; downloading source securely from PyPI is important to big companies as that's where they grab the source that they compile. But funding pip improvements only just happened in Decembe, starting with the new pip resolver work got money (https://pyfound.blogspot.com/2020/03/new-pip-resolver-to-roll-out-this-year.html https://pyfound.blogspot.com/2020/03/new-pip-resolver-to-rol...). And all that money came from Mozilla and the Chan Zuckerberg Initiative; two huge non-profits, not companies. Add on to the fact that the PSF is projected to have a $627,000 USD loss this year due to PyCon US 2020 being cancelled and it makes finding funding really hard (https://pyfound.blogspot.com/2020/03/psfs-projected-2020-financial-outcome.html https://pyfound.blogspot.com/2020/03/psfs-projected-2020-fin...). Python overall very much runs on volunteers, but the packaging system especially. The core team of Python itself might generously have a total of 3 devs/week being paid to work on CPython spread across all contributors (IOW it is nowhere near as impactful as 3 actual full-time devs who aren't context-switching constantly between "work work" and "Python work"). But for Python packaging in general? If you don't count conda I think packaging has less than even 3. But if anyone wants to donate to try and rectify this, please go to https://psfmember.org/civicrm/contribute/transact?reset=1&id=13 https://psfmember.org/civicrm/contribute/transact?reset=1&id... and donate to the packaging WG!

j88439h84 7y ago

Poetry is a modern packaging system for Python. It's really good. I hope everyone converges on it. http://python-poetry.org/ http://python-poetry.org/

xzel 7y ago
I just started up a project using this. It’s pretty good so far. The documentation on common tasks is a little bit lacking. VS Code integration is sort of broken in a few random ways (for example if I try and do a find all function definition it just hangs). I also wish there was a npm style script section in the definitions. I do think it really just works though and I think it’s a great solution compared to requirements.txt.
- j88439h84 7y ago
  > I also wish there was a npm style script section in the definitions. Is that like [tool.poetry.scripts] ?
  - xzel 7y ago
    I don't think so. From their docs: > This section describe the scripts or executable that will be installed when installing the package I want something such that I can run `poetry run script_name` and it will run that command line in the poetry context. Right now I either rely on my bash history or use a bash script to mimic the behavior of npm / yarn scripts. I want to turn this `poetry run python project_name/main.py` into an alias like `poetry run main` or `poetry run dev`.
    - j88439h84 7y ago
      That's what tool.poetry.scripts does.
      - xzel 7y ago
        O thats great. So just weirdly worded on the docs / I didn't understand. Thanks!

tych0 7y ago

FWIW, the issue you linked to was closed in favor of https://github.com/pypa/pipenv/issues/3369 https://github.com/pypa/pipenv/issues/3369. Looks like Canonical is paying him now.

aprdm 7y ago
I had some really bad experiences with pipenv. I don't think it is the way forward. It also seems like abandonware at the moment so I am glad there's at least work happening.

aprdm 7y ago

I totally agree with this. Don't forget all the bespoke systems that companies end up implementing because they either aren't aware of those other projects or they started doing it before some of those other projects existed or because they aren't a fit.

brettcannon 7y ago

I personally don't think that comic is fair to hold against pip or the packaging ecosystem. I did a separate post on deconstructing it at https://snarky.ca/deconstructing-xkcd-com-1987/ https://snarky.ca/deconstructing-xkcd-com-1987/.

harikb 7y ago

I teach Python to some middle school kids. Pip vs Pip3 and PYTHONPATH wasted entirety of two class sessions. By the way, Python should go beyond Go. It should single binary the someone can install without root privileges. The fact that I have hundreds of Chromebooks at my school and I can’t use any of them - school IT, correctly, doesn’t want me mucking with advanced settings.

brettcannon 7y ago
People would love to see that happen, but it requires time and effort from people to build such a solution. Some people are trying, e.g. https://pyoxidizer.readthedocs.io/ https://pyoxidizer.readthedocs.io/ and https://briefcase.readthedocs.io/ https://briefcase.readthedocs.io/ as well as stalwarts like PyInstaller and such. But as I said, it takes time and effort and there's only so much of that when it's being done by volunteers in their evenings and weekends. And as an aside, I would suggest teaching them `python -m pip` over `pip3`: https://snarky.ca/why-you-should-use-python-m-pip/ https://snarky.ca/why-you-should-use-python-m-pip/.
- judge2020 7y ago
  Thanks for the links - pyoxidizer looks promising with some good docs and being built on rust (but it won't work with QT's PySide2 likely until the next major release[0]), while briefcase looks to have been tailored to "beeware" usage and is lacking any standalone usage information. Hoping both these work out, PyInstaller works but other options would be welcomed. 0: https://github.com/indygreg/PyOxidizer/issues/228#issuecomment-601953256 https://github.com/indygreg/PyOxidizer/issues/228#issuecomme...
- j88439h84 7y ago
  I really hope the PSF can help the community move toward Poetry. It's a game changer relative to all the alternatives.
- harikb 7y ago
  Thanks for the link. About the lack of resources problem though, I have a few points to make. I think one of the unfortunate things that happened is indeed the arrival of new languages. Go, Rust, Swift, whatever is taking away a bunch of senior developers to work on those. Even though Python is still "most wanted" and "most popular", imho that does not reflect the reality among senior developers who are capable of taking on large tasks like this. Sometimes the frustration expressed by package management has more to do with build dependencies rather than the distribution itself. Like a library that has a C dependency. In this context, the world of Go is equally messy, when CGO is involved. Luckily in their case, most people do rewrite libraries in Pure Go.

brettcannon 7y ago

Do note that Python the language is developed by a separate group as Python the packaging ecosystem. So even when Guido was a BDFL he completely stayed out of the packaging situation. And there is a packaging BDFL, but there's a severe lack of time and effort in the form of volunteers to tackle a lot of the projects, big or small. Plus rolling these things out takes years and it takes even longer to gain traction as people are often reluctant to change their build tools once things work. As an example of timelines, take what my blog post covers. PEP 518 and the concept of pyproject.toml was conceived in 2016, four years ago. Getting PEP 517 finalized and it all worked into pip didn't happen until pip 19 in January 2019, over a year ago (https://pip.pypa.io/en/stable/news/#id203 https://pip.pypa.io/en/stable/news/#id203). And a year later there's still enough of a need to educate folks that I wrote this blog post to help get the word out. IOW the timelines are huge and the pay-off is way down the line. And that's if you manage to be motivated enough to do the work. Burn-out is a huge problem in the packaging world. I mean reading that you think "it is a fucking mess and it's unacceptable" doesn't motivate me and others to keep trying to improve things if that's how us volunteers will be treated for years to come while we try to rectify the situation. Add on to the fact that we aren't getting paid for this and it makes the idea of pulling out my laptop tonight to work on some packaging stuff feel rather pointless. So please be understanding that people are working on things as best they can and that overt negativity about it all will work against you getting what you want.

fermienrico 7y ago
Definitely, it is always good to keep in the back of our minds that FOSS is developed by people in their free time and on a volunteering basis. No one owes us anything. No one should feel entitled. I am sorry if my complains come out as such. Python runs the world and it is because of the contributions of many amazing people.
- idoubtit 7y ago
  > Definitely, it is always good to keep in the back of our minds that FOSS is developed by people in their free time and on a volunteering basis. Please don't over-generalize. Volunteering is not the only way FOSS is developed. E.g. the Linux kernel, Gitlab, Firefox, Golang main tools…

btashton 7y ago

How is go the poster child for clean dependency management? Every release since they started doing modules the defaults change and break some workflow. Add on top of that it activity pushed vendoring forever. I'm not super happy with package management under python but at least pip does not break every release.

earthboundkid 7y ago

I am an experienced Python programmer. I often have to collaborate with people who are not. There is no greater hell than talking them through installing a working Python environment. It’s impossible. Meanwhile, the pip project is full of developers who tell you that your problem is not a problem and you’re discouraging people by pointing out that they’re blame shifting: https://github.com/pypa/pip/issues/4995 https://github.com/pypa/pip/issues/4995 I honestly have zero faith that things will get better without a radical change in leadership. Pyproject.toml is a good step in the right direction, but it’s been around for years. They had time for adding the walrus operator, but not for replacing the setuptools, venv, pip Cerberus.

j0057 7y ago
I see pip's behaviour being called "bizarre" and "baffling", an expectation that the "fix" should be easy (it never is: you "simply" add a flag, now some code paths bifurcate, now you need more tests, documentation should be updated and reviewed, etc.), no pull request, and the pip developer is still patiently explaining why pip's behaviour is not meeting with expectations. But really, it takes a lot of entitlement to expect really anything whatsoever out of somebody else's free time.
- twic 7y ago
  I see a weird, surprising, unhelpful design, and a lot of critical comments being deleted.
  - j0057 7y ago
    Weird, surprising and unhelpful are value judgements on your part. It could be a security feature to someone else! The hidden (not deleted) comments are hidden not because they contain criticism, but because of an unnecessarily harsh tone, which is unhelpful to anyone wanting to read up on the discussion at hand, and definitely not helpful for the motivation of anybody currently doing the unthankful work of maintaining pip. They do a very difficult job, where they have to deal with 20+ years of legacy, many different operating systems, many different Python distributions, many different skill levels in their user base, and even having to fight OSS fatigue due to unconstructive criticism. Pip is as good as it is due to their hard work, and we should be thankful to them.

pjmlp 7y ago

Python is a nice scripting and introduction to programming language, but "It is so far ahead of anything out there for general purpose use." it is not. Maybe when it has a JIT/AOT story that matches Common Lisp, Scheme, Julia, Smalltalk, JavsScript, or I am able to do refactorings with the easiness that it works on Smalltalk, I can re-consider my opinion.

mixmastamyk 7y ago
What the OP probably means is that it is hard to beat Python on its knockout combo of pragmatism, readability, stdlib, and third-party support. Does that mean its the best at everything? Of course not. It is second best at most of the important things however, a more unique position than it sounds.
- pjmlp 7y ago
  That being the case, then Java and .NET come to mind.
  - mixmastamyk 7y ago
    Those are great when you need big or fast, but py is much better at the smaller, easier to write end.
    - pjmlp 7y ago
      Ah, so Python is for small and slow, doesn't seem to be the best in everything after all. Which brings us back to "nice scripting and introduction to programming language".
      - wildfire 7y ago
        Yup. Clearly no one is going to be able to convince you otherwise, so is there much point in you continuing to read this thread?

fulafel 7y ago

Containers solve this for non-GUI apps, and pip still works for 90+% of use cases. Not sure if yet another solution would improve the situation, would just be more fragmentation and competition with more general (non-Python-specific) solutions.

codeduck 7y ago

> Python is otherwise the most incredible language of our times This is a strong claim. Ruby could arguably make the same claim. So could Elixir.

BiteCode_dev 7y ago
Given none of them are used outside of their specialty, no. For Python, it's the reversed: it's used everywhere, except for a few topics. It's the only scripting language like that.
- ProZsolt 7y ago
  People have this bias that Ruby only used with Rails. Yes, Rails made it popular in the first place, but it has overgrown it. Other than data science/machine learning (because of the lack of libraries/frameworks) Ruby used in all over the place, just like Python. Ruby basically ruled the DevOps space before Go and still very relevant. Yes, Python is winning the popularity battle no doubt on that, but that nothing to do to the language or the surrounding tooling itself. It's like saying VHS is an incredible format compare to Betamax(Yes I'm old :D). I used Ruby (not Rails) and Python for many years by now. I think Ruby and its tooling are far better than Python's. But if anybody asked me which language should they learn as their first, it would be Python, just because there will be more jobs available.

avip 7y ago

python distribution is so annoying I always wrap it in docker and call it the day. I can never trust pip for on-premise installations of python code. Even the npm toolchain madness is somewhat more consolidated and organized.

collyw 7y ago

Yes the python environment space has exploded and now feels like JavaScript with multiple competing solutions and no obvious go-to one.

jakearmitage 7y ago

Wonderful language, crippling ecosystem.