15 ms·

Pypi.org is running a survey on the state of Python packaging

234 points by zbentley 4y ago

zbentley 4y ago
I imagine many of you have feedback that could be useful to folks making decisions about the future of Python packaging, a common subject of complaint in many discussions here. Remember not to just complain, but to offer specific problems/solutions--i.e. avoid statements like "virtualenvs suck, why can't it be like NPM?" and prefer instead feedback like "the difference between Python interpreter version and what virtualenv is being used causes confusion".
- Kwpolska 4y ago
  "virtualenvs suck, why can't it be like NPM?" is a specific problem and a specific solution. The problem being having to manage venvs (which have many gotchas and pitfalls, and no standarization), and the solution is to replace those with packages being installed into the project folder with standardized and well-known tools.
  - qbasic_forever 4y ago
    Keep an eye on https://peps.python.org/pep-0582/ https://peps.python.org/pep-0582/ it's a proposal to add local directory/node_modules-like behavior to package installs. It stalled out a few years ago but I heard there is a lot more discussion and push to get it in now. I think if this PEP makes it in then like 90% of people's pain with pip just completely goes away almost overnight. Love it or hate it the NPM/node_modules style of all dependencies dumped in a local directory solves a _ton_ of problems in the packaging world. It would go a long way towards making the experience much smoother for most python users.
  - jogjayr 4y ago
    I did a proof-of-concept for this a few years ago: https://github.com/jogjayr/pykg https://github.com/jogjayr/pykg It literally uses npm, the NPM registry, and node_modules for Python dependency management.
clintonb 4y ago
The survey is at https://www.surveymonkey.co.uk/r/M5XKQCT https://www.surveymonkey.co.uk/r/M5XKQCT.
- MonkeyMalarky 4y ago
  Seems more oriented to (potential) contributors than end users of the packaging system. Who cares about mission statements and inclusivity, secure funding and pay developers to make the tools.
  - woodruffw 4y ago
    > Who cares about mission statements and inclusivity, secure funding and pay developers to make the tools. These are connected things. I maintain a PyPA member project (and contribute to many others), and the latter is aided by the former: the mission statement keeps the community organized around shared goals (such as standardizing Python's packaging tooling), and inclusivity insures a healthy and steady flow of new contributors (and potential corporate funding sources).
    - electromech 4y ago
      > I maintain a PyPA member project (and contribute to many others) THANK YOU! > keeps the community organized around shared goals (such as standardizing Python's packaging tooling) Personally I felt some disconnect between "package manager for all" and the need for "standardizing Python's packaging tooling." Yes, communities should be welcoming and friendly to everyone, AND the community should have clear expectations for best practices that members should follow. E.g., is an experienced female developer more likely to give up on contributing because she couldn't find a local meetup, or because she didn't know whether to create pyproject.toml vs requirements.txt? In some sense, the bigger and more diverse the community, the greater the need for a clear, solid foundation. IDK if that's remotely clear; it's just a feeling I had going through some of those questions.
  - nomdep 4y ago
    The PSF are not engineers looking for a better developer experience, but politicians looking for power. That’s why the pipenv fiasco a few years ago
yewenjie 4y ago
Is there any hope that even if there emerges a consensus package management solution going forward, old packages will be easily portable to it?
- ferdowsi 4y ago
  Nothing will improve as long as the Python powers insist on packaging being an exercise for the community.
- black3r 4y ago
  Is there a problem with the package format itself though? There are lots of serious problems tied to distribution rather than package format which would make the experience way easier especially for beginners and people used to other package managers... lacking binary wheels on PyPI, problems with shipping project with dependencies, confusion about there being multiple "package managers" (pip, pip-tools, poetry, pipenv, conda) and multiple formats of dependency lists (setup.py, setup.cfg, requirements.txt, pyproject.toml, ...), sys.path associated confusion (global packages, user-level packages, and anything specified in PYTHONPATH, ...)
erwincoumans 4y ago
I am pretty happy with PyPi/pip, it is an easy way to distribute Python and C++ code wrapped in a Python extension to others. For a C++ developer it is becoming harder to distribute native executables, since MacOS and Windows require signing binaries. Python package version conflicts and backwards incompatibility can be an issue.
Kim_Bruning 4y ago
XKCD's opinion: https://xkcd.com/1987/ https://xkcd.com/1987/
- 7373737373 4y ago
  And that's just installing packages, creating packages is another hellscape altogether
- user8139471 4y ago
  Situation after the user survey has been evaluated: https://xkcd.com/927/ https://xkcd.com/927/
YetAnotherNick 4y ago
I wish there is some package manager in middle of conda and pip. Conda is too strict and often get stuck in SAT solving. pip doesn't even ask when reinstalling a version currently being used. Edit: Typo: reinstalling a version of package currently being used
- woodruffw 4y ago
  > pip doesn't even ask when reinstalling a version currently being used. Just as an explanation: a "version" in Python packaging can come from one of many potential distributions, including a local distribution (such as a path on disk) that might different from a canonical released distribution on PyPI. Having `pip install ...` always re-install based on its candidate selection rules is generally good (IMO), since an explicit run of `pip install` implies user intent to search for a potentially new or changed distribution.
  - YetAnotherNick 4y ago
    I meant this for dependency not the package I am installing.
    - woodruffw 4y ago
      I'm not sure I understand what you mean -- `pip` should not be reinstalling transitive dependencies. If you install A and B and both depend on C, C should only be installed once.
      - Izkata 4y ago
        I think they're referring to requirements files. I've seen the same behavior - on day 1, pip installs packages A and B, then on day 2 when someone else modified the requirements file it installs C and reinstalls B even though B hasn't changed. This one I've seen when the dependency doesn't specify an exact version and you included "--upgrade". There's a second case that I think was fixed in pip 21 or 22, where two transitive dependencies overlap - A and B depend on C, but with different version ranges. If A allows a newer version of C than B allows, C can get installed twice.
at_a_remove 4y ago
I have a terrible admission to make: one of the reason I like Python is its huge standard library, and I like that because I just ... despise looking for libraries, trying to install them, evaluating their fitness, and so on. I view dependencies outside of the standard library as a kind of technical debt, not because I suffer from Not Invented Here and want to code it myself, no, I look and think, "Why isn't this in the standard library with a working set of idioms around it?" I haven't developed anything with more than five digits of code to it, which is fine for me, but part of it is just ... avoidance of having to screw with libraries. Ran into a pip issue I won't go into (it requires a lot of justification to see how I got there) and just ... slumped over. This has been a bad spot in Python for a long, long time. While people are busy cramming their favorite feature from their last language into Python, this sort of thing has languished. Sadly, I have nothing to offer but encouragement, I don't know the complexities of packaging, it seems like a huge topic that perhaps nobody really dreamed Python would have to seriously deal with twenty years ago.
- itake 4y ago
  More packages in the standard library means it can run in less machines and more extra junk needs to be installed. Minimal standard library languages let you pick and choose what needs to be run. Golang is a nice happy medium since it’s compiled.
- samwillis 4y ago
  > despise looking for libraries, trying to install them, evaluating their fitness, and so on. This is exactly why I prefer the larger opinionated web frameworks (Django, Vue.js) to the smaller more composable frameworks (Flask, React). I don’t what to make decisions every time I need a new feature, I want something that “just works”. Python and Django just work, and brilliantly at that!
  - deleted 4y ago
    [deleted]
  - 5d8767c68926 4y ago
    Currently dealing with Flask and it makes me sad from the endless decision fatigue. Enormous variations in quality of code, documentation, SO answers, etc. To not even consider the potential for supply side attacks. With Django there is a happy path answer for most everything. If I run into a problem, I know I'm not the first.
black3r 4y ago
I just wish that PyPI would enforce binary wheels going forward (at least for linux x64/arm64 for people who use Docker, but ideally for all common platforms). They already supply the cibuildwheel tool to automate their builds, so it shouldn't be that hard for library developers... Software developers shouldn't need to figure out what build-time dependencies their libraries need...
- JonathonW 4y ago
  I haven't had too much trouble with packages missing binary wheels lately. Occasionally Pip doesn't find them, which looks the same as if the binary wheel were missing entirely (looking at you, Anaconda-- update your Pip already), but they're usually there. But I'm usually on Windows if I need binary wheels; maybe the coverage is a bit different on Linux.
  - verst 4y ago
    Try installing grpcio-tools on Darwin/arm64 with Python 3.10. More often than not I run into problems where low level headers required by some cryptography libraries cannot be found and as a result compilation fails.
- dalke 4y ago
  Should PyPI kick my project off because it don't support MS Windows? My package uses C and Cython extensions. While I support macOS and Linux-based OSes, I don't know how to develop on or support MS Windows. I've tried to be careful about the ILP32 vs LP64 differences, but I suspect there's going to be many places where I missed up. I also use "/dev/stdin" to work-around my use of a third-party library that has no way to read from stdin. As far as I can tell, there's no equivalent in Windows, so I'll have to raise an exception for that case, and modify my tests cases.
  - ciupicri 4y ago
    Can't you use "CON" instead of "/dev/stdin" on Windows?
mdmglr 4y ago
My wishlist: We need a way to configure an ordered list of indexes pip searches for packages. —extra-index-url or using a proxy index is not the solution. Also namespaces and not based on a domain. So for example: pip install apache:parquet Also some logic either in the pip client or index server to minimize typosquatting Also pip should adopt a lock file similar to npm/yarn. Instead of requirements.txt And also “pip list” should output a dependency tree like “npm list” I should not have to compile source when I install. Every package should have wheels available for the most common arch+OS combos. Also we need a way to download only what you need. Why does installing scipy or numpy install more dependencies than the conda version? For example pywin and scipy.
- tempest_ 4y ago
  If you are using poetry you can add something to the pyproject.toml to handle the indexes, though I am not sure if they are ordered or not [[tool.poetry.source]] name = "my-pypi" url = "https://my-pypi-index.wherever https://my-pypi-index.wherever" secondary = true
  - mdmglr 4y ago
    Thanks. I’ll look into this.
- ciupicri 4y ago
  > apache:parquet How are you going to name the file storing the wheel for that package? Using ":" on Windows is going to be problematic.
  - mdmglr 4y ago
    That’s right. Then some other delimiter.
- jbylund 4y ago
  Typosquatting is a thing that has been looked at and is being looked at: https://github.com/pypi/warehouse/pull/5001 https://github.com/pypi/warehouse/pull/5001 - had to be reverted because it was too noisy https://github.com/pypi/warehouse/issues/9527 https://github.com/pypi/warehouse/issues/9527
ryan29 4y ago
I didn't take the survey because I've never packaged anything for PyPI, but I wish all of the package managers would have an option for domain validated namespaces. If I own example.com, I should be able to have 'pypi.org/example.com/package'. The domain can be tied back to my (domain verified) GitHub profile and it opens up the possibility of using something like 'example.com/.well-known/pypi/' for self-managed signing keys, etc.. I could be using the same namespace for every package manager in existence if domain validated namespaces were common. Then, in my perfect world, something like Sigstore could support code signing with domain validated identities. Domain validated signatures make a lot of sense. Domains are relatively inexpensive, inexhaustible, and globally unique. For code signing, I recognize a lot of project names and developer handles while knowing zero real names for the companies / developers involved. If those were sitting under a recognizable organizational domain name (example.com/ryan29) I can do a significantly better job of judging something's trustworthiness than if it's attributed to 'Ryan Smith Inc.', right?
- simonw 4y ago
  That's a really interesting idea, but I worry about what happens when a domain name expires and is re-registered (potentially even maliciously) by someone else.
  - ryan29 4y ago
    I think you'd probably need some buy in from the domain registries and ICANN to make it really solid. Ideally, domains would have something similar to public certificate transparency logs where domain expirations would be recorded. I even think it would be reasonable to log registrant changes (legal registrant, not contact info). In both cases, it wouldn't need to include any identifiable info, just a simple expired/ownership changed trigger so others would know they need to revalidate related identities. I don't know if registries would play ball with something like that, but it would be useful and should probably exist anyway. I would even argue that once a domain rolls through grace, redemption, etc. and gets dropped / re-registered, that should invalidate it as an account recovery method everywhere it's in use. There's a bit of complexity when it comes to the actual validation because of stuff like that. I think you'd need buy in from at least one large company that could do the actual verification and attest to interested parties via something like OAuth. Think along the lines of "verify your domain by logging in with GitHub" and at GitHub an organization owner that's validated their domain would be allowed to grant OAuth permission to read the verified domain name.
shireboy 4y ago
I’ve been tinkering with stable diffusion lately and this has been a rude introduction to python. Coming from .net (nuget) and JavaScript (npm), it’s baffling that there isn’t an established solution for python. It looks to me like people are trying, but different libraries use different techniques. To a newcomer this is confusing.
- timtom39 4y ago
  ML/AI is the deep end of python dependencies. Lots of hardware specific requirements (e.g. CUDA, cuDNN, AVX2 TensorFlow binaries, etc). A typical python web application is a lot simpler.
  - shireboy 4y ago
    To be fair, this is a big part of the problem- I’m doing this on a m1 mac, PLUS am new to python. Fortunately there are some good guides. Still, this is a solvable problem in other package managers.
- antod 4y ago
  >I’ve been tinkering with stable diffusion lately and this has been a rude introduction to python. Coming from .net (nuget) and JavaScript (npm), it’s baffling that there isn’t an established solution for python. Python has had multiple legacy solutions going back a long time before nuget and npm existed, and before central registries of dependencies. Every new solution has to cope with all that compatibility/transitional baggage. Also a bunch of usecases .NET or JS never really had to deal much with - eg being a core system language for Linux distros, and supporting cross platform installs back in the download and run something days. The scope of areas Python gets used in means its packaging is pulled in more directions than most other languages who mostly stick to a main niche. So the history and surface area of problems to solve in Python packaging is larger than what most other languages have had to deal with. It also takes years for the many 3rd party tools to try out new approaches, gain traction and then slowly get their best ideas synthesized and adapted into the much more conservative core Python stdlib. Not saying it is great, just laying out some of the reasons it is what it is.
shrimpx 4y ago
I like poetry for its simplicity but I can’t tell how “official” it is in the python ecosystem. I hope it doesn’t die out. I think it’s the simplest possible way to maintain deps and publish to PyPI if you don’t have any weird edge cases.
- wlkr 4y ago
  Couldn't agree more. Poetry is fantastic and provides that 'just works' experience for most cases. It's not official (although possibly should be adopted) but has gained ground by virtue of its quality. Fortunately it's very actively developed so will hopefully stick around.
  - tempest_ 4y ago
    I like poetry. It still has a way to go though since it is slow as all hell doing almost anything and its error messages are closer to a stack trace than something actionable.
    - shrimpx 4y ago
      I agree that the stack trace error messages are weird. That aspect feels uncharacteristically hacky for an otherwise pretty polished tool.
      - ddejohn 4y ago
        Supposedly this has been improved in 1.2.0: https://python-poetry.org/blog/announcing-poetry-1.2.0/#non-verbose-error-handling https://python-poetry.org/blog/announcing-poetry-1.2.0/#non-...
- IceHegel 4y ago
  It should be make the official package manager IMO.
- nomel 4y ago
  The state of python packing changes so frequently that I wouldn't even be able to answer this survey, without looking up best practices. We're supposed to be using yaml files now, right?
orf 4y ago
For improvements I commented: Remove setup.py files and mandate wheels. This is the root cause of a lot of the evil in the ecosystem. Next on the list would be pypi namespaces, but there are good reasons why that is very hard. The mission statement they are proposing, “a packaging ecosystem for all”, completely misses the mark. How about a “packaging ecosystem that works” first? I spent a bunch of time recently fixing our internal packaging repo (nexus) because the switch from md5 hashes to sha256 hashes broke everything, and re-locking a bajillion lock files would take literally months of man hours time. I’ve been a Python user for the last 17 years, so I’m sympathetic of how we got to the current situation and aware that we’ve actually come quite far. But every time I use Cargo I am insanely jealous, impressed and sad that we don’t have something like it. Poetry is closest, but it’s a far cry.
- kevin_thibedeau 4y ago
  Setup.py can do things wheels can't. Most notably it's the only installation method that can invoke 2to3 at runtime without requiring a dev to create multiple packages.
  - orf 4y ago
    It’s lucky Python 2 isn’t supported anymore then, and everyone has had like a decade to run 2to3 once and publish a package for Python 3, so that use case becomes meaningless.
    - coldtea 4y ago
      You'd be surprised at how many billions lines of production code are still at 2 (and could not care less whether it's end-of-lined)
      - orf 4y ago
        I'm not surprised at all, but regardless they also should not be similarly surprised if people could not care less about that use case.
    - 4y ago
bzxcvbn 4y ago
I just wish they'd change their name so that my students stop snickering. (The name is pronounced like the French word for "piss".)
- d0mine 4y ago
  Isn't PyPI pronounced like: pie (food/π) + pea + eye (). that is different from the french "pipi": pea + pea.
  - di 4y ago
    Yep: https://pypi.org/help/#pronunciation https://pypi.org/help/#pronunciation
  - bzxcvbn 4y ago
    Sure. Now tell that to a bunch of immature college juniors. If you read the letters "pypi" in French, it sounds exactly like "pipi". And wait until you learn what "bit" sounds like in French.
    - user8139471 4y ago
      What does "bit" sound like in French?
      - bzxcvbn 4y ago
        The slang word for penis. "Megabit" and "gigabit" are endless sources of fun for the students.
- nomdep 4y ago
  Once upon a time, it’s name was “cheese shop”. Good times
_iyhy 4y ago
Pickup poetry and fix it. I thought it would be fun to use poetry but it smacks itself here and there.
- deleted 4y ago
  [deleted]
fsniper 4y ago
Python lacks the flexible universal binary distribution solution which nearly all of the new comers has. Consider golang, rust or docker images. Most probably docker image distribution is the only available solution for now and volume management is the worst problem on that front.
siproprio 4y ago
State of the art python packaging must include support for common use cases such as conda+machine learning. It’s incredible how even Julia’s Pkg.jl supports better python packaging in combination with conda than the official python packaging tools. This is very clearly a question of the culture of the core python developers (such as brett cannon) who seem to think the machine learning people with their compilers and JITs are not an important part of the community.
- klkqw 4y ago
  The Twitter hero faction of the Python core "developers" (many of whom have not done much actual work for a decade) has always been pro-web and anti-C-extensions. They have been peddling the dream of speeding up Python by neglecting the scientific ecosystem for a long time without having either the expertise or a clear plan how to achieve that. All that while the primarily useful part of Python is its C extension capabilities. The web does not need Python, query/response applications are handled just fine and better by functional languages.
  - siproprio 4y ago
    The pypy folks and a bunch of other alternative python implementations have pooled together and are working on a soft fork of the c extension api to enable existing established projects that make heavy use of the c api to better work with the superior faster python implementations. I believe this is the way forward. Existing packages are essentially tied to cpython because that’s what the python.org website offers for download, and people will just use that by default. Once all major packages support the new c api, it will be easier to just use something other than cpython by default.
mcdermott 4y ago
Installing packages, creating a manifest of dependancies, managing virtual environments, packaging, checking/formatting code, etc... should be built into the Python toolchain (the python binary itself). Needing to chose a bunch of third party tools to make it work... makes Python, well... un-pythonic.
- Const-me 4y ago
  Not sure about formatting code, I think that's a job for IDE or text editor, not the runtime. Absolutely agree about the rest of them.
  - mcdermott 4y ago
    I was thinking of something similar to go's "go fmt". Something that would standardize the indentation (tab to 4 spaces), sort imports in the correct order, etc.
IceHegel 4y ago
If I see some JS, Go, or Rust code online I know I can probably get it running on my machine in less than 5 min. Most of the time, it's a ‘git clone’ and a 'yarn' | 'go install' | 'cargo run', and it just works. With python, it feels like half the time I don't even have the right version of python installed, or it’s somehow not on the right path. And once I actually get to installing dependencies, there are often very opaque errors. (The last 2 years on M1 were really rough) Setting up Pytorch or Tensorflow + CUDA is a nightmare I've experienced many times. Having so many ways to manage packages is especially harmful for python because many of those writing python are not professional software engineers, but academics and researchers. If they write something that needs, for example, CUDA 10.2, Python 3.6, and a bunch of C audio drivers - good luck getting that code to work in less than a week. They aren’t writing install scripts, or testing their code on different platforms, and the python ecosystem makes the whole process worse by providing 15 ways of doing basically the same thing. My proposal: - Make poetry part of pip - Make local installation the default (must pass -g for global) - Provide official first party tooling for starting a new package - Provide official first party tooling for migrating old dependency setups to the new standard edit: fmt
- bno1 4y ago
  I wish pip had some package deduplication implemented. Even some basic local environments have >100MB of dependencies. ML environments go into the gigabytes range from what I remember.
  - smeagull 4y ago
    Cargo allows you to share a single package directory. Having a single site-packages and the ability to import a particular version (or latest by default) would solve this.
    - Asraelite 4y ago
      This is also nice because you can clean up your build files all in one go. Every now and again I delete my cargo target directory and get 5GB back.
smitty1e 4y ago
Packaging motto suggestion offered: "Mastery Over 'Ni!' Triumph Yields" (MONTY) Need to revive the MONTY Python spirit as a hedge against the general dreariness of the day.
tibbon 4y ago
> There should be one– and preferably only one –obvious way to do it. I think the answer to Python’s packaging woes has been in plain sight all along. There are so many competing package managers and systems that it is hard to keep track of them all. Of course; this is just what happens in open source, but the problem seems uniquely bad for Python. In Ruby, Swift, Rust or Golang I don’t encounter there being quite the multitude of options for each. Work on cross comparability and roadmap to merge them all down to eventually one obvious way to install packages. Lost an hour of time trying to get Stable Diffusion working on an M1 Mac yesterday
est 4y ago
Python packaging is fine, the difficult part is always C code with .pyd or .so files god damn those are nasty piece of shit. Pypi doesn't need users to manually to fill survey per se, they need to (optionally) prompt "install failed, submit traceback and error code to pypi for solutions" during setup.
electromech 4y ago
Thanks for posting this! I'm glad I had the chance to share some feedback. I only wish I had seen an "Other Feedback" box to simply say thanks to everyone trying to make things better.
EdwardDiego 4y ago
Nevermind Pypi packages, the monstrosity that is __init__.py must die. Namespace traversal that executes code is just so so bad.

234 points by zbentley 4y ago

I imagine many of you have feedback that could be useful to folks making decisions about the future of Python packaging, a common subject of complaint in many discussions here. Remember not to just complain, but to offer specific problems/solutions--i.e. avoid statements like "virtualenvs suck, why can't it be like NPM?" and prefer instead feedback like "the difference between Python interpreter version and what virtualenv is being used causes confusion".

Kwpolska 4y ago
"virtualenvs suck, why can't it be like NPM?" is a specific problem and a specific solution. The problem being having to manage venvs (which have many gotchas and pitfalls, and no standarization), and the solution is to replace those with packages being installed into the project folder with standardized and well-known tools.
- qbasic_forever 4y ago
  Keep an eye on https://peps.python.org/pep-0582/ https://peps.python.org/pep-0582/ it's a proposal to add local directory/node_modules-like behavior to package installs. It stalled out a few years ago but I heard there is a lot more discussion and push to get it in now. I think if this PEP makes it in then like 90% of people's pain with pip just completely goes away almost overnight. Love it or hate it the NPM/node_modules style of all dependencies dumped in a local directory solves a _ton_ of problems in the packaging world. It would go a long way towards making the experience much smoother for most python users.
- jogjayr 4y ago
  I did a proof-of-concept for this a few years ago: https://github.com/jogjayr/pykg https://github.com/jogjayr/pykg It literally uses npm, the NPM registry, and node_modules for Python dependency management.

clintonb 4y ago

The survey is at https://www.surveymonkey.co.uk/r/M5XKQCT https://www.surveymonkey.co.uk/r/M5XKQCT.

MonkeyMalarky 4y ago
Seems more oriented to (potential) contributors than end users of the packaging system. Who cares about mission statements and inclusivity, secure funding and pay developers to make the tools.
- woodruffw 4y ago
  > Who cares about mission statements and inclusivity, secure funding and pay developers to make the tools. These are connected things. I maintain a PyPA member project (and contribute to many others), and the latter is aided by the former: the mission statement keeps the community organized around shared goals (such as standardizing Python's packaging tooling), and inclusivity insures a healthy and steady flow of new contributors (and potential corporate funding sources).
  - electromech 4y ago
    > I maintain a PyPA member project (and contribute to many others) THANK YOU! > keeps the community organized around shared goals (such as standardizing Python's packaging tooling) Personally I felt some disconnect between "package manager for all" and the need for "standardizing Python's packaging tooling." Yes, communities should be welcoming and friendly to everyone, AND the community should have clear expectations for best practices that members should follow. E.g., is an experienced female developer more likely to give up on contributing because she couldn't find a local meetup, or because she didn't know whether to create pyproject.toml vs requirements.txt? In some sense, the bigger and more diverse the community, the greater the need for a clear, solid foundation. IDK if that's remotely clear; it's just a feeling I had going through some of those questions.
- nomdep 4y ago
  The PSF are not engineers looking for a better developer experience, but politicians looking for power. That’s why the pipenv fiasco a few years ago

yewenjie 4y ago

Is there any hope that even if there emerges a consensus package management solution going forward, old packages will be easily portable to it?

ferdowsi 4y ago
Nothing will improve as long as the Python powers insist on packaging being an exercise for the community.
black3r 4y ago
Is there a problem with the package format itself though? There are lots of serious problems tied to distribution rather than package format which would make the experience way easier especially for beginners and people used to other package managers... lacking binary wheels on PyPI, problems with shipping project with dependencies, confusion about there being multiple "package managers" (pip, pip-tools, poetry, pipenv, conda) and multiple formats of dependency lists (setup.py, setup.cfg, requirements.txt, pyproject.toml, ...), sys.path associated confusion (global packages, user-level packages, and anything specified in PYTHONPATH, ...)

erwincoumans 4y ago

I am pretty happy with PyPi/pip, it is an easy way to distribute Python and C++ code wrapped in a Python extension to others. For a C++ developer it is becoming harder to distribute native executables, since MacOS and Windows require signing binaries. Python package version conflicts and backwards incompatibility can be an issue.

Kim_Bruning 4y ago

XKCD's opinion: https://xkcd.com/1987/ https://xkcd.com/1987/

7373737373 4y ago
And that's just installing packages, creating packages is another hellscape altogether
user8139471 4y ago
Situation after the user survey has been evaluated: https://xkcd.com/927/ https://xkcd.com/927/

YetAnotherNick 4y ago

I wish there is some package manager in middle of conda and pip. Conda is too strict and often get stuck in SAT solving. pip doesn't even ask when reinstalling a version currently being used. Edit: Typo: reinstalling a version of package currently being used

woodruffw 4y ago
> pip doesn't even ask when reinstalling a version currently being used. Just as an explanation: a "version" in Python packaging can come from one of many potential distributions, including a local distribution (such as a path on disk) that might different from a canonical released distribution on PyPI. Having `pip install ...` always re-install based on its candidate selection rules is generally good (IMO), since an explicit run of `pip install` implies user intent to search for a potentially new or changed distribution.
- YetAnotherNick 4y ago
  I meant this for dependency not the package I am installing.
  - woodruffw 4y ago
    I'm not sure I understand what you mean -- `pip` should not be reinstalling transitive dependencies. If you install A and B and both depend on C, C should only be installed once.
    - Izkata 4y ago
      I think they're referring to requirements files. I've seen the same behavior - on day 1, pip installs packages A and B, then on day 2 when someone else modified the requirements file it installs C and reinstalls B even though B hasn't changed. This one I've seen when the dependency doesn't specify an exact version and you included "--upgrade". There's a second case that I think was fixed in pip 21 or 22, where two transitive dependencies overlap - A and B depend on C, but with different version ranges. If A allows a newer version of C than B allows, C can get installed twice.

at_a_remove 4y ago

I have a terrible admission to make: one of the reason I like Python is its huge standard library, and I like that because I just ... despise looking for libraries, trying to install them, evaluating their fitness, and so on. I view dependencies outside of the standard library as a kind of technical debt, not because I suffer from Not Invented Here and want to code it myself, no, I look and think, "Why isn't this in the standard library with a working set of idioms around it?" I haven't developed anything with more than five digits of code to it, which is fine for me, but part of it is just ... avoidance of having to screw with libraries. Ran into a pip issue I won't go into (it requires a lot of justification to see how I got there) and just ... slumped over. This has been a bad spot in Python for a long, long time. While people are busy cramming their favorite feature from their last language into Python, this sort of thing has languished. Sadly, I have nothing to offer but encouragement, I don't know the complexities of packaging, it seems like a huge topic that perhaps nobody really dreamed Python would have to seriously deal with twenty years ago.

itake 4y ago
More packages in the standard library means it can run in less machines and more extra junk needs to be installed. Minimal standard library languages let you pick and choose what needs to be run. Golang is a nice happy medium since it’s compiled.
samwillis 4y ago
> despise looking for libraries, trying to install them, evaluating their fitness, and so on. This is exactly why I prefer the larger opinionated web frameworks (Django, Vue.js) to the smaller more composable frameworks (Flask, React). I don’t what to make decisions every time I need a new feature, I want something that “just works”. Python and Django just work, and brilliantly at that!
- deleted 4y ago
  [deleted]
- 5d8767c68926 4y ago
  Currently dealing with Flask and it makes me sad from the endless decision fatigue. Enormous variations in quality of code, documentation, SO answers, etc. To not even consider the potential for supply side attacks. With Django there is a happy path answer for most everything. If I run into a problem, I know I'm not the first.

black3r 4y ago

I just wish that PyPI would enforce binary wheels going forward (at least for linux x64/arm64 for people who use Docker, but ideally for all common platforms). They already supply the cibuildwheel tool to automate their builds, so it shouldn't be that hard for library developers... Software developers shouldn't need to figure out what build-time dependencies their libraries need...

JonathonW 4y ago
I haven't had too much trouble with packages missing binary wheels lately. Occasionally Pip doesn't find them, which looks the same as if the binary wheel were missing entirely (looking at you, Anaconda-- update your Pip already), but they're usually there. But I'm usually on Windows if I need binary wheels; maybe the coverage is a bit different on Linux.
- verst 4y ago
  Try installing grpcio-tools on Darwin/arm64 with Python 3.10. More often than not I run into problems where low level headers required by some cryptography libraries cannot be found and as a result compilation fails.
dalke 4y ago
Should PyPI kick my project off because it don't support MS Windows? My package uses C and Cython extensions. While I support macOS and Linux-based OSes, I don't know how to develop on or support MS Windows. I've tried to be careful about the ILP32 vs LP64 differences, but I suspect there's going to be many places where I missed up. I also use "/dev/stdin" to work-around my use of a third-party library that has no way to read from stdin. As far as I can tell, there's no equivalent in Windows, so I'll have to raise an exception for that case, and modify my tests cases.
- ciupicri 4y ago
  Can't you use "CON" instead of "/dev/stdin" on Windows?

mdmglr 4y ago

My wishlist: We need a way to configure an ordered list of indexes pip searches for packages. —extra-index-url or using a proxy index is not the solution. Also namespaces and not based on a domain. So for example: pip install apache:parquet Also some logic either in the pip client or index server to minimize typosquatting Also pip should adopt a lock file similar to npm/yarn. Instead of requirements.txt And also “pip list” should output a dependency tree like “npm list” I should not have to compile source when I install. Every package should have wheels available for the most common arch+OS combos. Also we need a way to download only what you need. Why does installing scipy or numpy install more dependencies than the conda version? For example pywin and scipy.

tempest_ 4y ago
If you are using poetry you can add something to the pyproject.toml to handle the indexes, though I am not sure if they are ordered or not [[tool.poetry.source]] name = "my-pypi" url = "https://my-pypi-index.wherever https://my-pypi-index.wherever" secondary = true
- mdmglr 4y ago
  Thanks. I’ll look into this.
ciupicri 4y ago
> apache:parquet How are you going to name the file storing the wheel for that package? Using ":" on Windows is going to be problematic.
- mdmglr 4y ago
  That’s right. Then some other delimiter.
jbylund 4y ago
Typosquatting is a thing that has been looked at and is being looked at: https://github.com/pypi/warehouse/pull/5001 https://github.com/pypi/warehouse/pull/5001 - had to be reverted because it was too noisy https://github.com/pypi/warehouse/issues/9527 https://github.com/pypi/warehouse/issues/9527

ryan29 4y ago

I didn't take the survey because I've never packaged anything for PyPI, but I wish all of the package managers would have an option for domain validated namespaces. If I own example.com, I should be able to have 'pypi.org/example.com/package'. The domain can be tied back to my (domain verified) GitHub profile and it opens up the possibility of using something like 'example.com/.well-known/pypi/' for self-managed signing keys, etc.. I could be using the same namespace for every package manager in existence if domain validated namespaces were common. Then, in my perfect world, something like Sigstore could support code signing with domain validated identities. Domain validated signatures make a lot of sense. Domains are relatively inexpensive, inexhaustible, and globally unique. For code signing, I recognize a lot of project names and developer handles while knowing zero real names for the companies / developers involved. If those were sitting under a recognizable organizational domain name (example.com/ryan29) I can do a significantly better job of judging something's trustworthiness than if it's attributed to 'Ryan Smith Inc.', right?

simonw 4y ago
That's a really interesting idea, but I worry about what happens when a domain name expires and is re-registered (potentially even maliciously) by someone else.
- ryan29 4y ago
  I think you'd probably need some buy in from the domain registries and ICANN to make it really solid. Ideally, domains would have something similar to public certificate transparency logs where domain expirations would be recorded. I even think it would be reasonable to log registrant changes (legal registrant, not contact info). In both cases, it wouldn't need to include any identifiable info, just a simple expired/ownership changed trigger so others would know they need to revalidate related identities. I don't know if registries would play ball with something like that, but it would be useful and should probably exist anyway. I would even argue that once a domain rolls through grace, redemption, etc. and gets dropped / re-registered, that should invalidate it as an account recovery method everywhere it's in use. There's a bit of complexity when it comes to the actual validation because of stuff like that. I think you'd need buy in from at least one large company that could do the actual verification and attest to interested parties via something like OAuth. Think along the lines of "verify your domain by logging in with GitHub" and at GitHub an organization owner that's validated their domain would be allowed to grant OAuth permission to read the verified domain name.

shireboy 4y ago

I’ve been tinkering with stable diffusion lately and this has been a rude introduction to python. Coming from .net (nuget) and JavaScript (npm), it’s baffling that there isn’t an established solution for python. It looks to me like people are trying, but different libraries use different techniques. To a newcomer this is confusing.

timtom39 4y ago
ML/AI is the deep end of python dependencies. Lots of hardware specific requirements (e.g. CUDA, cuDNN, AVX2 TensorFlow binaries, etc). A typical python web application is a lot simpler.
- shireboy 4y ago
  To be fair, this is a big part of the problem- I’m doing this on a m1 mac, PLUS am new to python. Fortunately there are some good guides. Still, this is a solvable problem in other package managers.
antod 4y ago
>I’ve been tinkering with stable diffusion lately and this has been a rude introduction to python. Coming from .net (nuget) and JavaScript (npm), it’s baffling that there isn’t an established solution for python. Python has had multiple legacy solutions going back a long time before nuget and npm existed, and before central registries of dependencies. Every new solution has to cope with all that compatibility/transitional baggage. Also a bunch of usecases .NET or JS never really had to deal much with - eg being a core system language for Linux distros, and supporting cross platform installs back in the download and run something days. The scope of areas Python gets used in means its packaging is pulled in more directions than most other languages who mostly stick to a main niche. So the history and surface area of problems to solve in Python packaging is larger than what most other languages have had to deal with. It also takes years for the many 3rd party tools to try out new approaches, gain traction and then slowly get their best ideas synthesized and adapted into the much more conservative core Python stdlib. Not saying it is great, just laying out some of the reasons it is what it is.

shrimpx 4y ago

I like poetry for its simplicity but I can’t tell how “official” it is in the python ecosystem. I hope it doesn’t die out. I think it’s the simplest possible way to maintain deps and publish to PyPI if you don’t have any weird edge cases.

wlkr 4y ago
Couldn't agree more. Poetry is fantastic and provides that 'just works' experience for most cases. It's not official (although possibly should be adopted) but has gained ground by virtue of its quality. Fortunately it's very actively developed so will hopefully stick around.
- tempest_ 4y ago
  I like poetry. It still has a way to go though since it is slow as all hell doing almost anything and its error messages are closer to a stack trace than something actionable.
  - shrimpx 4y ago
    I agree that the stack trace error messages are weird. That aspect feels uncharacteristically hacky for an otherwise pretty polished tool.
    - ddejohn 4y ago
      Supposedly this has been improved in 1.2.0: https://python-poetry.org/blog/announcing-poetry-1.2.0/#non-verbose-error-handling https://python-poetry.org/blog/announcing-poetry-1.2.0/#non-...
IceHegel 4y ago
It should be make the official package manager IMO.
nomel 4y ago
The state of python packing changes so frequently that I wouldn't even be able to answer this survey, without looking up best practices. We're supposed to be using yaml files now, right?

orf 4y ago

For improvements I commented: Remove setup.py files and mandate wheels. This is the root cause of a lot of the evil in the ecosystem. Next on the list would be pypi namespaces, but there are good reasons why that is very hard. The mission statement they are proposing, “a packaging ecosystem for all”, completely misses the mark. How about a “packaging ecosystem that works” first? I spent a bunch of time recently fixing our internal packaging repo (nexus) because the switch from md5 hashes to sha256 hashes broke everything, and re-locking a bajillion lock files would take literally months of man hours time. I’ve been a Python user for the last 17 years, so I’m sympathetic of how we got to the current situation and aware that we’ve actually come quite far. But every time I use Cargo I am insanely jealous, impressed and sad that we don’t have something like it. Poetry is closest, but it’s a far cry.

kevin_thibedeau 4y ago
Setup.py can do things wheels can't. Most notably it's the only installation method that can invoke 2to3 at runtime without requiring a dev to create multiple packages.
- orf 4y ago
  It’s lucky Python 2 isn’t supported anymore then, and everyone has had like a decade to run 2to3 once and publish a package for Python 3, so that use case becomes meaningless.
  - coldtea 4y ago
    You'd be surprised at how many billions lines of production code are still at 2 (and could not care less whether it's end-of-lined)
    - orf 4y ago
      I'm not surprised at all, but regardless they also should not be similarly surprised if people could not care less about that use case.
  - 4y ago

bzxcvbn 4y ago

I just wish they'd change their name so that my students stop snickering. (The name is pronounced like the French word for "piss".)

d0mine 4y ago
Isn't PyPI pronounced like: pie (food/π) + pea + eye (). that is different from the french "pipi": pea + pea.
- di 4y ago
  Yep: https://pypi.org/help/#pronunciation https://pypi.org/help/#pronunciation
- bzxcvbn 4y ago
  Sure. Now tell that to a bunch of immature college juniors. If you read the letters "pypi" in French, it sounds exactly like "pipi". And wait until you learn what "bit" sounds like in French.
  - user8139471 4y ago
    What does "bit" sound like in French?
    - bzxcvbn 4y ago
      The slang word for penis. "Megabit" and "gigabit" are endless sources of fun for the students.
nomdep 4y ago
Once upon a time, it’s name was “cheese shop”. Good times

_iyhy 4y ago

Pickup poetry and fix it. I thought it would be fun to use poetry but it smacks itself here and there.

deleted 4y ago
[deleted]

fsniper 4y ago

Python lacks the flexible universal binary distribution solution which nearly all of the new comers has. Consider golang, rust or docker images. Most probably docker image distribution is the only available solution for now and volume management is the worst problem on that front.

siproprio 4y ago

State of the art python packaging must include support for common use cases such as conda+machine learning. It’s incredible how even Julia’s Pkg.jl supports better python packaging in combination with conda than the official python packaging tools. This is very clearly a question of the culture of the core python developers (such as brett cannon) who seem to think the machine learning people with their compilers and JITs are not an important part of the community.

klkqw 4y ago
The Twitter hero faction of the Python core "developers" (many of whom have not done much actual work for a decade) has always been pro-web and anti-C-extensions. They have been peddling the dream of speeding up Python by neglecting the scientific ecosystem for a long time without having either the expertise or a clear plan how to achieve that. All that while the primarily useful part of Python is its C extension capabilities. The web does not need Python, query/response applications are handled just fine and better by functional languages.
- siproprio 4y ago
  The pypy folks and a bunch of other alternative python implementations have pooled together and are working on a soft fork of the c extension api to enable existing established projects that make heavy use of the c api to better work with the superior faster python implementations. I believe this is the way forward. Existing packages are essentially tied to cpython because that’s what the python.org website offers for download, and people will just use that by default. Once all major packages support the new c api, it will be easier to just use something other than cpython by default.

mcdermott 4y ago

Installing packages, creating a manifest of dependancies, managing virtual environments, packaging, checking/formatting code, etc... should be built into the Python toolchain (the python binary itself). Needing to chose a bunch of third party tools to make it work... makes Python, well... un-pythonic.

Const-me 4y ago
Not sure about formatting code, I think that's a job for IDE or text editor, not the runtime. Absolutely agree about the rest of them.
- mcdermott 4y ago
  I was thinking of something similar to go's "go fmt". Something that would standardize the indentation (tab to 4 spaces), sort imports in the correct order, etc.

IceHegel 4y ago

If I see some JS, Go, or Rust code online I know I can probably get it running on my machine in less than 5 min. Most of the time, it's a ‘git clone’ and a 'yarn' | 'go install' | 'cargo run', and it just works. With python, it feels like half the time I don't even have the right version of python installed, or it’s somehow not on the right path. And once I actually get to installing dependencies, there are often very opaque errors. (The last 2 years on M1 were really rough) Setting up Pytorch or Tensorflow + CUDA is a nightmare I've experienced many times. Having so many ways to manage packages is especially harmful for python because many of those writing python are not professional software engineers, but academics and researchers. If they write something that needs, for example, CUDA 10.2, Python 3.6, and a bunch of C audio drivers - good luck getting that code to work in less than a week. They aren’t writing install scripts, or testing their code on different platforms, and the python ecosystem makes the whole process worse by providing 15 ways of doing basically the same thing. My proposal: - Make poetry part of pip - Make local installation the default (must pass -g for global) - Provide official first party tooling for starting a new package - Provide official first party tooling for migrating old dependency setups to the new standard edit: fmt

bno1 4y ago
I wish pip had some package deduplication implemented. Even some basic local environments have >100MB of dependencies. ML environments go into the gigabytes range from what I remember.
- smeagull 4y ago
  Cargo allows you to share a single package directory. Having a single site-packages and the ability to import a particular version (or latest by default) would solve this.
  - Asraelite 4y ago
    This is also nice because you can clean up your build files all in one go. Every now and again I delete my cargo target directory and get 5GB back.

smitty1e 4y ago

Packaging motto suggestion offered: "Mastery Over 'Ni!' Triumph Yields" (MONTY) Need to revive the MONTY Python spirit as a hedge against the general dreariness of the day.

tibbon 4y ago

> There should be one– and preferably only one –obvious way to do it. I think the answer to Python’s packaging woes has been in plain sight all along. There are so many competing package managers and systems that it is hard to keep track of them all. Of course; this is just what happens in open source, but the problem seems uniquely bad for Python. In Ruby, Swift, Rust or Golang I don’t encounter there being quite the multitude of options for each. Work on cross comparability and roadmap to merge them all down to eventually one obvious way to install packages. Lost an hour of time trying to get Stable Diffusion working on an M1 Mac yesterday

est 4y ago

Python packaging is fine, the difficult part is always C code with .pyd or .so files god damn those are nasty piece of shit. Pypi doesn't need users to manually to fill survey per se, they need to (optionally) prompt "install failed, submit traceback and error code to pypi for solutions" during setup.

electromech 4y ago

Thanks for posting this! I'm glad I had the chance to share some feedback. I only wish I had seen an "Other Feedback" box to simply say thanks to everyone trying to make things better.

EdwardDiego 4y ago

Nevermind Pypi packages, the monstrosity that is __init__.py must die. Namespace traversal that executes code is just so so bad.