11 ms·

This is great, but sometimes I think that python needs a new package manager from scratch instead of more tools trying to mix and mash a bunch of flawed tools t

by cderwin 10y ago

This is great, but sometimes I think that python needs a new package manager from scratch instead of more tools trying to mix and mash a bunch of flawed tools together in a way that's palatable by most of us. Python packaging sucks, the whole lot of it. Maybe I'm just spoiled by rust and elixir, but setuptools, distutils, pip, ez_install, all of it is really subpar. But of course everything uses pypi and pip now, so it's not like any of it can actually be replaced. The state of package management in python makes me sad. I wish there was a good solution, but I just don't see it. Edit: I don't mean to disparage projects like this and pipfile. Both are great efforts to bring the packaging interface in line with what's available in other languages, and might be the only way up and out of the current state of affairs.

sametmax 10y ago
I strongly agree but this can work now and is a big improvement over what we currently have. So while I would literally pay to see somebody work on a better package manager (which can generate exe, deb and use a conf file indead of .py), this is a good filler.
- X-Istence 10y ago
  Why should it generate a deb file? How is this useful on anything but Debian based systems? Why exe? How do you package libraries using this new tool you are envisioning?
  - sametmax 10y ago
    You do package lib the same way as before, although cargo like dependency handling would be a nice thing. Especially for upgrades. But a good package manager should ALSO allow you to produce a: - a stand alone executable for most OS. - a standard package for major OS (msi, deb, snap, rpm, dmg, etc). Doing that right now with Python requires you to setup stuff like nuikta and the like. It works but it's much harder than it should be.
    - snuxoll 10y ago
      debhelper pretty much automates the process of packaging any standard distutils or setuptools package, Red Hat distributions have templates for packaging Python libraries as well (and rpmdev-newspec python-mypackage will automatically generate an appropriate .spec file). Windows and OS X are always a pain in the ass, but that's more an issue of the platforms lacking in package management than anything else.
      - sametmax 10y ago
        See my point ? There is a way to do it, it's just a pain. Now pipenv centralize stuff we were doing anyway. We should have a tool to centralize those as well.
        snuxoll 10y ago
        I would hardly call it a pain, it takes me all of 3 minutes to write a .spec for most python packages and from there it's basically 'tito release'. Sure, if I wanted to package for debian-based distributions it'd take a little more time, but it's worth it to make a quality package that a distribution itself can decide to pick up (packagers love other people doing the work for them, though they won't refuse doing it themselves) with minimal effort.
        sametmax 10y ago
        3 minutes because you know how.
        snuxoll 10y ago
        And it took me all of 60 minutes to learn, packaging isn't anywhere near as hard as people make it out to be. If you can use a build system of your choosing to build and test your project, you can learn how to write an rpmspec or debian control file in under an hour.
        10y ago
ploggingdev 10y ago
Just curious, what aspects of pip/virtualenv specifically do you find subpar in comparison to other languages' package managers.
- cderwin 10y ago
  I would look at this comment[0] by sametmax for a critique of pip. My main gripe with virtualenv is that it's required at all: other interpreted languages, like node and elixir for example, have figured out how to handle non-global dependencies without a third-party package. Beyond that, it's frustrating to deploy because its non-relocatable (in our build/deploy scripts at my last python job we had to use sed all over the place to fix paths), and I find it semi-annoying to have a bunch of different copies of interpreter and all that goes with it (though this is mostly a minor annoyance -- it doesn't take up that much space and it doesn't matter if it gets out of sync. Also notable, IMO, is the lack of a tool like rbenv or rustup for python. I can't tell you how many times I have had to try to figure out which python version a given pip executable worked with. [0] https://news.ycombinator.com/item?id=13460490 https://news.ycombinator.com/item?id=13460490
  - catdog 10y ago
    > Beyond that, it's frustrating to deploy because its non-relocatable (in our build/deploy scripts at my last python job we had to use sed all over the place to fix paths) pip does cache the wheels so instead of moving the virtualenvs around, just recreate them. This also ensures the virtualenv is up to date. Using tox this is fairly easy to do. Sure virtualenv is a bit of a hack but it's not that bad.
    - cderwin 10y ago
      I'm mostly talking about moving between different machines. I would like to be able to tar up my source code and venv, distribute it to multiple machines, untar it and run it. However that's not possible with virtualenv unless you do a lot of hackery in your build. In particular, creating a virtualenv on each server during deploy is not an option.
  - 10y ago
renesd 10y ago
I think python packaging has gotten LOTS better in the last few years. I find it quite pleasurable to use these days. From binary wheels (including on different linux architectures), to things like local caching of packages (taking LOTS of load off the main servers). To the organisation github of pypa [0], to `python -m venv` working. Also lots of work around standardising things in peps, and writing documentation for people. I would like to applaud all the hard work people have done over the years on python packaging. It really is quite nice these days, and I look forward to all the improvements coming up (like pipenv!). I'd suggest people checkout fades [1] (for running scripts and automatically downloading dependencies in a venv), as well as conda [2] the alternative package manager. [0] https://github.com/pypa/ https://github.com/pypa/ [1] https://fades.readthedocs.io/en/release-5/readme.html#what-does-it-do https://fades.readthedocs.io/en/release-5/readme.html#what-d... [2] http://conda.pydata.org/docs/intro.html http://conda.pydata.org/docs/intro.html
- sametmax 10y ago
  +1. Relatively to what we have before, it's so much better. But compared to the JS/Rust ecosystem, we are behind. Now it's hard to compete with JS on some stuff : it's the only language in the most popular dev plateform (the web) and it has one implicit standardized async model by default. It's hard to compete with rust on some stuff : it's compiled and is fast, can provide stand alone binaries easily and has a checker that can avoid many bugs. But this. The package manager. We can compete. And yet we are late. It's partially my fault since it's a project I had in mind for years and never took the time to work on. It's partially everybody's fault I guess :)
  - Dowwie 10y ago
    It was definitely your fault. Thanks for nothing.
    - sametmax 10y ago
      :) In the open source world, and given my skills, I always feel a bit responsible when a tool doesn't exist.
ghshephard 10y ago
Okay - maybe I'm missing something, but pip is the only Python package manager I've ever used. And it's basically "pip install xxx", "pip install --upgrade xxx", "pip show xxx", "pip list", "pip uninstall xxx" I'm curious what I've been missing about pip that makes it problematic - I've never used the other tools you mentioned (setuptools/distutils/ez_install) - so I can't comment on them, but, on the flip side, I've never had to use them, so maybe my requirements are somewhat more simple than yours.
- sametmax 10y ago
  One things is a good dependency management. Right now if you want to upgrade your Python version, or one of your packages, it's a mountain of manual work. There is nothing in the stack helping you with the dependency graph. Another thing is providing a stand alone build. Something you can just ship without asking the client to run commands in the terminal to make it work. I use nuikta (http://nuitka.net/ http://nuitka.net/) for this. It's a fantastic project, but man it's a lot of work for something that works out of the box in Go or Rust. One last thing is to generate packages for OS (msi/deb/rpm/dmg/snap). Your sysadmin will like you. Pex (https://pypi.python.org/pypi/pex https://pypi.python.org/pypi/pex) is the closest, but not very standard. Other pet peeves of mine: - you can't easily move virtualenvs; - creating a setup.py is very hard for a beginner and has numerous traps; - setup.py are executable files. Meh. - what's with this setup.cfg containing 2 lines ? And the MANIFEST.IN being a separate files. Why do I have to put conf also in tox.ini ? And one for each of my linters ? I want ONE setup.cfg file with all config for all tools for my project inside and be done with it. TOML can handle rich sections, just stop creating new files. - accessing file with pkg_resources() is way harder than it should be. I made a wrapper for this (http://sametmax.com/embarquer-un-fichier-non-python-proprement/ http://sametmax.com/embarquer-un-fichier-non-python-propreme...). - one place to have __version__, please. I want it readable in my code AND in my package metadata, without having to use regex or have side effects on imports. - remove the "can't build wheel message" when it's useless. It scares newcomers. - README is the long_description. Don't make me read it manually. - how do I provide vendors in a clean way ? - install_requires, extras_requires, setup_requires, tests_requires... Make it one require with hooks and tags and be done with it. - creating a setup.py test config is way harder than it should be and breaks in CI on strange edge cases. - can we get a PEP on the standard project structure and built it in our tools to be done with it? We all have src/package + setup.py on root anyway. - pip installs packages in the site-packages dir of the python executable it's installed for. It makes sense, and I think Python deals pretty well with the fact you can have various versions installed on the same machine. But God people are confused by this. Now you can recommend to do "python -m pip", but it's very verbose and it assumes people know what version of Python is behind the "python" executable. On windows it can be any, and they must chose with... yet another command ("py")! pipenv just bypass that by assuming you want a virtualenv, and be able to access it. It's a very good call. - pip install --user will create commands you can't use unless you edit your PATH. This makes newcomers go mad.
deleted 10y ago
[deleted]
edblarney 10y ago
"This is great, but sometimes I think that python needs a new package manager from scratch" Ha, that's what I came here to say! Or better - a new packaging paradigm. Maybe it's extremely uncool to say ... but I think Java still has the best packaging paradigm of all languages. Jars rule. Of course 'gradle' is kind of a confusing mess so they don't have dependencies worked out very well ... Nevertheless I do feel that Python's packaging and dependency/versioning woes create a much bigger systematic problem than many realize. Kudos to the author though ...
ellisv 10y ago
I don't disagree but your suggestion sort of reminds me of this xkcd: http://xkcd.com/927/ http://xkcd.com/927/
barbs 10y ago
All you've said here is "python packaging sucks", with no explanation why, and with no alternative. Not a substantial comment, and I'm disappointed that it's been voted to the top. I'd ask for your reasoning but it seems sametmax has done a good job of that for you: https://news.ycombinator.com/item?id=13460490 https://news.ycombinator.com/item?id=13460490
- jessaustin 10y ago
  Comments that make a single unambiguous point are fine. It's no problem to leave detailed support to replies. A more controversial statement with the same content probably wouldn't have been voted up, but thread parent is objectively true even though it doesn't contain its own proof.

by cderwin 10y ago

sametmax 10y ago

I strongly agree but this can work now and is a big improvement over what we currently have. So while I would literally pay to see somebody work on a better package manager (which can generate exe, deb and use a conf file indead of .py), this is a good filler.

X-Istence 10y ago
Why should it generate a deb file? How is this useful on anything but Debian based systems? Why exe? How do you package libraries using this new tool you are envisioning?
- sametmax 10y ago
  You do package lib the same way as before, although cargo like dependency handling would be a nice thing. Especially for upgrades. But a good package manager should ALSO allow you to produce a: - a stand alone executable for most OS. - a standard package for major OS (msi, deb, snap, rpm, dmg, etc). Doing that right now with Python requires you to setup stuff like nuikta and the like. It works but it's much harder than it should be.
  - snuxoll 10y ago
    debhelper pretty much automates the process of packaging any standard distutils or setuptools package, Red Hat distributions have templates for packaging Python libraries as well (and rpmdev-newspec python-mypackage will automatically generate an appropriate .spec file). Windows and OS X are always a pain in the ass, but that's more an issue of the platforms lacking in package management than anything else.
    - sametmax 10y ago
      See my point ? There is a way to do it, it's just a pain. Now pipenv centralize stuff we were doing anyway. We should have a tool to centralize those as well.
      - snuxoll 10y ago
        I would hardly call it a pain, it takes me all of 3 minutes to write a .spec for most python packages and from there it's basically 'tito release'. Sure, if I wanted to package for debian-based distributions it'd take a little more time, but it's worth it to make a quality package that a distribution itself can decide to pick up (packagers love other people doing the work for them, though they won't refuse doing it themselves) with minimal effort.
        sametmax 10y ago
        3 minutes because you know how.
        snuxoll 10y ago
        And it took me all of 60 minutes to learn, packaging isn't anywhere near as hard as people make it out to be. If you can use a build system of your choosing to build and test your project, you can learn how to write an rpmspec or debian control file in under an hour.
        10y ago

ploggingdev 10y ago

Just curious, what aspects of pip/virtualenv specifically do you find subpar in comparison to other languages' package managers.

cderwin 10y ago
I would look at this comment[0] by sametmax for a critique of pip. My main gripe with virtualenv is that it's required at all: other interpreted languages, like node and elixir for example, have figured out how to handle non-global dependencies without a third-party package. Beyond that, it's frustrating to deploy because its non-relocatable (in our build/deploy scripts at my last python job we had to use sed all over the place to fix paths), and I find it semi-annoying to have a bunch of different copies of interpreter and all that goes with it (though this is mostly a minor annoyance -- it doesn't take up that much space and it doesn't matter if it gets out of sync. Also notable, IMO, is the lack of a tool like rbenv or rustup for python. I can't tell you how many times I have had to try to figure out which python version a given pip executable worked with. [0] https://news.ycombinator.com/item?id=13460490 https://news.ycombinator.com/item?id=13460490
- catdog 10y ago
  > Beyond that, it's frustrating to deploy because its non-relocatable (in our build/deploy scripts at my last python job we had to use sed all over the place to fix paths) pip does cache the wheels so instead of moving the virtualenvs around, just recreate them. This also ensures the virtualenv is up to date. Using tox this is fairly easy to do. Sure virtualenv is a bit of a hack but it's not that bad.
  - cderwin 10y ago
    I'm mostly talking about moving between different machines. I would like to be able to tar up my source code and venv, distribute it to multiple machines, untar it and run it. However that's not possible with virtualenv unless you do a lot of hackery in your build. In particular, creating a virtualenv on each server during deploy is not an option.
- 10y ago

renesd 10y ago

I think python packaging has gotten LOTS better in the last few years. I find it quite pleasurable to use these days. From binary wheels (including on different linux architectures), to things like local caching of packages (taking LOTS of load off the main servers). To the organisation github of pypa [0], to `python -m venv` working. Also lots of work around standardising things in peps, and writing documentation for people. I would like to applaud all the hard work people have done over the years on python packaging. It really is quite nice these days, and I look forward to all the improvements coming up (like pipenv!). I'd suggest people checkout fades [1] (for running scripts and automatically downloading dependencies in a venv), as well as conda [2] the alternative package manager. [0] https://github.com/pypa/ https://github.com/pypa/ [1] https://fades.readthedocs.io/en/release-5/readme.html#what-does-it-do https://fades.readthedocs.io/en/release-5/readme.html#what-d... [2] http://conda.pydata.org/docs/intro.html http://conda.pydata.org/docs/intro.html

sametmax 10y ago
+1. Relatively to what we have before, it's so much better. But compared to the JS/Rust ecosystem, we are behind. Now it's hard to compete with JS on some stuff : it's the only language in the most popular dev plateform (the web) and it has one implicit standardized async model by default. It's hard to compete with rust on some stuff : it's compiled and is fast, can provide stand alone binaries easily and has a checker that can avoid many bugs. But this. The package manager. We can compete. And yet we are late. It's partially my fault since it's a project I had in mind for years and never took the time to work on. It's partially everybody's fault I guess :)
- Dowwie 10y ago
  It was definitely your fault. Thanks for nothing.
  - sametmax 10y ago
    :) In the open source world, and given my skills, I always feel a bit responsible when a tool doesn't exist.

ghshephard 10y ago

Okay - maybe I'm missing something, but pip is the only Python package manager I've ever used. And it's basically "pip install xxx", "pip install --upgrade xxx", "pip show xxx", "pip list", "pip uninstall xxx" I'm curious what I've been missing about pip that makes it problematic - I've never used the other tools you mentioned (setuptools/distutils/ez_install) - so I can't comment on them, but, on the flip side, I've never had to use them, so maybe my requirements are somewhat more simple than yours.

sametmax 10y ago
One things is a good dependency management. Right now if you want to upgrade your Python version, or one of your packages, it's a mountain of manual work. There is nothing in the stack helping you with the dependency graph. Another thing is providing a stand alone build. Something you can just ship without asking the client to run commands in the terminal to make it work. I use nuikta (http://nuitka.net/ http://nuitka.net/) for this. It's a fantastic project, but man it's a lot of work for something that works out of the box in Go or Rust. One last thing is to generate packages for OS (msi/deb/rpm/dmg/snap). Your sysadmin will like you. Pex (https://pypi.python.org/pypi/pex https://pypi.python.org/pypi/pex) is the closest, but not very standard. Other pet peeves of mine: - you can't easily move virtualenvs; - creating a setup.py is very hard for a beginner and has numerous traps; - setup.py are executable files. Meh. - what's with this setup.cfg containing 2 lines ? And the MANIFEST.IN being a separate files. Why do I have to put conf also in tox.ini ? And one for each of my linters ? I want ONE setup.cfg file with all config for all tools for my project inside and be done with it. TOML can handle rich sections, just stop creating new files. - accessing file with pkg_resources() is way harder than it should be. I made a wrapper for this (http://sametmax.com/embarquer-un-fichier-non-python-proprement/ http://sametmax.com/embarquer-un-fichier-non-python-propreme...). - one place to have __version__, please. I want it readable in my code AND in my package metadata, without having to use regex or have side effects on imports. - remove the "can't build wheel message" when it's useless. It scares newcomers. - README is the long_description. Don't make me read it manually. - how do I provide vendors in a clean way ? - install_requires, extras_requires, setup_requires, tests_requires... Make it one require with hooks and tags and be done with it. - creating a setup.py test config is way harder than it should be and breaks in CI on strange edge cases. - can we get a PEP on the standard project structure and built it in our tools to be done with it? We all have src/package + setup.py on root anyway. - pip installs packages in the site-packages dir of the python executable it's installed for. It makes sense, and I think Python deals pretty well with the fact you can have various versions installed on the same machine. But God people are confused by this. Now you can recommend to do "python -m pip", but it's very verbose and it assumes people know what version of Python is behind the "python" executable. On windows it can be any, and they must chose with... yet another command ("py")! pipenv just bypass that by assuming you want a virtualenv, and be able to access it. It's a very good call. - pip install --user will create commands you can't use unless you edit your PATH. This makes newcomers go mad.

deleted 10y ago

[deleted]

edblarney 10y ago

"This is great, but sometimes I think that python needs a new package manager from scratch" Ha, that's what I came here to say! Or better - a new packaging paradigm. Maybe it's extremely uncool to say ... but I think Java still has the best packaging paradigm of all languages. Jars rule. Of course 'gradle' is kind of a confusing mess so they don't have dependencies worked out very well ... Nevertheless I do feel that Python's packaging and dependency/versioning woes create a much bigger systematic problem than many realize. Kudos to the author though ...

ellisv 10y ago

I don't disagree but your suggestion sort of reminds me of this xkcd: http://xkcd.com/927/ http://xkcd.com/927/

barbs 10y ago

All you've said here is "python packaging sucks", with no explanation why, and with no alternative. Not a substantial comment, and I'm disappointed that it's been voted to the top. I'd ask for your reasoning but it seems sametmax has done a good job of that for you: https://news.ycombinator.com/item?id=13460490 https://news.ycombinator.com/item?id=13460490

jessaustin 10y ago
Comments that make a single unambiguous point are fine. It's no problem to leave detailed support to replies. A more controversial statement with the same content probably wouldn't have been voted up, but thread parent is objectively true even though it doesn't contain its own proof.