7 ms·

Pipenv has spawned so much controvery.. One big shitfest. Python packaging in general is such a messy ecosystem

by mikkelam 7y ago

briandilley 7y ago
Biome. We no longer say ecosystem, we say "Biome".
- emddudley 7y ago
  OK so we're sidetracking here, but I have never seen anyone use "biome". Why change from "ecosystem"?
  - MS90 7y ago
    Curious as well. The thesaurus lists them as synonyms, though it appears there are some minor differences in meaning when used as scientific terms.
josteink 7y ago
> Python packaging in general is such a messy ecosystem Not just messy. It’s probably worst in class. I at least can’t come up with a single worse package-management story which I do know of. Edit: I’m talking about platforms with actual package-management which sucks, not platforms with the absence of package-management all together.
- deleted 7y ago
  [deleted]
- rthomas6 7y ago
  C/C++.
  - gwd 7y ago
    C/C++'s package management story is "defer it to distributions". Personally I much prefer `apt-get install libfoo-dev` to 1) each language inventing its own incompatible system, and 2) each developer self-publishing, so there's little to no safety or accountability when adding a new dependency.
    - rthomas6 7y ago
      That's a fair point. I live in embedded world most of the time, and using third party libraries in that space is not always so easy :)
    - yjftsjthsd-h 7y ago
      So, to be clear, I also tend to prefer distro-supported packaging. However. Forcing everything to go through the distro means that you massively limit what's available by raising the barriers to entry, you slow pushing new versions (ranging from Arch's "as soon as a packager gets to it" to CentOS's "new major version will be available in 5 years"), and you lock yourself into each distro's packages and make portability a pain (Ubuntu ships libfoo-dev 1.2, Arch ships libfoo-dev 1.3, CentOS has libfoo-devel 0.6, and Debian doesn't package it at all). When distro packages work, they're great, but they do have shortcomings.
      - teddyh 7y ago
        > limit what's available by raising the barriers to entry But that’s what you would have to do youself anyway. You can’t use all fresh upstream version of everything, since they don’t all work together. So some versions you’ll have to hold off on, some other versions might require minor patching. But this is exactly what distro maintainers do.
    - MadWombat 7y ago
      > Personally I much prefer `apt-get install libfoo-dev` So, what do you do when different projects need different versions of libfoo? Right, you download and build it locally inside the project tree and fiddle with makefiles to link to the local version rather than the one installed by apt-get. So basically you do your own dependency management. Good luck with that.
  - McP 7y ago
    Conan is actually quite nice. I've been porting a series of projects to it and it's been a pleasant experience - Conan is very flexible, the documentation is thorough and the developers are very responsive on Slack/GitHub. https://conan.io/ https://conan.io/
- Tajnymag 7y ago
  Well, C/C++ dependencies are also a hell to reliably setup on different platforms.
  - josteink 7y ago
    But C/C++ is messy because it doesn’t have a language/platform-provided package-management system, while python actually does. So that’s apples to no oranges, I guess. And it still doesn’t leave python looking particularly good.
    - Scarblac 7y ago
      But the Python mess because initially, it didn't have such a system either. Then a bunch of them were introduced, and some of them became official but there was never one that solved all the problems.
    - linuxftw 7y ago
      I would consider the packages shipped by most distros as platform-provided package-management. It's not specific to C/C++, but if I'm developing something for say, Debian, I'm going to use as many system-provided libs as possible.
  - amlozano 7y ago
    Have you tried Conan[1] by chance? I've only used it for small projects, but it was pretty nice. https://conan.io/ https://conan.io/
- theamk 7y ago
  Cmake? (And especially Kitware Superbuild... try pinning packages there!)
  - mateuszf 7y ago
    Isn't it a build tool, not package manager?
    - theamk 7y ago
      It is used as package manager often enough -- stick "git clone", or ExternalProject into the CMakefile and you have the (bad) equivalent of "pip install" for C++. For example, I mentioned "kitware superbuild" above. It describes itself [0] as: > It is basically a poor man’s package manager that you make consistently work across your target build platforms [0] https://blog.kitware.com/cmake-superbuilds-git-submodules/ https://blog.kitware.com/cmake-superbuilds-git-submodules/
- madeofpalk 7y ago
  I thought NPM was everyone's favourite hated package manager?
- hultner 7y ago
  As someone who's been in software development for two the better part of two decades now I would say that's far beyond the truth. Yes it's not great right now and it's not completely clear if I should recommend a new user pip+venv, pipenv, pyenv, poetry or conda and that's a big problem, Python was actually quite early to handle dependencies and packaging in a standardized and structured way. I remember that most of the projects I worked on early on in my career more or less completely relied on vendoring dependencies, if packaging systems existed they were of either very complex or the one that came with your os/distro. However Python currently needs to catch up, we need some alignment and a clear path forward, it would also be great to have an official way of building artefacts. There's plenty of ways today but no great blessed way to recommend a newcomer.
  - reledi 7y ago
    To add to the confusion, there’s also setuptools, easy_install, wheels, eggs, ...
  - cure 7y ago
    > As someone who's been in software development for two the better part of two decades now I would say that's far beyond the truth. As someone who has had to deal with Python's packaging a lot, I would say that it's really, really bad. Maybe not the worst in the world, but it is much closer to the worst than to the best. There are 15 ways to package Python modules, none of which are feature complete (but a lot of them pretend to be). Every year or so someone decides that they need to write another packaging tool for Python, and then they give up before it is feature complete. A tool like virtualenv is useful, but it overpromises and underdelivers (this is the overarching theme of Python packaging imo): it does not completely separate your environment from the system. E.g. virtualenv still uses the host system python packages cache, and it does something rather nasty with .so's that get copied from the host system... And don't get me started on dependency management in Python. When the dependencies of your dependencies make breaking interface changes, you're in for a world of pain. I find it amusing, in a sad way, that the language that prides itself on "there is one way to do it" screwed up so badly in the packaging department by not having one good way to do it. Meanwhile the language whose motto is "there is more than one way to do it" has a standard, sane way to package things, with multiple tools that work together in a coherent way. Perl got this one right.
- madeofpalk 7y ago
  I thought NPM was everyone's favourite hated package manager/repository?
  - josteink 7y ago
    It clearly has its faults, but it is simple to use, requires no “venvs”, is CI-friendly and mostly does what it’s supposed to with few surprises. Much unlike how things work with python.
    - fock 7y ago
      And it comes from a time, where a 2MB-webpage is considered acceptable. The whole idea of venvs is basically still considering some reuse. If you shout out loud "SCREW YOU LIBRARY REUSE", then npm is perfect of course...
  - rpedela 7y ago
    It is, at least on HN, but it is also one of the best overall.
  - ufmace 7y ago
    The faults aren't so much with the actual NPM software as with the whole ecosystem. The real root of the problem IMO is that Javascript has such a tiny std lib compared to other popular languages that have package management systems. This encourages lots of people who are missing various functions normally found in std libs to write packages implementing various combinations of those functions. Those who are writing larger packages then depend on various combinations of those little helper packages. So of course, if you need to use multiple large packages to do something useful, they'll tend to pull in a huge forest of tiny packages in a bunch of versions.
    - egeozcan 7y ago
      NPM itself was also not great, but did become great in the last majors. The weakness of the stdlib is still a problem but very, very slowly getting better with the new Ecmascript and node.js versions.
- toraobo 7y ago
  What would be an example of a good packaging ecosystem?
  - nonbirithm 7y ago
    Probably cargo. - It was built in to Rust from the beginning and officially sanctioned so fifteen different people don't have to build their own incomplete, buggy package managers. - You can add plugins for things like automatically updating or adding dependencies. - It handles projects and subprojects. - All you have to do to install dependencies and run a project is "cargo run", reducing friction for getting into new projects. - For the most part, it just works. That said it isn't perfect, especially when needing custom build scripts, but it's good.
- cassianoleal 7y ago
  Golang is probably the worst. Python a close second.
- sosodev 7y ago
  I might be alone but I hate dealing with Java’s package management system.
WilliamEdward 7y ago
Just curious, you don't find learning entire languages hard or confusing but suddenly writing 3 commands to set up a venv is too 'messy'?
- nwsm 7y ago
  >you don't find learning entire languages hard or confusing source?
- yboris 7y ago
  I think the challenge is that some dependencies have sub-dependencies that need to be different versions, and venv doesn't handle that automatically.
- kennydude 7y ago
  Not everyone uses the same 3 commands which means for a new python dev, finding out about venvs isn't as immediately obvious as it should be. Even then it's shooved (sadly) as an optional thing
- earthboundkid 7y ago
  It's a pain in the ass when you have colleagues who only know JS or HTML/CSS or whatever and they need to install the project to run it on their machines, and you have to explain "virtual envs" and other bullshit that doesn't exist in sane packaging ecosystems to get them up and running.
fock 7y ago
maybe for web-development and ML. For my core scientific workflows I have about, uuuh, 10(?) libraries. And those are generally stable enough to upgrade 1 without breaking the other. Seems like the problem is with the users.
tiborsaas 7y ago
Do you think Poetry solves this package management mess?
- snypox 7y ago
  Well, I'd say it makes it so much smoother, almost 'not a pain' anymore.
- scrollaway 7y ago
  To a point. Poetry still isn't fully in board with being a project manager, it focuses too much on packaging. For example it doesn't do project scripts like nom scripts.
  - williamdclt 7y ago
    To be fair, NPM really doesn't do a great job at project scripts
  - xkcd-sucks 7y ago
    It doesn't support shell scripts which kind of sucks, but could be reasonable if it is expected to run on windows as well as posix. This is the scripts section from one project I used Poetry in. It seems to be fine for most things, even though the focus on managing a python module for publication is a little bit annoying. Once or twice I've run into version resolution troubles, but generally it's stable and much more ergonomic+easier to explain than pip+venv. One can of course use `poetry run pip freeze` to generate the standard `requirements.txt` file [tool.poetry.scripts] # NB: Poetry won't run arbitrary callables, so these are wrapped in a python script. # Poetry needs the root dir to be a package (e.g. module `my_repo` must exist) in order to run scripts # So they are just shoved into a dummy "my_repo.py" instead of going in e.g. "scripts.py" # Adds Git hooks from `git_hooks` path (overwrites existing!) hookup = "my_repo:hookup" # Run formatter+linter, does not modify anything, errors if fail lint = "my_repo:lint" # runs `black` (formatter), modifies stuff format = "my_repo:format" # A list of files which failed linting. Most helpful with flake8 vim integration https://github.com/nvie/vim-flake8 lintyfiles = "my_repo:lintyfiles" # unit tests utest = "my_repo:utest" # unit + integration tests in container citest = "my_repo:citest" cutest = "my_repo:cutest"

Pipenv has spawned so much controvery.. One big shitfest. Python packaging in general is such a messy ecosystem

briandilley 7y ago

Biome. We no longer say ecosystem, we say "Biome".

emddudley 7y ago
OK so we're sidetracking here, but I have never seen anyone use "biome". Why change from "ecosystem"?
- MS90 7y ago
  Curious as well. The thesaurus lists them as synonyms, though it appears there are some minor differences in meaning when used as scientific terms.

josteink 7y ago

> Python packaging in general is such a messy ecosystem Not just messy. It’s probably worst in class. I at least can’t come up with a single worse package-management story which I do know of. Edit: I’m talking about platforms with actual package-management which sucks, not platforms with the absence of package-management all together.

deleted 7y ago
[deleted]
rthomas6 7y ago
C/C++.
- gwd 7y ago
  C/C++'s package management story is "defer it to distributions". Personally I much prefer `apt-get install libfoo-dev` to 1) each language inventing its own incompatible system, and 2) each developer self-publishing, so there's little to no safety or accountability when adding a new dependency.
  - rthomas6 7y ago
    That's a fair point. I live in embedded world most of the time, and using third party libraries in that space is not always so easy :)
  - yjftsjthsd-h 7y ago
    So, to be clear, I also tend to prefer distro-supported packaging. However. Forcing everything to go through the distro means that you massively limit what's available by raising the barriers to entry, you slow pushing new versions (ranging from Arch's "as soon as a packager gets to it" to CentOS's "new major version will be available in 5 years"), and you lock yourself into each distro's packages and make portability a pain (Ubuntu ships libfoo-dev 1.2, Arch ships libfoo-dev 1.3, CentOS has libfoo-devel 0.6, and Debian doesn't package it at all). When distro packages work, they're great, but they do have shortcomings.
    - teddyh 7y ago
      > limit what's available by raising the barriers to entry But that’s what you would have to do youself anyway. You can’t use all fresh upstream version of everything, since they don’t all work together. So some versions you’ll have to hold off on, some other versions might require minor patching. But this is exactly what distro maintainers do.
  - MadWombat 7y ago
    > Personally I much prefer `apt-get install libfoo-dev` So, what do you do when different projects need different versions of libfoo? Right, you download and build it locally inside the project tree and fiddle with makefiles to link to the local version rather than the one installed by apt-get. So basically you do your own dependency management. Good luck with that.
- McP 7y ago
  Conan is actually quite nice. I've been porting a series of projects to it and it's been a pleasant experience - Conan is very flexible, the documentation is thorough and the developers are very responsive on Slack/GitHub. https://conan.io/ https://conan.io/
Tajnymag 7y ago
Well, C/C++ dependencies are also a hell to reliably setup on different platforms.
- josteink 7y ago
  But C/C++ is messy because it doesn’t have a language/platform-provided package-management system, while python actually does. So that’s apples to no oranges, I guess. And it still doesn’t leave python looking particularly good.
  - Scarblac 7y ago
    But the Python mess because initially, it didn't have such a system either. Then a bunch of them were introduced, and some of them became official but there was never one that solved all the problems.
  - linuxftw 7y ago
    I would consider the packages shipped by most distros as platform-provided package-management. It's not specific to C/C++, but if I'm developing something for say, Debian, I'm going to use as many system-provided libs as possible.
- amlozano 7y ago
  Have you tried Conan[1] by chance? I've only used it for small projects, but it was pretty nice. https://conan.io/ https://conan.io/
theamk 7y ago
Cmake? (And especially Kitware Superbuild... try pinning packages there!)
- mateuszf 7y ago
  Isn't it a build tool, not package manager?
  - theamk 7y ago
    It is used as package manager often enough -- stick "git clone", or ExternalProject into the CMakefile and you have the (bad) equivalent of "pip install" for C++. For example, I mentioned "kitware superbuild" above. It describes itself [0] as: > It is basically a poor man’s package manager that you make consistently work across your target build platforms [0] https://blog.kitware.com/cmake-superbuilds-git-submodules/ https://blog.kitware.com/cmake-superbuilds-git-submodules/
madeofpalk 7y ago
I thought NPM was everyone's favourite hated package manager?
hultner 7y ago
As someone who's been in software development for two the better part of two decades now I would say that's far beyond the truth. Yes it's not great right now and it's not completely clear if I should recommend a new user pip+venv, pipenv, pyenv, poetry or conda and that's a big problem, Python was actually quite early to handle dependencies and packaging in a standardized and structured way. I remember that most of the projects I worked on early on in my career more or less completely relied on vendoring dependencies, if packaging systems existed they were of either very complex or the one that came with your os/distro. However Python currently needs to catch up, we need some alignment and a clear path forward, it would also be great to have an official way of building artefacts. There's plenty of ways today but no great blessed way to recommend a newcomer.
- reledi 7y ago
  To add to the confusion, there’s also setuptools, easy_install, wheels, eggs, ...
- cure 7y ago
  > As someone who's been in software development for two the better part of two decades now I would say that's far beyond the truth. As someone who has had to deal with Python's packaging a lot, I would say that it's really, really bad. Maybe not the worst in the world, but it is much closer to the worst than to the best. There are 15 ways to package Python modules, none of which are feature complete (but a lot of them pretend to be). Every year or so someone decides that they need to write another packaging tool for Python, and then they give up before it is feature complete. A tool like virtualenv is useful, but it overpromises and underdelivers (this is the overarching theme of Python packaging imo): it does not completely separate your environment from the system. E.g. virtualenv still uses the host system python packages cache, and it does something rather nasty with .so's that get copied from the host system... And don't get me started on dependency management in Python. When the dependencies of your dependencies make breaking interface changes, you're in for a world of pain. I find it amusing, in a sad way, that the language that prides itself on "there is one way to do it" screwed up so badly in the packaging department by not having one good way to do it. Meanwhile the language whose motto is "there is more than one way to do it" has a standard, sane way to package things, with multiple tools that work together in a coherent way. Perl got this one right.
madeofpalk 7y ago
I thought NPM was everyone's favourite hated package manager/repository?
- josteink 7y ago
  It clearly has its faults, but it is simple to use, requires no “venvs”, is CI-friendly and mostly does what it’s supposed to with few surprises. Much unlike how things work with python.
  - fock 7y ago
    And it comes from a time, where a 2MB-webpage is considered acceptable. The whole idea of venvs is basically still considering some reuse. If you shout out loud "SCREW YOU LIBRARY REUSE", then npm is perfect of course...
- rpedela 7y ago
  It is, at least on HN, but it is also one of the best overall.
- ufmace 7y ago
  The faults aren't so much with the actual NPM software as with the whole ecosystem. The real root of the problem IMO is that Javascript has such a tiny std lib compared to other popular languages that have package management systems. This encourages lots of people who are missing various functions normally found in std libs to write packages implementing various combinations of those functions. Those who are writing larger packages then depend on various combinations of those little helper packages. So of course, if you need to use multiple large packages to do something useful, they'll tend to pull in a huge forest of tiny packages in a bunch of versions.
  - egeozcan 7y ago
    NPM itself was also not great, but did become great in the last majors. The weakness of the stdlib is still a problem but very, very slowly getting better with the new Ecmascript and node.js versions.
toraobo 7y ago
What would be an example of a good packaging ecosystem?
- nonbirithm 7y ago
  Probably cargo. - It was built in to Rust from the beginning and officially sanctioned so fifteen different people don't have to build their own incomplete, buggy package managers. - You can add plugins for things like automatically updating or adding dependencies. - It handles projects and subprojects. - All you have to do to install dependencies and run a project is "cargo run", reducing friction for getting into new projects. - For the most part, it just works. That said it isn't perfect, especially when needing custom build scripts, but it's good.
cassianoleal 7y ago
Golang is probably the worst. Python a close second.
sosodev 7y ago
I might be alone but I hate dealing with Java’s package management system.

WilliamEdward 7y ago

Just curious, you don't find learning entire languages hard or confusing but suddenly writing 3 commands to set up a venv is too 'messy'?

nwsm 7y ago
>you don't find learning entire languages hard or confusing source?
yboris 7y ago
I think the challenge is that some dependencies have sub-dependencies that need to be different versions, and venv doesn't handle that automatically.
kennydude 7y ago
Not everyone uses the same 3 commands which means for a new python dev, finding out about venvs isn't as immediately obvious as it should be. Even then it's shooved (sadly) as an optional thing
earthboundkid 7y ago
It's a pain in the ass when you have colleagues who only know JS or HTML/CSS or whatever and they need to install the project to run it on their machines, and you have to explain "virtual envs" and other bullshit that doesn't exist in sane packaging ecosystems to get them up and running.

fock 7y ago

maybe for web-development and ML. For my core scientific workflows I have about, uuuh, 10(?) libraries. And those are generally stable enough to upgrade 1 without breaking the other. Seems like the problem is with the users.

tiborsaas 7y ago

Do you think Poetry solves this package management mess?

snypox 7y ago
Well, I'd say it makes it so much smoother, almost 'not a pain' anymore.
scrollaway 7y ago
To a point. Poetry still isn't fully in board with being a project manager, it focuses too much on packaging. For example it doesn't do project scripts like nom scripts.
- williamdclt 7y ago
  To be fair, NPM really doesn't do a great job at project scripts
- xkcd-sucks 7y ago
  It doesn't support shell scripts which kind of sucks, but could be reasonable if it is expected to run on windows as well as posix. This is the scripts section from one project I used Poetry in. It seems to be fine for most things, even though the focus on managing a python module for publication is a little bit annoying. Once or twice I've run into version resolution troubles, but generally it's stable and much more ergonomic+easier to explain than pip+venv. One can of course use `poetry run pip freeze` to generate the standard `requirements.txt` file [tool.poetry.scripts] # NB: Poetry won't run arbitrary callables, so these are wrapped in a python script. # Poetry needs the root dir to be a package (e.g. module `my_repo` must exist) in order to run scripts # So they are just shoved into a dummy "my_repo.py" instead of going in e.g. "scripts.py" # Adds Git hooks from `git_hooks` path (overwrites existing!) hookup = "my_repo:hookup" # Run formatter+linter, does not modify anything, errors if fail lint = "my_repo:lint" # runs `black` (formatter), modifies stuff format = "my_repo:format" # A list of files which failed linting. Most helpful with flake8 vim integration https://github.com/nvie/vim-flake8 lintyfiles = "my_repo:lintyfiles" # unit tests utest = "my_repo:utest" # unit + integration tests in container citest = "my_repo:citest" cutest = "my_repo:cutest"