8 ms·

> I am sick and tired of the python packaging space pushing this godawful tool for Python. Please just use pip-tools or poetry instead. As an outsider, Python

by orra 4y ago

> I am sick and tired of the python packaging space pushing this godawful tool for Python. Please just use pip-tools or poetry instead. As an outsider, Python packaging seems like a shit show, sorry. This bug in pipenv is bad, yes. But poetry was deliberately non-deterministically failing CI builds that used a deprecated feature. Maybe pip-tools is reliable and has a reasonable maintainer, but having to manually create venvs is mediocre UX. PDM sounds interesting because it doesn't need venvs under the hood, thanks to a recent PEP.

BerislavLopac 4y ago
> Python packaging seems like a shit show It does, but primarily because most of the tools - and that includes pipenv, Poetry, pdm and hatch - have crammed up to three separate functionalities into one (probably following the example of npm): packaging on one hand, dependency management on the other, and virtual environments on the third. The packaging question has mostly been resolved: distribute your packages as wheels and install them using pip; that's all. Dependency managament inherently has more complexity, and adding virtual environments in the mix, especially with the introduction of PEP 582 (as you mention), doesn't help things.
- coldtea 4y ago
  >primarily because most of the tools - and that includes pipenv, Poetry, pdm and hatch - have crammed up to three separate functionalities into one (probably following the example of npm): packaging on one hand, dependency management on the other, and virtual environments on the third. That never stopped npm to be way better than all of them... >The packaging question has mostly been resolved: distribute your packages as wheels and install them using pip; that's all. Dependency managament inherently has more complexity, and adding virtual environments in the mix, especially with the introduction of PEP 582 (as you mention), doesn't help things. So, the "solution" is "just give up"...
  - viraptor 4y ago
    Npm doesn't do environments. You have the choice of "in this directory" or "globally" and pray there are no symlinks involved or things fail mysteriously. It doesn't quite do packaging either. So it lands in a similar space to what the parent describes - mostly a good dependency manager.
    - coldtea 4y ago
      >Npm doesn't do environments. "This directory" (which can be overriden with env setting or similar advise for where to lookup) works fine for me, and never have been a problem in quite complex setups. Not sure why "symlinks" would be involved. In any case, what NPM does is leaps and bounds ahead of any shit Python offering we have to deal with, which is a constant source of trouble. IIRC, it's also the model ("this directory") that there's a PEP to imitatate, not sure what happened to that...
      - wirrbel 4y ago
        Back in the days we had the .eggs directory in python it worked fairly well except for binary packages so we got virtual environments.
        coldtea 4y ago
        What was the problem with binary packages for .eggs? Because, iirc, npm does handle e.g. .so files in packages that are based on C/C++ code, and puts them and everything else in the same, relocatable even (another pain point with virtualenvs), .node_modules directory.
      - viraptor 4y ago
        > which can be overriden with env setting Support for this is not great. Lots of packages still don't support this properly. My experience matches the 2015 comment https://github.com/npm/npm/issues/775#issuecomment-71294085 https://github.com/npm/npm/issues/775#issuecomment-71294085 > Not sure why "symlinks" would be involved. If you make your node_modules a symlink, multiple packages will fail. Even if you're not interested in doing that, others are. I tried the be symlinks because the env var option is not reliable. > What NPM does is leaps and bounds ahead Unless you change your node / gyp version. It doesn't really have a concept of runtime version. You can restrict it, but not have two concurrent versions if they conflict.
  - wirrbel 4y ago
    You meant to write „I am more used to npm and intuitively work around the quieks while in python i run into these“
    - coldtea 4y ago
      No, I meant to write "I've worked with Python for 20+ years, and Node for 8, and Python has always been a shit show on that front, whereas npm haven't given me any trouble (1)". But thanks for the uncharitable strawman. Couldn't be that I know both and don't like it, must be that I'm not familiar enough with the wonderland that is the Python package management... (1) At least not any trouble of that kind. I've had some issues regarding codependencies for add-ons, but that's a problem far beyond the kind of problems the Python package management has. If only Python's packaging stuff only had the same kind of issues...
    - nicoburns 4y ago
      What makes NPM a cut above Python package managers is that I can clone a JavaScript project, run npm install, and it will just work in such a way that it will not conflict with other projects. With python I have to work out which package manager it's using (the JavaScript ecosystem also has multiple package managers - yarn and pnpm - but they're compatible with each other!), and there's a significant chance it's using pip which installs packages globally unless I manually create and manage a virtualenv. The gold standard is probably Rust's "cargo" (and "rustup" for toolchain version management), which broadly follows the NPM model but with a better implementation with fewer quirks. But NPM has gotten much better (it uses lock files by default now for example), and is mostly painless at this point.
      - Tainnor 4y ago
        > But NPM has gotten much better (it uses lock files by default now for example) Last time I checked, "npm install" will still overwrite your lock file, you have to explicitly use "npm ci" for that not to happen. Or maybe that changed recently?
        nicoburns 4y ago
  - baq 4y ago
    what? npm is, for a lack of better word, shit. not worse than Python situation (coming from somebody who was excited when new-style classes were introduced), but that is a very low bar. ($DAYJOB makes me use npm _and_ pnpm, nx, nvm, corepack and I guess some other tools I don't know about just to make the goddamn thing build.)
    - coldtea 4y ago
      Any argument why is it shit?
      - baq 4y ago
        it's so slow there's a whole ecosystem to avoid using it, edited my comment above.
        coldtea 4y ago
        I think it's not npm/yarn that is slow, it's that Node project have tons of dependencies and transitive dependencies and such (leftpad, anyone?) But that's more a fault of the library ecosystem and culture, than the design of npm/yarn
  - orra 4y ago
    > That never stopped npm to be way better than all of them... npm isn't my favourite, but it's definitely improved with the competition from yarn and pnpm. I think that Stack (Haskell) and Cargo (Rust) are state of the art, for ease of use and reproducibility. The other user would describe these as cramming three functionalities into one, but they do it so well.
    - Tainnor 4y ago
      Stack is good, but the Haskell community seems split between Stack and pure Cabal users... plus there's also ghcup and it's not easy to understand whether one should have stack manage GHC versions or no... every source tells you something different. I also hate that there's no good way to install a depedency globally so you can play around with it (you kinda can, but it's very hard to remove it afterwards). But the real shitshow is the HLS that just crashes on me all the time and doesn't always work correctly with Stack.
- mathstuf 4y ago
  Alas, wheels are terrible for providing anything but a Python API. If you have headers, compiled tools, or some other complicated SDK, good luck (NumPy is excepted here because it only ships headers this way and chooses to live with wheel limitations). Wheels depending on you also ship copies of your library (if you use PyPI), so the lackluster dependency management is just waiting to mix things up and give your users "fun" linker/runtime issues.
- Tainnor 4y ago
  > It does, but primarily because most of the tools - and that includes pipenv, Poetry, pdm and hatch - have crammed up to three separate functionalities into one (probably following the example of npm): packaging on one hand, dependency management on the other, and virtual environments on the third. Many build tools in other languages also do all of these three things - for example Ruby's Bundler or maven/gradle for JVM languages. And they're pretty reliable at all of these tasks. (I guess, "virtual environments" is maybe not the right description. What these tools all do is isolating dependencies of different projects from each other, although they might do it in slightly different ways.)
- fabioz 4y ago
  Package management in Python is mostly solved to me with conda+conda devenv+conda lock. Sometimes I still need to get packages from pip in that scenario, but conda makes package management a lot better than pip+alternatives for me.
baggiponte 4y ago
PDM in my experience has been the best, plus it's rapidly growing, and is more PEP compliant than poetry. My go-to choice, been advocating it since I found out about it!
- wirrbel 4y ago
  I have used python for 15 years but less frequently nowadays because engineering manager. What irks me out about poetry is that it tries to be too magic and in the end just is oddly surprising. Will pdm feel „more solid“ to me?
andreareina 4y ago
> But poetry was deliberately non-deterministically failing CI builds that used a deprecated feature. I’m having difficulty finding a reference on this, could you elaborate?
- orra 4y ago
  https://github.com/python-poetry/poetry/blob/a613347bf0a9fd7b7ec782cd45222b3a0f0e7401/get-poetry.py#L473 https://github.com/python-poetry/poetry/blob/a613347bf0a9fd7...
n8henrie 4y ago
Interesting that you call out venv creation as a UX problem. I never considered it any worse than having to `cargo new` a project. I find having to activate and deactivate vs trust a 3rd party project that watches my every `cd` a much bigger issue.

by orra 4y ago

BerislavLopac 4y ago

> Python packaging seems like a shit show It does, but primarily because most of the tools - and that includes pipenv, Poetry, pdm and hatch - have crammed up to three separate functionalities into one (probably following the example of npm): packaging on one hand, dependency management on the other, and virtual environments on the third. The packaging question has mostly been resolved: distribute your packages as wheels and install them using pip; that's all. Dependency managament inherently has more complexity, and adding virtual environments in the mix, especially with the introduction of PEP 582 (as you mention), doesn't help things.

coldtea 4y ago
>primarily because most of the tools - and that includes pipenv, Poetry, pdm and hatch - have crammed up to three separate functionalities into one (probably following the example of npm): packaging on one hand, dependency management on the other, and virtual environments on the third. That never stopped npm to be way better than all of them... >The packaging question has mostly been resolved: distribute your packages as wheels and install them using pip; that's all. Dependency managament inherently has more complexity, and adding virtual environments in the mix, especially with the introduction of PEP 582 (as you mention), doesn't help things. So, the "solution" is "just give up"...
- viraptor 4y ago
  Npm doesn't do environments. You have the choice of "in this directory" or "globally" and pray there are no symlinks involved or things fail mysteriously. It doesn't quite do packaging either. So it lands in a similar space to what the parent describes - mostly a good dependency manager.
  - coldtea 4y ago
    >Npm doesn't do environments. "This directory" (which can be overriden with env setting or similar advise for where to lookup) works fine for me, and never have been a problem in quite complex setups. Not sure why "symlinks" would be involved. In any case, what NPM does is leaps and bounds ahead of any shit Python offering we have to deal with, which is a constant source of trouble. IIRC, it's also the model ("this directory") that there's a PEP to imitatate, not sure what happened to that...
    - wirrbel 4y ago
      Back in the days we had the .eggs directory in python it worked fairly well except for binary packages so we got virtual environments.
      - coldtea 4y ago
        What was the problem with binary packages for .eggs? Because, iirc, npm does handle e.g. .so files in packages that are based on C/C++ code, and puts them and everything else in the same, relocatable even (another pain point with virtualenvs), .node_modules directory.
    - viraptor 4y ago
      > which can be overriden with env setting Support for this is not great. Lots of packages still don't support this properly. My experience matches the 2015 comment https://github.com/npm/npm/issues/775#issuecomment-71294085 https://github.com/npm/npm/issues/775#issuecomment-71294085 > Not sure why "symlinks" would be involved. If you make your node_modules a symlink, multiple packages will fail. Even if you're not interested in doing that, others are. I tried the be symlinks because the env var option is not reliable. > What NPM does is leaps and bounds ahead Unless you change your node / gyp version. It doesn't really have a concept of runtime version. You can restrict it, but not have two concurrent versions if they conflict.
- wirrbel 4y ago
  You meant to write „I am more used to npm and intuitively work around the quieks while in python i run into these“
  - coldtea 4y ago
    No, I meant to write "I've worked with Python for 20+ years, and Node for 8, and Python has always been a shit show on that front, whereas npm haven't given me any trouble (1)". But thanks for the uncharitable strawman. Couldn't be that I know both and don't like it, must be that I'm not familiar enough with the wonderland that is the Python package management... (1) At least not any trouble of that kind. I've had some issues regarding codependencies for add-ons, but that's a problem far beyond the kind of problems the Python package management has. If only Python's packaging stuff only had the same kind of issues...
  - nicoburns 4y ago
    What makes NPM a cut above Python package managers is that I can clone a JavaScript project, run npm install, and it will just work in such a way that it will not conflict with other projects. With python I have to work out which package manager it's using (the JavaScript ecosystem also has multiple package managers - yarn and pnpm - but they're compatible with each other!), and there's a significant chance it's using pip which installs packages globally unless I manually create and manage a virtualenv. The gold standard is probably Rust's "cargo" (and "rustup" for toolchain version management), which broadly follows the NPM model but with a better implementation with fewer quirks. But NPM has gotten much better (it uses lock files by default now for example), and is mostly painless at this point.
    - Tainnor 4y ago
      > But NPM has gotten much better (it uses lock files by default now for example) Last time I checked, "npm install" will still overwrite your lock file, you have to explicitly use "npm ci" for that not to happen. Or maybe that changed recently?
      - nicoburns 4y ago
- baq 4y ago
  what? npm is, for a lack of better word, shit. not worse than Python situation (coming from somebody who was excited when new-style classes were introduced), but that is a very low bar. ($DAYJOB makes me use npm _and_ pnpm, nx, nvm, corepack and I guess some other tools I don't know about just to make the goddamn thing build.)
  - coldtea 4y ago
    Any argument why is it shit?
    - baq 4y ago
      it's so slow there's a whole ecosystem to avoid using it, edited my comment above.
      - coldtea 4y ago
        I think it's not npm/yarn that is slow, it's that Node project have tons of dependencies and transitive dependencies and such (leftpad, anyone?) But that's more a fault of the library ecosystem and culture, than the design of npm/yarn
- orra 4y ago
  > That never stopped npm to be way better than all of them... npm isn't my favourite, but it's definitely improved with the competition from yarn and pnpm. I think that Stack (Haskell) and Cargo (Rust) are state of the art, for ease of use and reproducibility. The other user would describe these as cramming three functionalities into one, but they do it so well.
  - Tainnor 4y ago
    Stack is good, but the Haskell community seems split between Stack and pure Cabal users... plus there's also ghcup and it's not easy to understand whether one should have stack manage GHC versions or no... every source tells you something different. I also hate that there's no good way to install a depedency globally so you can play around with it (you kinda can, but it's very hard to remove it afterwards). But the real shitshow is the HLS that just crashes on me all the time and doesn't always work correctly with Stack.
mathstuf 4y ago
Alas, wheels are terrible for providing anything but a Python API. If you have headers, compiled tools, or some other complicated SDK, good luck (NumPy is excepted here because it only ships headers this way and chooses to live with wheel limitations). Wheels depending on you also ship copies of your library (if you use PyPI), so the lackluster dependency management is just waiting to mix things up and give your users "fun" linker/runtime issues.
Tainnor 4y ago
> It does, but primarily because most of the tools - and that includes pipenv, Poetry, pdm and hatch - have crammed up to three separate functionalities into one (probably following the example of npm): packaging on one hand, dependency management on the other, and virtual environments on the third. Many build tools in other languages also do all of these three things - for example Ruby's Bundler or maven/gradle for JVM languages. And they're pretty reliable at all of these tasks. (I guess, "virtual environments" is maybe not the right description. What these tools all do is isolating dependencies of different projects from each other, although they might do it in slightly different ways.)
fabioz 4y ago
Package management in Python is mostly solved to me with conda+conda devenv+conda lock. Sometimes I still need to get packages from pip in that scenario, but conda makes package management a lot better than pip+alternatives for me.

baggiponte 4y ago

PDM in my experience has been the best, plus it's rapidly growing, and is more PEP compliant than poetry. My go-to choice, been advocating it since I found out about it!

wirrbel 4y ago
I have used python for 15 years but less frequently nowadays because engineering manager. What irks me out about poetry is that it tries to be too magic and in the end just is oddly surprising. Will pdm feel „more solid“ to me?

andreareina 4y ago

> But poetry was deliberately non-deterministically failing CI builds that used a deprecated feature. I’m having difficulty finding a reference on this, could you elaborate?

orra 4y ago
https://github.com/python-poetry/poetry/blob/a613347bf0a9fd7b7ec782cd45222b3a0f0e7401/get-poetry.py#L473 https://github.com/python-poetry/poetry/blob/a613347bf0a9fd7...

n8henrie 4y ago

Interesting that you call out venv creation as a UX problem. I never considered it any worse than having to `cargo new` a project. I find having to activate and deactivate vs trust a 3rd party project that watches my every `cd` a much bigger issue.