5 ms·

Because venv/pip/requirements is good enough for most cases. Most developers are introduced early to these tools and learn them good enough to work fine for the

Because venv/pip/requirements is good enough for most cases. Most developers are introduced early to these tools and learn them good enough to work fine for them. If there's no real problem with the existing tools, then people will not change their tested and trusted way of working for some fancy new tool, no matter what that new tool promises! There's an old engineering proverb about that: if it works don't mess with it.

rocmcd 5y ago
I think this is it. I still don’t know what benefits poetry adds over the typical venv + pip + requirements flow, because that flow has worked pretty well for me. Is there something crucial I’m missing? I feel like package management has largely been solved in python, but I admittedly haven’t done a lot of research into what other workflows are out there.
- PaulHoule 5y ago
  If you think ‘pip works’ then the programs you are building are too simple. I worked at a place where we did machine learning builds and neither pip or conda worked reliably. What was insidious was that it almost worked and people thought they could live around it’s limitations which turns every 2 hour job into a 15 hour job. (But they are data scientists…. A reliable build is like Bigfoot, cold fusion or Linear A to them.) Pip’s dependency solving strategy is not correct, especially if you install pip packages one at a time. A real dependency solver would download the metadata for all matching versions (can be done with two seeks for a wheel but gotta run setup.py for an egg…) do an smt solve and then install the wheels. Pip just starts installing packages, hopes for the best, sometimes backtracks, often gets stuck or does the wrong thing.
  - rocmcd 5y ago
    That's a fair point, most of the applications I've worked on have been relatively simple (or I've just gotten lucky). It makes sense that different issues crop up at different scales, given the complexity of different workflows and package dependencies. I also haven't worked in ML before, so that may be a class of complexity all on its own.
    - PaulHoule 5y ago
      Two issues turn up in in ML. One is that the dependencies are a beast. The risk that it won't find a solution between a number of libraries that are only compatible with certain versions is high. The other one is that ML projects themselves contain data, often large amounts of data. For instance a "word2vec" style model might be 1 gigabyte and it might be a part of another model. (Say you turn the words to vectors then put the vectors through a CNN or RNN.) The Python packaging system might be a logically correct place to store this data (a necessary part of the the model) but it's a big file that will cause hassles if you do everything right, big hassles if you do things wrong (like compress anaconda packages with slow bzip2, use whatever algorithm that Docker uses to superamplify I/O, ...) It is nice to pack all your "empty" models (ready to train) as python packages and maybe even your "trained models" (some data files to supplement the empty files) but it will take some iteration to make it all go smoothly.
- lmns 5y ago
  How do you pin all of your dependencies to a specific version with just venv, pip and a requirements.txt? How do you upgrade them later?
  - rocmcd 5y ago
    Versions are specified within the requirements.txt, if that's what you mean. Upgrading can be done by installing a later version of the package and rewriting the requirements file. I suspect my use cases haven't been as complicated as some of the others listed in this thread, which may be why I've never felt the need to look into poetry and others.
    - lmns 5y ago
      I know what you mean, but as soon as you have transitive dependencies it doesn't work at all. At that point you can't reproduce the same state of your venv at a later date because some minor version of a dependent package could change and maybe break your build.
umvi 5y ago
I've had more trouble with pip than any other package manager. Heck, one of my projects on GitHub currently has a broken CI pipeline because some newer version of pip broke deployments to pypi and I haven't investigated why yet. And plus, it's just super confusing. It's never been clear to me what the difference between pip and pip3 is. I thought pip3 was for python3 packages, but it seems like pip can also be used for python3 as well. But then sometimes it seems like if I use pip I only get the python2 version. With npm or go get, if I install a package I know exactly where it got installed a how to mnaually remove if needed. If I install something with pip, the newly installed package goes into some mysterious directory whose location is never quite certain. So yes, there are major problems with pip, hence why competitors are showing up like poetry.
- psidex 5y ago
  > It's never been clear to me what the difference between pip and pip3 is. I thought pip3 was for python3 packages, but it seems like pip can also be used for python3 as well. But then sometimes it seems like if I use pip I only get the python2 version. It's pretty simple; pip3 will always be for python3, pip2 will always be for python2. pip is context dependent, for example if you are in a python2 venv, it will use py2, and vice versa. > With npm or go get, if I install a package I know exactly where it got installed a how to mnaually remove if needed. If I install something with pip, the newly installed package goes into some mysterious directory whose location is never quite certain. If you pip install a library, it will almost definitely be installed in pythondir/lib/site-packages, unless you have messed around with your installation
- bostonsre 5y ago
  Think you might need to downgrade pip (using pip) to the version that is less strict when checking dependencies (i forget the exact version, think maybe 22). pip install "pip<22" To check which pip is for which python: pip --version Pyenv is pretty great for managing multiple python versions.
- dragonwriter 5y ago
  > It's never been clear to me what the difference between pip and pip3 is. pip3 is the name of the system installation of the pip executable corresponding to the python 3.x installation on linux systems (maybe Mac, too) when installed from system package managers on systems where python3 is the name of the python 3.x executable when installed from the package manager. This is not really a pip issue, but a “how python is packaged to support side-by-side system installations” issue.
protomikron 5y ago
Agreed. Similar to why Plan9 did not replace Unix, because Unix is good enough. It does not mean we can't do better, but the leap forward was (is) not big enough. At least that's my understanding.
- smitty1e 5y ago
  My theory on tech acceptance is that it's not good enough to be an improvement. To gain traction and acceptance, the offering must be several times better. Marketing also counts. Hence the lingering demise of python2.7: python3 didn't offer sufficient sizzle until ~3.6 or so.
golergka 5y ago
Npm is also good enough for most cases, but it hadn't stopped yarn from getting traction. May be Python developers are just more conservative?
- nurettin 5y ago
  They are simply uninterested. Poetry and pipenv both give you the opportunity to separate dev dependencies, keep your venv folder separate from the project, add platform specific packages and easily update all package versions. Pipenv will even load the .env file. After reading all this, your python developer will go "well I still don't know what it does that I can't do with venv/pip".
- uranusjr 5y ago
  Yarn got most of its popularity during a time when it was clearly ahead of npm in every way, but saw its share dropped when npm caught up. IMO Poetry came a bit too late; it would have been a tremendous hit in like 2015, but as the timeline be it is the leap forward is simply not enough. (Incidentally I think the same is happening for Yarn v2 as well; very good ideas, but too minor for most users to care.)

rocmcd 5y ago

I think this is it. I still don’t know what benefits poetry adds over the typical venv + pip + requirements flow, because that flow has worked pretty well for me. Is there something crucial I’m missing? I feel like package management has largely been solved in python, but I admittedly haven’t done a lot of research into what other workflows are out there.

PaulHoule 5y ago
If you think ‘pip works’ then the programs you are building are too simple. I worked at a place where we did machine learning builds and neither pip or conda worked reliably. What was insidious was that it almost worked and people thought they could live around it’s limitations which turns every 2 hour job into a 15 hour job. (But they are data scientists…. A reliable build is like Bigfoot, cold fusion or Linear A to them.) Pip’s dependency solving strategy is not correct, especially if you install pip packages one at a time. A real dependency solver would download the metadata for all matching versions (can be done with two seeks for a wheel but gotta run setup.py for an egg…) do an smt solve and then install the wheels. Pip just starts installing packages, hopes for the best, sometimes backtracks, often gets stuck or does the wrong thing.
- rocmcd 5y ago
  That's a fair point, most of the applications I've worked on have been relatively simple (or I've just gotten lucky). It makes sense that different issues crop up at different scales, given the complexity of different workflows and package dependencies. I also haven't worked in ML before, so that may be a class of complexity all on its own.
  - PaulHoule 5y ago
    Two issues turn up in in ML. One is that the dependencies are a beast. The risk that it won't find a solution between a number of libraries that are only compatible with certain versions is high. The other one is that ML projects themselves contain data, often large amounts of data. For instance a "word2vec" style model might be 1 gigabyte and it might be a part of another model. (Say you turn the words to vectors then put the vectors through a CNN or RNN.) The Python packaging system might be a logically correct place to store this data (a necessary part of the the model) but it's a big file that will cause hassles if you do everything right, big hassles if you do things wrong (like compress anaconda packages with slow bzip2, use whatever algorithm that Docker uses to superamplify I/O, ...) It is nice to pack all your "empty" models (ready to train) as python packages and maybe even your "trained models" (some data files to supplement the empty files) but it will take some iteration to make it all go smoothly.
lmns 5y ago
How do you pin all of your dependencies to a specific version with just venv, pip and a requirements.txt? How do you upgrade them later?
- rocmcd 5y ago
  Versions are specified within the requirements.txt, if that's what you mean. Upgrading can be done by installing a later version of the package and rewriting the requirements file. I suspect my use cases haven't been as complicated as some of the others listed in this thread, which may be why I've never felt the need to look into poetry and others.
  - lmns 5y ago
    I know what you mean, but as soon as you have transitive dependencies it doesn't work at all. At that point you can't reproduce the same state of your venv at a later date because some minor version of a dependent package could change and maybe break your build.

umvi 5y ago

I've had more trouble with pip than any other package manager. Heck, one of my projects on GitHub currently has a broken CI pipeline because some newer version of pip broke deployments to pypi and I haven't investigated why yet. And plus, it's just super confusing. It's never been clear to me what the difference between pip and pip3 is. I thought pip3 was for python3 packages, but it seems like pip can also be used for python3 as well. But then sometimes it seems like if I use pip I only get the python2 version. With npm or go get, if I install a package I know exactly where it got installed a how to mnaually remove if needed. If I install something with pip, the newly installed package goes into some mysterious directory whose location is never quite certain. So yes, there are major problems with pip, hence why competitors are showing up like poetry.

psidex 5y ago
> It's never been clear to me what the difference between pip and pip3 is. I thought pip3 was for python3 packages, but it seems like pip can also be used for python3 as well. But then sometimes it seems like if I use pip I only get the python2 version. It's pretty simple; pip3 will always be for python3, pip2 will always be for python2. pip is context dependent, for example if you are in a python2 venv, it will use py2, and vice versa. > With npm or go get, if I install a package I know exactly where it got installed a how to mnaually remove if needed. If I install something with pip, the newly installed package goes into some mysterious directory whose location is never quite certain. If you pip install a library, it will almost definitely be installed in pythondir/lib/site-packages, unless you have messed around with your installation
bostonsre 5y ago
Think you might need to downgrade pip (using pip) to the version that is less strict when checking dependencies (i forget the exact version, think maybe 22). pip install "pip<22" To check which pip is for which python: pip --version Pyenv is pretty great for managing multiple python versions.
dragonwriter 5y ago
> It's never been clear to me what the difference between pip and pip3 is. pip3 is the name of the system installation of the pip executable corresponding to the python 3.x installation on linux systems (maybe Mac, too) when installed from system package managers on systems where python3 is the name of the python 3.x executable when installed from the package manager. This is not really a pip issue, but a “how python is packaged to support side-by-side system installations” issue.

protomikron 5y ago

Agreed. Similar to why Plan9 did not replace Unix, because Unix is good enough. It does not mean we can't do better, but the leap forward was (is) not big enough. At least that's my understanding.

smitty1e 5y ago
My theory on tech acceptance is that it's not good enough to be an improvement. To gain traction and acceptance, the offering must be several times better. Marketing also counts. Hence the lingering demise of python2.7: python3 didn't offer sufficient sizzle until ~3.6 or so.

golergka 5y ago

Npm is also good enough for most cases, but it hadn't stopped yarn from getting traction. May be Python developers are just more conservative?

nurettin 5y ago
They are simply uninterested. Poetry and pipenv both give you the opportunity to separate dev dependencies, keep your venv folder separate from the project, add platform specific packages and easily update all package versions. Pipenv will even load the .env file. After reading all this, your python developer will go "well I still don't know what it does that I can't do with venv/pip".
uranusjr 5y ago
Yarn got most of its popularity during a time when it was clearly ahead of npm in every way, but saw its share dropped when npm caught up. IMO Poetry came a bit too late; it would have been a tremendous hit in like 2015, but as the timeline be it is the leap forward is simply not enough. (Incidentally I think the same is happening for Yarn v2 as well; very good ideas, but too minor for most users to care.)