6 ms·

Pipenv: Promises a Lot, Delivers Little (2020)

65 points by BerislavLopac 4y ago

greut 4y ago
It's a bit of an old post, and yes Pipenv is not the go-to tool anymore. pip-tools is okay for people that really, really love their requirements.txt; otherwise we tend to go with Poetry at work. Any folks having a good experience with PDM https://github.com/pdm-project/pdm https://github.com/pdm-project/pdm ?
- strbean 4y ago
  How is the maturity/stability of Poetry these days? I despise Pipenv, and was hoping to push for a switch to Poetry at my place of work a couple years ago, but I ran into blocking bugs across multiple versions (latest N versions affected by bug A, prior M versions affected by bug B). Had to chock it up to "not yet mature enough" and resign myself to the absurd lock times and countless terrible behaviors of Pipenv.
  - aden1ne 4y ago
    We use poetry for all python projects. I haven't seen an actual poetry internal bug in quite a while, but using poetry effectively does require one keep some things in mind that are probably non-obvious to newcomers: 1. Poetry's default assumption on packages respecting semver simply does not hold up in reality. There are very few packages actually sticking to semver. Thus the `^x.y.z` default version range is quite often too loose. I've found that using `~x.y.z` for most packages is far more stable. 2. Imho, `poetry update` is a footgun. Without specifier, it will attempt to update the entire dependency tree. Not only is this slow, together with 1) it's all too likely one ends up with dependencies that actually are incompatible at runtime. I'd much rather have a `poetry update --all` flag instead for the rare instance I do want to update everything. The default behaviour should be to require a list of packages to update. 3. There are some common packages that cause very long resolution times if they are not restricted. Case in point: boto3. Even if one doesn't use boto3 oneself, it's very likely a transient dependency. Many packages simply specify `'*'` as their version dependency (they shouldn't, but it's the unfortunate reality many do). This will cause poetry to consider every possible boto3 version. With hundreds of versions - boto3 has a release every other day - this gets unwieldy fast. So I often end up specifying boto3 myself with some sensible range in my toml file, even when it's not a strict dependency of my own project. 4. The datascience ecosystem needs particular attention. Best to simply pin those, as every pandas update is guaranteed to break something. ABI changes to numpy are a particular nightmare. This is again due to too many packages simply specifying `'*'` for their numpy dependency. Which is further complicated by the fact that most don't distinguish between build-time dependencies and run-time dependencies. The numpy ABI is only forward compatible. Hence one should build with the oldest supported numpy[0]. [0]: https://pypi.org/project/oldest-supported-numpy/ https://pypi.org/project/oldest-supported-numpy/
    - strbean 4y ago
      I'll take the behavior of 2 over Pipenv's "any re-lock aggressively updates everything" approach. What's the point of lock files if you have to peg everything in order to have stability / control over versions??
    - 41b696ef1113 4y ago
      Updating dependencies (#2) does seem needlessly painful. I have wondered if I am missing some obvious workflow.
  - bamboozled 4y ago
    Seems very good, except they do seem to have some long running per-releases going which seems troubling. Just ship it already!
- gooseyard 4y ago
  After a series of bad experiences with Poetry, I switched the packages I maintain to PDM. Although I have hit a few minor snags, the maintainer and other users in the project's github discussions have never failed to help with a fix, workaround, or advice. It's pleasant to use, and I think the only feature I cared about which was in Poetry but not PDM (a publish-to-pypi command), has proven easier to do with the Twine tool anyway. My situation is I think unusual, in that I need to use a private pypi repo which requires mTLS for both fetching and publishing. Had it not been for that I suspect I'd still be using Poetry, but given the experiences I've had with PDM I wouldn't switch back even if the situation with my repo changed.
  - bamboozled 4y ago
    See my above comment, something to be aware of.
- bamboozled 4y ago
  PDM is based on PEP-582, which is only a draft and you will hit edge cases where it's not supported, or projects refuse to support it because of this. I'd avoid it for that reason personally. virtualenvs are a much more supported.
  - radus 4y ago
    You can opt out of PEP-582, and in PDM 2.0 (just released), it becomes opt in.
    - bamboozled 4y ago
      What’s the point of using it over out Poetry then ? Why do we need a other ?
number6 4y ago
My journey went from pipenv to poetry to pipenv back to pip and then to pip-tools (pip-compile and pip-sync). While both pipenv and poetry were nice and had a lot of comfort build in they broke in nasty ways and it was hard to debug so I traded the comfort for less complexity.
- brabel 4y ago
  At work, we have large projects that use lots of technologies, including for example Java, Erlang, Rust, Kotlin, Groovy, even a little bit of C code, which tends to be problematic... and Python. Our developer environment is well polished and can be started up with installing a couple of packages then running a single command... very nice, except for the Python part , which is extremely tiny compared to the other stacks. We used pipenv, but that broke very often, so we moved to Poetry... it looked like it worked for a while, but as we got more people, specially ones using Linux and Mac M1 had to spend lots of time fixing issues with Poetry/Python... something to do with cpython dependencies downloaded for the wrong architecture. We've had zero, absolutely zero issues with all the other stacks. We're trying to find solutions, but IMHO we'll just have to bite the bullet and re-write the small Python component we have into any of the other many languages we use without any issues.
UncleEntity 4y ago
If you don’t like a tool then don’t use the tool. Or fix what you perceive is wrong with it. Don’t demand the tool do what you want or insist on a certain release cadence unless the developers are on your payroll. I admittedly started skimming around halfway through but still want a refund on the time I wasted trying to determine if there was more to TFA than a whinefest.
drekipus 4y ago
I'm really sorry, but I can never quite understand what the deal is with these tools are. I understand that virtual environments are important, but how are these tools any better than python3 -m venv .venv --prompt="foobar" . .venv/bin/activate pip install -r requirements.txt ? Whenever I see all these other tools I just get the feeling like there's some big elephant in the room that everyone is battling against, but I've never come across it as a python dev, and the moment I try to user/understand these tools I feel like they're against the "keep it simple, stupid" vibe that python gives me. There's auto-venv for the truly lazy (like me) but other than that I don't rely on any of these Env/requirement wrappers or tools in any of my projects (virtualenv-wrapper for work but that was in their setup guide). Is it a legacy thing?
- throwaway234232 4y ago
  You don't need virtual environments when using npm or composer or cargo. They also create lock files with hashes by default. pip still doesn't use lock files. It's subpar compared to other ecosystems.
  - traverseda 4y ago
    I worry that the inclusion of lockfiles will make semver less relevant. You shouldn't need to use lock files if you're using semver properly. If your ecosystem is healthy than pinning exact versions with lock files shouldn't be done. Making it so that every dev uses the latest patch or minor when they run your program. Using lock files for libraries should absolutely never happen, you should at most fix your dependencies to the latest major version.
    - throwaway234232 4y ago
      I agree, but they also include hashes.
    - mirashii 4y ago
      Semver promises are insufficient in a world where supply chain attacks are increasingly common. Pulling untested and invalidated code in at every project build is how that transitive dependency on a package that was taken over for a small window wrecks your development team. You should never be pulling in new code by surprise, it should always be something I’m aware of and signing up for. The Rust ecosystem is good evidence that locking doesn’t kill semver. Semver is still widely used and has all of its meaning.
wodenokoto 4y ago
Is there a reliable source that tells what the current recommended way of doing this in Python is? I’ve been using Anaconda and been finding that it install incompatible versions of jupyter and ipython dependencies and that a lot of tools I need only work from pip, so I’ve been wondering if maybe I should just switch to the “pythonic way of doing it” and have absolutely no idea what that is.
- p4ul 4y ago
  I have seen the Hypermodern Python series of blog posts referenced here a few times. I'm a big fan of that approach personally. I'd also be curious what others recommend! https://cjolowicz.github.io/posts/hypermodern-python-01-setup/ https://cjolowicz.github.io/posts/hypermodern-python-01-setu...
- gh02t 4y ago
  Slight digression, but I'm always baffled by how unbelievably slow Conda is at resolving dependencies as soon as you have more than a handful of packages installed. Mamba basically proves it's a solvable issue, but you'd think the Anaconda folks would be prioritizing it.
- bb88 4y ago
  Today, we (and I mean me and my company) use poetry for prod code delivery and it just works. Poetry uses pip and virtualenv under the hood. It's worth understanding virtualenv regardless. For every project you're developing, there will be virtualenv which has all the dependencies that project needs (which may be different than what's installed in the system, and different than what other projects may need). "python -m venv init project/venv" will create it. "rm -rf project/venv" will delete it. Usually the virtualenv goes somewhere well known, like "project/venv". Sourcing the activate script ( "source project/venv/bin/activate" ) changes your shell environment to use the virtualenv instead of the system python environment. Once activated "pip install package" installs to the activated virtualenv. "deactivate" will turn off the virtualenv. This is semantic sugar for what's really happening behind the curtains. There's a copy of python in the virtualenv: "project/venv/bin/python" which runs in the virtual env regardless of whether the virtualenv is activated or not. "activate" just adds "project/venv/bin" to the start of the PATH. "deactivate" removes it. Regardless you can always see which python you're using by typing "which python". System python (/usr/bin/python) will use system packages. The venv python (project/venv/bin/python) will use the virtualenv python packages. This allows you to have different virtualenvs to try out things like new versions of python, say. And each virtualenv is isolated from every other virtualenv. And poetry is just a nice wrapper around this process that also figures out total project dependencies and creates a "lock" file to freeze all deps to a particular version for consistent releases. Pipenv is basically the same thing. "pip install --user poetry" will install it in your home directory. It's not recommended to use conda and pip together, basically the recommendation is to use one or the other, though I've heard miniconda is better in this regard. YMMV.
bb88 4y ago
This is in 2020 and should be marked as such.
donarb 4y ago
I've started to get away from virtual environments and just use Docker containers where all the pip dependencies can live as first class citizens. Also eliminates having to source the activate file or wonder if I'm calling the correct version of Python when I want to ssh in and run some code by hand in the container.
- akvadrako 4y ago
  Containers are basically language agnostic virtual environments done right. Having python or js specific solutions is really a hack.
js2 4y ago
(May 2020) so this piece is out of date. I use pipenv on one of my projects, including (in combination with direnv) for setting up my local dev environment on macOS and as part of CI/CD for deploying into a Docker container based on one of the Python images. The project doesn't have a huge number of dependencies, but pipenv has worked well. The only time I fought with it was over its PyUp safety checks, but "pipenv check" is opt-in anyway. For most of my other projects, I install from either a setup.py or requirements file using pip, often driven by a Makefile. Again, deploying into Linux, but using macOS for local development, again with direnv. In 20 years of professional Python development on mostly small-to-medium size projects, I've never had any issues I can recall getting packages installed. Conversely, I semi-regularly fight with cocoapods, npm and gems.
deleted 4y ago
[deleted]
bastawhiz 4y ago
Lots of folks are saying pipenv is outdated (and I agree), but it is still listed on the packaging docs [0] with little warning, and as the second tool on the official list of recommended tools [1]. If the community wants to deprecate this extremely undermaintained tool, we need to stop pointing people at it. 0: https://packaging.python.org/en/latest/key_projects/ https://packaging.python.org/en/latest/key_projects/ 1: https://packaging.python.org/en/latest/guides/tool-recommendations/ https://packaging.python.org/en/latest/guides/tool-recommend...
bit-rot 4y ago
What is the currently accepted way of nailing down a specific python version if not using something like pipenv?
alanwreath 4y ago
Seems like instead of focusing on python's execution speed, we've all been saying (for a long time) what we really really need in a language (like python) is simplified/centralized/pythonic(one way to do it) tooling. Python, isn't popular because it executes close to C speeds. It's popular because it generally removes common obstacles for the developer. The package manager for python is a common obstacle that many open source repo's are trying to solve because python proper hasn't solved it (or attempted to with enough focus).

65 points by BerislavLopac 4y ago

It's a bit of an old post, and yes Pipenv is not the go-to tool anymore. pip-tools is okay for people that really, really love their requirements.txt; otherwise we tend to go with Poetry at work. Any folks having a good experience with PDM https://github.com/pdm-project/pdm https://github.com/pdm-project/pdm ?

strbean 4y ago
How is the maturity/stability of Poetry these days? I despise Pipenv, and was hoping to push for a switch to Poetry at my place of work a couple years ago, but I ran into blocking bugs across multiple versions (latest N versions affected by bug A, prior M versions affected by bug B). Had to chock it up to "not yet mature enough" and resign myself to the absurd lock times and countless terrible behaviors of Pipenv.
- aden1ne 4y ago
  We use poetry for all python projects. I haven't seen an actual poetry internal bug in quite a while, but using poetry effectively does require one keep some things in mind that are probably non-obvious to newcomers: 1. Poetry's default assumption on packages respecting semver simply does not hold up in reality. There are very few packages actually sticking to semver. Thus the `^x.y.z` default version range is quite often too loose. I've found that using `~x.y.z` for most packages is far more stable. 2. Imho, `poetry update` is a footgun. Without specifier, it will attempt to update the entire dependency tree. Not only is this slow, together with 1) it's all too likely one ends up with dependencies that actually are incompatible at runtime. I'd much rather have a `poetry update --all` flag instead for the rare instance I do want to update everything. The default behaviour should be to require a list of packages to update. 3. There are some common packages that cause very long resolution times if they are not restricted. Case in point: boto3. Even if one doesn't use boto3 oneself, it's very likely a transient dependency. Many packages simply specify `'*'` as their version dependency (they shouldn't, but it's the unfortunate reality many do). This will cause poetry to consider every possible boto3 version. With hundreds of versions - boto3 has a release every other day - this gets unwieldy fast. So I often end up specifying boto3 myself with some sensible range in my toml file, even when it's not a strict dependency of my own project. 4. The datascience ecosystem needs particular attention. Best to simply pin those, as every pandas update is guaranteed to break something. ABI changes to numpy are a particular nightmare. This is again due to too many packages simply specifying `'*'` for their numpy dependency. Which is further complicated by the fact that most don't distinguish between build-time dependencies and run-time dependencies. The numpy ABI is only forward compatible. Hence one should build with the oldest supported numpy[0]. [0]: https://pypi.org/project/oldest-supported-numpy/ https://pypi.org/project/oldest-supported-numpy/
  - strbean 4y ago
    I'll take the behavior of 2 over Pipenv's "any re-lock aggressively updates everything" approach. What's the point of lock files if you have to peg everything in order to have stability / control over versions??
  - 41b696ef1113 4y ago
    Updating dependencies (#2) does seem needlessly painful. I have wondered if I am missing some obvious workflow.
- bamboozled 4y ago
  Seems very good, except they do seem to have some long running per-releases going which seems troubling. Just ship it already!
gooseyard 4y ago
After a series of bad experiences with Poetry, I switched the packages I maintain to PDM. Although I have hit a few minor snags, the maintainer and other users in the project's github discussions have never failed to help with a fix, workaround, or advice. It's pleasant to use, and I think the only feature I cared about which was in Poetry but not PDM (a publish-to-pypi command), has proven easier to do with the Twine tool anyway. My situation is I think unusual, in that I need to use a private pypi repo which requires mTLS for both fetching and publishing. Had it not been for that I suspect I'd still be using Poetry, but given the experiences I've had with PDM I wouldn't switch back even if the situation with my repo changed.
- bamboozled 4y ago
  See my above comment, something to be aware of.
bamboozled 4y ago
PDM is based on PEP-582, which is only a draft and you will hit edge cases where it's not supported, or projects refuse to support it because of this. I'd avoid it for that reason personally. virtualenvs are a much more supported.
- radus 4y ago
  You can opt out of PEP-582, and in PDM 2.0 (just released), it becomes opt in.
  - bamboozled 4y ago
    What’s the point of using it over out Poetry then ? Why do we need a other ?

number6 4y ago

My journey went from pipenv to poetry to pipenv back to pip and then to pip-tools (pip-compile and pip-sync). While both pipenv and poetry were nice and had a lot of comfort build in they broke in nasty ways and it was hard to debug so I traded the comfort for less complexity.

brabel 4y ago
At work, we have large projects that use lots of technologies, including for example Java, Erlang, Rust, Kotlin, Groovy, even a little bit of C code, which tends to be problematic... and Python. Our developer environment is well polished and can be started up with installing a couple of packages then running a single command... very nice, except for the Python part , which is extremely tiny compared to the other stacks. We used pipenv, but that broke very often, so we moved to Poetry... it looked like it worked for a while, but as we got more people, specially ones using Linux and Mac M1 had to spend lots of time fixing issues with Poetry/Python... something to do with cpython dependencies downloaded for the wrong architecture. We've had zero, absolutely zero issues with all the other stacks. We're trying to find solutions, but IMHO we'll just have to bite the bullet and re-write the small Python component we have into any of the other many languages we use without any issues.

UncleEntity 4y ago

If you don’t like a tool then don’t use the tool. Or fix what you perceive is wrong with it. Don’t demand the tool do what you want or insist on a certain release cadence unless the developers are on your payroll. I admittedly started skimming around halfway through but still want a refund on the time I wasted trying to determine if there was more to TFA than a whinefest.

drekipus 4y ago

I'm really sorry, but I can never quite understand what the deal is with these tools are. I understand that virtual environments are important, but how are these tools any better than python3 -m venv .venv --prompt="foobar" . .venv/bin/activate pip install -r requirements.txt ? Whenever I see all these other tools I just get the feeling like there's some big elephant in the room that everyone is battling against, but I've never come across it as a python dev, and the moment I try to user/understand these tools I feel like they're against the "keep it simple, stupid" vibe that python gives me. There's auto-venv for the truly lazy (like me) but other than that I don't rely on any of these Env/requirement wrappers or tools in any of my projects (virtualenv-wrapper for work but that was in their setup guide). Is it a legacy thing?

throwaway234232 4y ago
You don't need virtual environments when using npm or composer or cargo. They also create lock files with hashes by default. pip still doesn't use lock files. It's subpar compared to other ecosystems.
- traverseda 4y ago
  I worry that the inclusion of lockfiles will make semver less relevant. You shouldn't need to use lock files if you're using semver properly. If your ecosystem is healthy than pinning exact versions with lock files shouldn't be done. Making it so that every dev uses the latest patch or minor when they run your program. Using lock files for libraries should absolutely never happen, you should at most fix your dependencies to the latest major version.
  - throwaway234232 4y ago
    I agree, but they also include hashes.
  - mirashii 4y ago
    Semver promises are insufficient in a world where supply chain attacks are increasingly common. Pulling untested and invalidated code in at every project build is how that transitive dependency on a package that was taken over for a small window wrecks your development team. You should never be pulling in new code by surprise, it should always be something I’m aware of and signing up for. The Rust ecosystem is good evidence that locking doesn’t kill semver. Semver is still widely used and has all of its meaning.

wodenokoto 4y ago

Is there a reliable source that tells what the current recommended way of doing this in Python is? I’ve been using Anaconda and been finding that it install incompatible versions of jupyter and ipython dependencies and that a lot of tools I need only work from pip, so I’ve been wondering if maybe I should just switch to the “pythonic way of doing it” and have absolutely no idea what that is.

p4ul 4y ago
I have seen the Hypermodern Python series of blog posts referenced here a few times. I'm a big fan of that approach personally. I'd also be curious what others recommend! https://cjolowicz.github.io/posts/hypermodern-python-01-setup/ https://cjolowicz.github.io/posts/hypermodern-python-01-setu...
gh02t 4y ago
Slight digression, but I'm always baffled by how unbelievably slow Conda is at resolving dependencies as soon as you have more than a handful of packages installed. Mamba basically proves it's a solvable issue, but you'd think the Anaconda folks would be prioritizing it.
bb88 4y ago
Today, we (and I mean me and my company) use poetry for prod code delivery and it just works. Poetry uses pip and virtualenv under the hood. It's worth understanding virtualenv regardless. For every project you're developing, there will be virtualenv which has all the dependencies that project needs (which may be different than what's installed in the system, and different than what other projects may need). "python -m venv init project/venv" will create it. "rm -rf project/venv" will delete it. Usually the virtualenv goes somewhere well known, like "project/venv". Sourcing the activate script ( "source project/venv/bin/activate" ) changes your shell environment to use the virtualenv instead of the system python environment. Once activated "pip install package" installs to the activated virtualenv. "deactivate" will turn off the virtualenv. This is semantic sugar for what's really happening behind the curtains. There's a copy of python in the virtualenv: "project/venv/bin/python" which runs in the virtual env regardless of whether the virtualenv is activated or not. "activate" just adds "project/venv/bin" to the start of the PATH. "deactivate" removes it. Regardless you can always see which python you're using by typing "which python". System python (/usr/bin/python) will use system packages. The venv python (project/venv/bin/python) will use the virtualenv python packages. This allows you to have different virtualenvs to try out things like new versions of python, say. And each virtualenv is isolated from every other virtualenv. And poetry is just a nice wrapper around this process that also figures out total project dependencies and creates a "lock" file to freeze all deps to a particular version for consistent releases. Pipenv is basically the same thing. "pip install --user poetry" will install it in your home directory. It's not recommended to use conda and pip together, basically the recommendation is to use one or the other, though I've heard miniconda is better in this regard. YMMV.

bb88 4y ago

This is in 2020 and should be marked as such.

donarb 4y ago

I've started to get away from virtual environments and just use Docker containers where all the pip dependencies can live as first class citizens. Also eliminates having to source the activate file or wonder if I'm calling the correct version of Python when I want to ssh in and run some code by hand in the container.

akvadrako 4y ago
Containers are basically language agnostic virtual environments done right. Having python or js specific solutions is really a hack.

js2 4y ago

(May 2020) so this piece is out of date. I use pipenv on one of my projects, including (in combination with direnv) for setting up my local dev environment on macOS and as part of CI/CD for deploying into a Docker container based on one of the Python images. The project doesn't have a huge number of dependencies, but pipenv has worked well. The only time I fought with it was over its PyUp safety checks, but "pipenv check" is opt-in anyway. For most of my other projects, I install from either a setup.py or requirements file using pip, often driven by a Makefile. Again, deploying into Linux, but using macOS for local development, again with direnv. In 20 years of professional Python development on mostly small-to-medium size projects, I've never had any issues I can recall getting packages installed. Conversely, I semi-regularly fight with cocoapods, npm and gems.

deleted 4y ago

[deleted]

bastawhiz 4y ago

Lots of folks are saying pipenv is outdated (and I agree), but it is still listed on the packaging docs [0] with little warning, and as the second tool on the official list of recommended tools [1]. If the community wants to deprecate this extremely undermaintained tool, we need to stop pointing people at it. 0: https://packaging.python.org/en/latest/key_projects/ https://packaging.python.org/en/latest/key_projects/ 1: https://packaging.python.org/en/latest/guides/tool-recommendations/ https://packaging.python.org/en/latest/guides/tool-recommend...

bit-rot 4y ago

What is the currently accepted way of nailing down a specific python version if not using something like pipenv?

alanwreath 4y ago

Seems like instead of focusing on python's execution speed, we've all been saying (for a long time) what we really really need in a language (like python) is simplified/centralized/pythonic(one way to do it) tooling. Python, isn't popular because it executes close to C speeds. It's popular because it generally removes common obstacles for the developer. The package manager for python is a common obstacle that many open source repo's are trying to solve because python proper hasn't solved it (or attempted to with enough focus).