7 ms·
A not so unfortunate sharp edge in Pipenv
- rnestler 4y agoThere is an issue from 2019 on the pipenv tracker for this issue: https://github.com/pypa/pipenv/issues/3893 https://github.com/pypa/pipenv/issues/3893 According to the last comment there it should be fixed: > I don't think this should be an issue in the current 2022.4.8 release because of the requirement to use index restricted packages for locking. Please advise if you think this is still a concern and we can revisit it. But apparently it is still an issue (assuming the author did use a recent enough pipenv version) From the article: > Overall this is pipenv working exactly as it should. I don't agree, IMO pipenv should just install from the source package and ignore the newer wheel, maybe showing a warning about the new wheel. Edit: Just noticed that the author agrees in the end: > I suppose the behavior I’d like to see here is that if my Pipfile.lock has hashes in it for a distribution, then even if additional distributions are available, pipenv sync should be allowed to continue to install from the distribution corresponding to the “trusted” hash. I prefer to use https://python-poetry.org/ https://python-poetry.org/ instead of pipenv, since it has less issues in my experience. But I wonder how it would behave in this situation.
- aniforprez 4y agoI genuinely don't understand how someone can be a "proponent" of pipenv. In the mangled space of python package maintenance, pipenv is probably the absolute worst solution - pipenv install updates your lockfile on install exactly like how `npm install` updates the package-lock.json and requires additional flags to use the lockfile instead - pipenv's dependency resolution is the slowest among all the tools in this space including poetry, pip-tools, hatch and so on - pipenv has consistently failed to do its job to simply install dependencies because of issues like the one described in the article where it fails to resolve changes in hash simply because a new wheel was added for the platform - Tons of longstanding bugs that fail to be resolved in any sort of timely manner like this one: https://github.com/pypa/pipenv/issues/2413 https://github.com/pypa/pipenv/issues/2413 I am sick and tired of the python packaging space pushing this godawful tool for Python. Please just use pip-tools or poetry instead.
- 323 4y ago> exactly like how `npm install` updates the package-lock.json and requires additional flags to use the lockfile instead `npm install` doesn't do that anymore since at least a year
- Galanwe 4y agoNot to mention the horrible configuration anti-pattern where everything depends on environment variables. Environment variable tweaks are for local-only specific fixes, not for reproducible/distributable configuration.
- funklute 4y ago> not for reproducible/distributable configuration At my work, we use environment variables all the time for configurations, e.g. for telling a kubernetes pod where to find the database. In your opinion, what is the better option(s)?
- xvector 4y agoConfig files, and command line arguments as a very distant second
- BerislavLopac 4y agoSecret managers. All cloud providers have them built in, and on other platforms you can easily whip up a self-hosted one. See https://blog.diogomonica.com//2017/03/27/why-you-shouldnt-use-env-variables-for-secret-data/ https://blog.diogomonica.com//2017/03/27/why-you-shouldnt-us...
- xvector 4y agoI recommend against with using secrets managers as opposed to config files for storing configuration. Ideally, your configuration is versioned and code-reviewed. Config files are optimal. Even secrets used by your deployment should be stored in configuration - encrypted, of course. The benefit of versioned configuration files, secrets included, cannot be overstated. You can encrypt sensitive configuration fields with tools like `mozilla/sops` [1], which will reach out to your KMS or secrets manager of choice to encrypt/decrypt sensitive fields on the fly. This way, you are minimizing the splitting of state across secrets managers and your code. Your configuration is stored at code, code reviewed, and versioned, which has devops and security benefits of its own. [1]: https://github.com/mozilla/sops https://github.com/mozilla/sops
- kohlerm 4y agowell, another example that pythons dependency management is seriously broken. Why does the lock file not contain information about whether it was wheel or not?
- BerislavLopac 4y agoIt's not broken, it's incomplete. For example, there is no universally accepted lock file standard, so every tool whips up its own.
- BerislavLopac 4y agoMandatory read whenever the topic of pipenv comes back: https://chriswarrick.com/blog/2018/07/17/pipenv-promises-a-lot-delivers-very-little/ https://chriswarrick.com/blog/2018/07/17/pipenv-promises-a-l...
- danpalmer 4y agoPipenv was artificially elevated to its position of “popularity” by its maintainer who was able to use various connections to make it seem more official than it was. I think this has done a lot of harm to Python packaging over the years. Packaging wasn’t in a great place, but the confusion, and Pipenv’s promises for newcomers really threw everything off course for a generation of new Python engineers.
- wirrbel 4y agoI think this is part of it, it did not help that during the process it got abandoned and that poetry filled the void. I liked pipenv from its user interface it just never got from the PoC stage to „polished“. The way forward would actually be to unify the dependency resolution mechanisms into a PIP implementation (iirc pip has drastically improved its dependency resolution in recent years) and converge the various projects.
- nurettin 4y agoI've been experimenting with a new requirements.txt and requirements.txt.lock route. Pip allows your requirements file to omit versions. That means: you can manually add your root requirements into your requirements.txt. Then you can pip install -r requirements.txt Then you can do pip freeze > rewuirements.txt.lock commit that, and now everyone can use the prepared lock file. If you want to refresh your package versions, try pip install -U -r requirements.txt and it will update your dependencies. If anything breaks, you can uninstall it. If it all works out, you can lock again. This way I can stay within pip tools and speed up package installs. One negative is having to manually fix dependency conflicts. But that is a price I'd be willing to pay for ease if development and deployment. Edit: Also note that the article makes an excellent point concerning supply chain attacks which pipenv would have successfully mitigated in this particular case.
- Dunedan 4y agoCheck out pip-tools [1] which does exactly that, albeit in a slightly more polished way. [1]: https://github.com/jazzband/pip-tools https://github.com/jazzband/pip-tools
- toyg 4y agoPretty much all stuff that tools like pipenv do, is something that anyone proficient with pip can do on their own. Their "advantage" lies in standardizing, so people don't have to find out if you called the lockfile "requirements.txt.lock", "requirement.lock", or something else. Unfortunately they create another learning curve, another leaky abstraction layer, another point of failure, etc etc...
- nurettin 4y agoTo add to that, the suspicious steps are: manually adding root dependencies. What if you forget? Not locking. What if you forget? Locking invalid versions. What if you don't check all is compatible first? How do you handle dev requirements? requirements_dev? but then how do you lock? The good thing is: you got rid of dependency walking and slow locking. As with everything, there is a tradeoff that I was willing to take.
- baggiponte 4y agoI wholeheartedly recommend PDM as a modern python package management tool: https://pdm.fming.dev/latest/ https://pdm.fming.dev/latest/ * greater PEP compliance with respect to poetry (and no upper boundary constraints on packages) * support for scripting * parallel installation Here is the opinion of somebody way more qualified than me: - https://iscinumpy.dev/post/poetry-versions/ https://iscinumpy.dev/post/poetry-versions/ - https://iscinumpy.dev/post/bound-version-constraints/ https://iscinumpy.dev/post/bound-version-constraints/
- angus_gh 4y agoSidenote: this website performs poorly on Firefox for Android; every time the background updates, the page freezes for a second, so scrolling is jumpy
- senko 4y agoI use a small shell script I wrote in jest a year ago (https://github.com/senko/pvp https://github.com/senko/pvp), it boils down to virtualenv and pip, and it basically solves all Python package management issues for me. I was burned by pipenv before (naively trying to use it because a certain prominent member of Python community hyped it up, and I hadn't known he'd gone off the rails). I can find no redeeming qualities to it whatsoever, and it scarred me enough that I don't want to touch other newer tools (poetry, pdm, whatever) with a 10ft pole. I find solace in pip and virtualenv just working and not trying to be too clever. I pin all my immediate reqs, and if there's a conflict with indirect deps (which happened maybe once or twice), I figure out the version I need and pin that manually. For packaging I rely on specialized tools that do just that (build, twine), and don't need one-with-everything spaceship of a tool. I also develop in Node and have suffered much more grief by npm. I lost count of the number of times where I had to rm-rf node modules and npm cache, rerun and hope for the best.
- OkayPhysicist 4y agoOrthogonal to the Python-specific issues, the problem of "All of my past commits in version control are now broken" is an interesting one. My first instinct would be to rebase the entire project tree onto a new commit that follows the original addition of the dependency. This isn't a great solution, mostly because git doesn't make such an operation easy, but also because there's something a little spooky about a mass history rewrite like that.
- fb03 4y agoWhat do I use? - Pipenv? - Nice deterministic builds - Dogslow to lock - Has some quirky defaults (pipenv install also updates other packages) - Poetry? - Pretty - 10x locking speed - Heard (in several places) problems about it failing to deterministically resolve a dep tree in some edge cases, which makes me a little scared (am I going to hit the dep jackpot at some point?) - PDM? - Hatch? - Bawl in a corner? Thanks in advance.
- sc68cal 4y agoas other commenters have stated, pip-tools is a decent alternative. It does have some issues around platform dependent packages (like libuv and windows), but that's solved by managing a different compiled requirements per environment.
- AndyKluger 4y agoIf you or anyone else enjoying pip-tools is a Zsh user and interested in trying out my higher level functions to ease interactive use of pip-tools, venvs, and also isolated app installs (like pipx), I would love some feedback on zpy: https://github.com/AndydeCleyre/zpy https://github.com/AndydeCleyre/zpy I'm very happy to answer any questions about it right here or as GitHub issues.