33 ms·
Pipfile for Python
- nubela 10y agoI don't understand the need for this. The main benefit of the Pipfile seem minute, and can already be achieved with requirements.txt. But I'm sure I'm missing something. Please feel free to convince me I'm wrong :)
- stickperson 10y agoGrouping of sub-dependency groups (e.g. a testing group). That seems like an improvement over having multiple requirement files. Then again, I'm perfectly happy with separate dev/testing files.
- davvid 10y agoOne use case I don't see covered is grouping by python versions. Sometimes "soft" dependencies are only available/supported/not-broken on certain python versions. It'd be nice to be able to mark a dependency as being conditional based on some expression, e.g. `$PYTHON_VERSION_MAJOR >= 3`.
- examancer 10y agoGive it time. Moving to a native python DSL will make extending the dependency system more straightforward. Also, if they continue copying the best of Ruby's bundler they will have a way to specify python version/platform specific dependencies.
- witten 10y agoBTW, you can do that already with requirements.txt. See https://stackoverflow.com/questions/19559247/requirements-txt-depending-on-python-version#33451105 https://stackoverflow.com/questions/19559247/requirements-tx...
- deleted 10y ago[deleted]
- civilian 10y agoI'd like something like `npm install --save Django` -- a command that finds the most recent stable version of Django, and then adds _that_ package to requirements.txt, and leaves Django's dependencies out of reqs.txt. I know that it's just a few simple operations for me to do it myself, but npm has spoiled me.
- deleted 10y ago[deleted]
- INTPenis 10y agoIf I'm reading you correctly, the missing feature in pip is to list dependencies for one package so they can be put into requirements.txt. This is a missing feature, there are various hacks but nothing concrete. If you're working on a large project you have to keep track of all your dependencies and preferably only add the main dependency, not its dependencies, into requirements. I tend to avoid doing pip freeze > requirements.txt because it lists packages I know shouldn't be in requirements.
- morgante 10y agoI think you misunderstood. Like civilian, my main ask for an improved pip would be a way to save a single explicit package into my requirements.txt by adding a --save flag when installing it. I don't want to include sub-dependencies in my requirements.txt.
- INTPenis 10y agoI believe I follow. The way to maintain requirements.txt with single dependencies (base dependencies maybe, I'm self-taught on the lingo), is to manually add them one by one. Pip freeze is useless because it adds a lot of things you don't need into requirements, making it cluttered and hard to follow.
- morgante 10y agoYes, I (and civilian) know that. The point is that it's comparatively cumbersome. You have to manually add the specific version into requirements.txt instead of being able to automatically save it as you install.
- examancer 10y agoThe main reason this is better is explained in the third paragraph in the README: Deterministic builds. You can specify target versions in your Pipfile, while your Pipfile.lock will contain the actual exact versions pip installed (even the exact git commit) so if your app is built on another machine you know the libraries are exactly the same, preventing slight version differences from causing you problems without requiring you to be ultra-specific in your dependencies. Another benefit is named groups, which allow you to more succinctly specify dependencies for various environments (dev/test/production/etc.) Along with this you get the benefits of the lock file so you can be assured the subset of libraries you use in production will be the exact tested libraries you use in your larger dev/test environment.
- witten 10y agoHow is the Pipfile.lock distributed? Is it intended to be checked into source control alongside the Pipfile? If so, how would that help someone pip install my Python project (say from pypi) using that Pipfile.lock and get the benefits of a deterministic build?
- examancer 10y agoYes, source control. Installing the project would cause it to be built from the lock file (assuming the Pipefile hasn't changed). This will mean your users won't get "some version" of a library you depend on between verion 1.0 and 2.0 or whatever range you specified. They'll get the exact package you last successfully used and checked in yourself, right down to the git commit if applicable. Once you modify the Pipfile then pip will resolve your dependencies and try to make the specified changes by adding/removing/upgrading packages. If Pypa continues following bundler conventions this will be done by making the fewest changes possible from your existing versions in your lock file. You'll also have an upgrade mode where pip will rebuild you project from the Pipfile looking for the most recent versions of all libraries or a specified library within your specified version ranges. When done and your app/tests are working, check in the new version and you can ensure all your users/environments will be able to upgrade cleanly.
- Groxx 10y ago
- misterbowfinger 10y agoAbout time. The Ruby, Elixir, and Rust communities are far, far along in their package management tools. Working with pip feels like going back in time these days.
- techman9 10y agoEven npm (despite its many flaws) is in far better shape than pip.
- efrafa 10y agoI might be the only one, but I think npm is even better than bundler.
- morgante 10y agoI'm with you there too. npm is by far the best package manager I've ever used.
- LukaD 10y agoFor me npm has always been the worst dependency manager I've used. What bugs me the the most is that it installs all packages to node_modules by default. It is possible to specify another location but then your application will probably break because it has to know where your node_modules directory is. Then there's the whole non-determinism thing: https://docs.npmjs.com/how-npm-works/npm3-nondet https://docs.npmjs.com/how-npm-works/npm3-nondet Also check out https://github.com/npm/npm/issues/10999 https://github.com/npm/npm/issues/10999 This is just insane!
- Groxx 10y agoPip has the same kinds of issues with transitive dependencies, but way way way worse since it doesn't allow multiple versions. When you have lib A requiring C ("A -> C1") and "B -> C2", the version of C you get depends on which of A or B was installed first, because the second's requirements are flat-out ignored. Probably like 90% of the python projects I've looked at have requirement-conflicts because pip doesn't even warn you about this.
- nicois 10y agothis is what I have used for the past few years. I put my unversioned requirements in a subdir and run this when I want to bump versions. #!/bin/bash -x WHEELHOUSE="/usr/local/wheelhouse" [ -d "$WHEELHOUSE" ] || ( sudo mkdir -p /usr/local/wheelhouse/ ; sudo chmod -R 0777 /usr/local/wheelhouse/ ) deactivate set -e cd .requirements for reqfile in requirements*txt ; do TEMPDIR="$(mktemp -d)" virtualenv -ppython3 "$TEMPDIR" . "$TEMPDIR"/bin/activate pip install -U pip pip install -U wheel pip wheel --find-links="$WHEELHOUSE" --wheel-dir="$WHEELHOUSE" -r $reqfile pip install --find-links="$WHEELHOUSE" -r $reqfile pip freeze | grep -v "pkg-resources" | sort > "../$reqfile" rm -rf "$TEMPDIR" done wait
- breerly 10y agoWth happened to pip-tools? The Python ecosystem is a mess...
- dimino 10y agoVery happy about this, the Pypa crew are saving Python from itself.
- examancer 10y agoThis is almost identical to how bundler in Ruby works, right down to the language native dependency DSL, named groups, file name conventions (Pipfile = Gemfile, Pipfile.lock = Gemfile.lock), and deterministic builds. It's identical because bundler mostly got it right and dependency management in Ruby, while still not great/perfet, is better than just about everywhere else. Kudos to python for moving forward.
- INTPenis 10y agoWow I'm amazed at how skewed my world view was. The first two comments I read are praising ruby dependency management and scorning pythons requirements. I'm sitting here thinking, "what the hell is wrong?". I've honestly only experienced trouble with Ruby while being completely satisified with how python virtualenv works. I guess if anything this proves that it's about habit. Habitual use of something makes it the easiest product for the habitual user. Ruby is something I force myself through when I want to try a product while Python is something I develop my own products in.
- examancer 10y agoI used to think PHP was amazing and build tools were weird and unnecessary when I built things mostly in PHP. It's hard to see the point of many tools if you aren't working with the all the time. I think this is the product of people who got to know both Python and Ruby very well and found Python lacking here. There are lots of things Ruby developers were gifted from people who also know Python and found Ruby lacking. Python is generally something I force myself through so I'm not one of those people but so glad they exist. BTW, Ruby has tools similar to virtualenv: chruby, rbenv, and rvm all do basically the same thing.
- mypalmike 10y agoMy experience has been quite the opposite wrt ruby and python. I spent 2 years with Ruby as my primary language, and the regularity in which I would end up with a subtly (or drastically) broken Ruby environment was astounding. With python, I've rarely if ever run into such problems.
- witten 10y agoThis seems like a huge step backwards to me. Why would you want to go from a parse-able, machine-readable, data-driven syntax to something that can't easily be introspected, isn't machine-readable without firing up a full Python interpreter, is as flexible as actual code and is thus subject to all the abuse you can introduce with actual code, etc.. From the original prototype's comments: https://gist.github.com/dstufft/2904d2e663461f010bbf https://gist.github.com/dstufft/2904d2e663461f010bbf "- If there's a corner case not thought of, file is still Python and allows people to easily extend" ... and ... "- Using Python file might cause the same problems as setup.py" Ugh. Also, how is the "lock file" actually distributed? Unless you can pip install a wheel and have it include an embedded lock file, then you've still got to have some out-of-band mechanism for copying the lock file around like you would a requirements.txt or even a fully-fledged virtual environment.
- examancer 10y agoYour lock file goes into version control. This ensures all your developers and environments are using the exact same version of every library. You generate a new lock file when upgrading/adding libraries. Don't be so scared. Ruby developers have been doing this for over 7 years now. Trust us, this is much better. Your Python friends at Pypa are copying Ruby because dependencies are less painful there than just about anywhere. They are still painful, but this will help.
- witten 10y agoI'm not sure age is a good indication of goodness in this regard. Python has been doing setup.py for much longer than 7 years. It's a similar format, and ripe for abuse because you have full access to a general purpose programming language in it. I've seen setup.pys call out to Inkscape. I don't want a general purpose programming language in my requirements.txt (or equivalent). I want a dumb declarative data file. Also, I'm not sure how a lock file in source control will help me when someone who is not checking out source wants to pip install my project using said lock file.
- examancer 10y ago
- Animats 10y agoIf you need old-version retrieval on a routine basis, library regression testing has a problem. This gives package developers an excuse to break backwards compatibility. Historically, Python has avoided that, except for the Python 3 debacle.
- civilian 10y agoSorry, maybe it's late, but I don't follow. Can you elaborate?
- chaosagent 10y agoThey're probably claiming that using a lockfile excessively would put less incentive on library developers to keep backwards compatibility, because nothing would break if the user doesn't explicitly upgrade.
- Groxx 10y agoI'm a bit confused as well, unless you're referring to "installing libs at any version except HEAD", in which case I vehemently disagree with you. Deterministic builds are a must-have for production systems.
- icebraining 10y agoThe question is whether that should be included in the project, or as a configuration of the deployment tool.
- yxhuvud 10y agoThere is no question. If it isn't included in the project, then any errors you get are not repeatable.
- icebraining 10y agoWhy can't you get the versions in question from the origin of the error (deployment config if it's an internal error, or asking the user if it's an external report)?
- holografix 10y agoCan't we just leave as it is? I love the fact that requirements.txt is stupid simple. No over engineered json shenanigans. Wanna group things into prod, dev etc? Create 2 files.
- joss82 10y agoThis. Requirements.txt is very pythonic in the sense that it's as simple as possible, but no simpler.
- deleted 10y ago[deleted]
- urda 10y agoAgreed, I'll take a pass on this project and stick with the accepted standards which are exactly that. Accepted standards.
- djm_ 10y agopypa, the Python Packaging Authority, is a working group that maintains many of the Python packaging projects (e.g pip) and therefore pipfile in time will likely be the new accepted standard.
- fphilipe 10y agoThis is great news. Coming from Ruby and being used to Bundler, doing anything in Python or JS always was a huge pain. Countless times I deleted the current virtual environment or did an `rm -rf node_modules` to start fresh. So I'm excited to see Yarn for JS show up and now this. The main problem with requirements.txt, as I see it, is that you don't get exact versions unless you specify it in your requirements.txt. So you'd have to have a loose requirements.txt and then generate a second requirements file after having done `pip install -r requirements.txt` to get the exact versions that were installed. Further, if you happen to "accidentally" `pip install some-package` in your virtual environment, your app might now be using different packages locally without you noticing. With Pipfile the need for virtual environments is pretty much gone, assuming that at runtime it will automatically load the version of a package specified in the lockfile, which is not clear to me yet from the README.
- genofon 10y agomaybe I misunderstood your comment, but it's pretty straightforward to get the exact version, if you do pip freeze > requirements.txt it will have all the files' versions specified
- fphilipe 10y agoThat's what I meant by generate a second requirements file after having done `pip install -r requirements.txt` to get the exact versions that were installed So you'd need to have a `requirements.txt` with loose versions suitable for upgrading your apps deps, run `pip install -r requirements.txt` and then `pip freeze > requirements.locked.txt`. Then everyone should be using `pip install -r requirements.locked.txt` as well as during your build. But that's cumbersome and error prone and doesn't free you from having the wrong version of a dep in case you `pip install some-package` later on.
- corney91 10y agoI'm not sure why you'd need two requirements.txt. You'd normally create a virtualenv, pip install what you need, then lock the versions with "pip freeze > requirements.txt". You don't need an initial requirements.txt to install new packages.
- shadowmint 10y agoFirst of all, this is great, and I'm hugely in favour of something like this going forward. Great work! ...however, I strongly disagree on the benefit of making the `Pipfile` executable python. Just read this gist: https://gist.github.com/kennethreitz/4745d35e57108f5b766b8f6ff396de85 https://gist.github.com/kennethreitz/4745d35e57108f5b766b8f6... - This file will be "compiled" down to json. Then why does it exist? We know it'll be abused; and we should have learnt our lesson from scons and setup.py that it wasn't a great idea before, and still isn't a great idea using python code itself as a declarative DSL. Just use a standard hierarchical file format (json, toml, xml, whatever) Features of introspecting and editing `Pipfile.lock` should be rolled into pip and exported as a core python module; an api for editing pipfile.lock is a good idea, but executing a `Pipefile`, is not.
- tlocke 10y agoI'm afraid I never saw what was wrong with specifying dependencies in setup.py. For me, having requirements.txt and setup.py is confusing. Can't we just stick with setup.py? (and yes, I've read Donald Stufft's post https://caremad.io/posts/2013/07/setup-vs-requirement/ https://caremad.io/posts/2013/07/setup-vs-requirement/ and remain unconvinced).
- sambe 10y agoI best way this was explained to me was to keep your unversioned dependencies in setup.py and keep developing and testing against latest. Then when you release, requirements.txt should be a result of the build (not an input) which says "these versions work, this is how you install this release". It makes more sense for an application than for a library, but I think making that the primary distinction only adds confusion. All that said, I have no clue what Pipfile is adding. The rationale appears to be that people sometimes don't use requirements.txt properly? Can one of the very enthusiastic commentators explain their enthusiasm?
- Sean1708 10y agoAnd since this Pipfile is just a bunch of function calls, why can't it just be put in setup.py? Or is the idea to eventually get rid of setup.py?
- nchammas 10y ago> And since this Pipfile is just a bunch of function calls That hasn't been settled yet. It's being actively discussed [0]. > Or is the idea to eventually get rid of setup.py? Yes, a replacement for setup.py has already been agreed on in PEP 518 [1]. It's called pyproject.toml. [0] https://github.com/pypa/pipfile/issues/10 https://github.com/pypa/pipfile/issues/10 [1] https://www.python.org/dev/peps/pep-0518/ https://www.python.org/dev/peps/pep-0518/
- theptip 10y agoThat PEP leads down an interesting path -- hadn't realized there was a completely new packaging tool called "flit" that's aiming to be much more lightweight than setuptools. https://pypi.python.org/pypi/flit https://pypi.python.org/pypi/flit Looks quite nice, a flat .ini file for specifying deps, though it only builds wheels.
- yladiz 10y agoThis is great, and a definite step up from the current requirements.txt. Why are the requirements like function calls though? Why can't they just keep a simple text format that's more human readable and editable, rather than something that just looks like a bunch of code? I think it's better in Ruby because you can omit parentheses and it looks cleaner, but my preference above both is package.json. Cargo is also really good. So in my eyes, moving to something like this is somewhat of a step backwards from requirements.txt, but taken together it looks like it will be better overall. There are arguments for and against, but in general I think a configuration file should be human readable and editable, as well as easily understood by an IDE without having to run an interpreter. So something like YAML or TOML, or even a simple INI would be better than the function calls in the Pipfile. However, the lock file isn't meant to be edited by hand or really ever looked at, so it being in JSON or something less human readable is fine. Also, why doesn't pip just make it, by default, look for the requirements.txt or the Pipfile? It's silly having to type pip install -r requirements.txt and I will also find it silly having to type pip -p. It's how NPM, Bundler, and many other package managers work; why do I need to call a flag to install requirements?
- scarygliders 10y ago> why do I need to call a flag to install requirements? Because pip is used extensively as a manual command to install something - you don't expect something like DIR or ls to automatically get its list of files to list from a DIR.txt or a ls.txt file, do you?
- SFJulie 10y agojson + code of conduct? .... Oh dear, python going on its way to the ruby ghetto with security holes to predict.
- blr246 10y agoAs rickycook mentions in another comment, pip-tools is a great solution for maintaining the distinction between allowed version ranges and locked, fully qualified versions for the given environment. I've been using pip-tools with tox for a couple of years now. I maintain a requirements.in and requirements.testing.in, and then I can run $ tox -e pip-compile to generate my fully qualified requirements. The pip-compile command is handled by a tox.ini section. [testenv:pip-compile] commands = pip-compile {posargs} pip-compile -o requirements.testing.txt {posargs} requirements.testing.in deps = pip pip-tools The remaining nasty part is automated extraction requirements for setup.py's install_requires and dependency_links. I wrote a function to handle VCS links and other complicated syntax that I'm copying around to all of my projects. Otherwise, pip-tools has been a great solution.
- StavrosK 10y agoExactly, this is a much better way of doing pinning, if only because it's much more human readable and easily parsable. I've been using it for a while as well, and find it very convenient. It seems to me that that's where we should be heading towards.
- ftxdri 10y agoMy initial reaction to reading this was "finally". Coming from a ruby background I found the package manager of Python to be seriously lacking in ease of use respects. Kudos to the pypa team!
- joss82 10y agoWhat is wrong with requirements.txt seriously? I've used it in countless projects and it just works. It's also simple to understand.
- somecallitblues 10y agoInstalling from GitHub is a pain and if you upgrade packages and freeze into requirements.txt you can't tell which versions you've overwritten. You can easily stuff this up with a deploy script.
- kenneth_reitz 10y agoThe problem is that it can be used in both a deterministic and non-deterministic way (no version-specifiers specified, not all sub-dependencies listed). People want the best of both worlds. This facilitates that.
- joss82 10y agoBut since you use one virtual env per project, you don't overwrite anything.
- GlennS 10y agoMy main problem with Pip is that, if you include multiple requirements.txt files which declare exactly the same dependencies, then that's an error. If you don't control all the projects involved, this needs some quite convoluted inheritance chains to work around. I can't tell if this change will fix that.
- tuco86 10y agoI have not used ruby extensively, but a lot of times i had to install something with it i got stuck with version conflicts of ruby itself and package deps. In my python workflow everything lives within a project virtualenv. Dependencies are defined in setup.py with install_requires, extras_require and tests_require. I build against latest, version constraints are added mostly when the latest version of a package has a problem. Now when i commit to dev, stage or prod branch our ci generates a version pinned requirements.txt which is used to install the virtualenv on stage/prod. I don't really see the improvements in Pipfile, feels like https://xkcd.com/927/ https://xkcd.com/927/.
- teaearlgraycold 10y ago>but a lot of times i had to install something with it i got stuck with version conflicts of ruby itself and package deps. >In my python workflow everything lives within a project virtualenv Were you using rbenv or rvm with Ruby?
- tuco86 10y agono, don't remember why that wasn't an option - maybe i was not even aware of it. I am not really experienced in the ruby toolset, just trying to use tools written in ruby.
- Groxx 10y agoYeah, sadly Ruby has the same environment issues as Python. Python has virtualenv, Ruby has rbenv - they're essentially required, sometimes even for global binaries. For Python, there's `pipsi` for automatically creating unique virtualenvs for binaries - that's the right approach, and it's worth adopting ASAP for future sanity. It even lets you mix python 2 and 3 binaries without any issues. I don't know what the equivalent would be for Ruby.
- djsumdog 10y agoI feel like Docker can be used in place of both python and ruby virtual environments. There are official containers for both.
- fermigier 10y agoI've been using pip-tools for the last year or so. It does a similar job so far. I'd be happy to switch to something that's supported out of the box by pip, though.
- orblivion 10y agoI wonder if they're using the term "deterministic build" in the same way that, say, Debian uses it, and whether they should choose something else to describe this.
- cx1000 10y agoWhile we're on the topic... the name requirements.txt is so generic and undescriptive. It breaks the principle of least astonishment. Something like packages.pip would make more sense.
- msane 10y agoIt contains requirements. The format is custom and it's text.
- cx1000 10y ago> The format is custom What does that even mean? A .txt file is meant for human readable sentences, yet this file is meant to be consumed by pip. You can only "customize" it working within the confines of pip freeze. If you were new to programming, would you suspect that requirements.txt was somehow linked to Python? Probably not. Why can't ruby or JavaScript or any language make use of requirements.txt? They could, and make the same claim you just made, which would break pip install -r.
- dismantlethesun 10y agoText simply means text format, not that "humans should read this". The human editability of text is a nice-to-have, but not a requirment of the format structure. For a comparision, NPM has a packages.json which is hand-editable, and lots of people do customize it by hand to get their NPM scripts running, but: a) If you're new to programming, you might be confused by which language it belongs to just like a requirements.txt b) if you peek into it, you may not think it's generated, consumed, or editted by machines because JSON format is so readable
- cx1000 10y agoI never said humans should read it, just that it's human readable. And of course requirements.txt is technically human readable, but the problem is that it breaks scripts if you do anything other than pip freeze to it. This is obvious in the npm world (you better put JSON in a .json file!). My point is that the .txt extension doesn't give any hints to its use or that it's breakable, and that the name "requirements" doesn't describe the intended domain of the file (Python only). Even naming it "python_requirements.txt" would be an improvement.
- sleek 10y agoAll I ask is that if you add a freeze command, make sure that it remembers a git-style install of a package so that it yields the right directive later on. (the full git path and ref, NOT the egg name)
- grincho 10y agoDon't forget that pip 8 and above have hash verification built-in if that's all you need: https://pip.readthedocs.io/en/stable/reference/pip_install/#hash-checking-mode https://pip.readthedocs.io/en/stable/reference/pip_install/#.... A hash-happy `pip freeze` equivalent is https://pypi.python.org/pypi/hashin https://pypi.python.org/pypi/hashin.
- SonOfLilit 10y agoI have utmost respect for the author, so much that I blindly trust the result will be great. (He made the requests package, among other things.) Just one request: please, please add a --save option to pip like npm's.
- falsedan 10y agoThis feels like putting the cart before the horse: proposing a tool to solve a problem, without opening the problem to discussion, gathering requirements/desired features, and comparing it to similar tools in other languages. I welcome any improvement to python packaging, but I'd start with improving pip itself & making it into a library that tools can wrap around.
- mattlong 10y agoAs someone who has been dealing with managing Python requirements a lot lately, I was excited to see what Pipfile is all about. After reading the post and all the comments here, it's still not really clear to me what the value add is over existing solutions. There are a lot of mentions of deterministic builds, but that is already very achievable with pip-compile (part of pip-tools) or just pip freeze. The grouping functionality allows you to have just one requirements file instead of one per environment (i.e. production, test, development, etc) which is mostly just a personal preference IMO. This isn't particularly compelling to me, but that could be because I'm already used to the traditional "pythonic" way of having one file per environment. Using the -r command within a requirement file allows one to recursively include other requirements files to avoid duplication of common dependencies across environments. The difference in syntax between traditional requirements files and Pipfiles is indeed pretty large. The Pipfile syntax is quite a bit more verbose which I'm personally not a fan of, but this will come down to personal preference and what one is familiar with. It's unclear if Pipfiles as proposed here is meant to include the dependency resolution functionality of the pip-compile command provided by pip-tools. That is a very critical step as vanilla pip makes no guarantees about respecting version pins of nested dependencies; only that some version of a nested dependency will be present but not necessarily the one intended. Another big unknown that others have asked about as well is how Pipfiles can be used to manage requirements for a library in a way that allows other libraries/apps that do not use Pipfiles themselves to still list said library in their requirements.txt. Apologies if my comments comes across as overly negative or dismissive; I applaud any effort to improve the tooling around Python dependencies. But as someone already familiar with the Python/pip ecosystem, it's not clear how this would improve or simplify the solutions that are already out there.