28 ms·
A Better Pip Workflow
- xuan 11y agoNice post as alway from Kenneth. However, this workflow has a little drawback. If you have a dependency not from pipy, e.g. `pip install git+ssh://github.com/kennethreitz/requests.git@master`, it won't work.
- shadowmint 11y agoyes. Its a continual irritation to me 'pip freeze' doesnt support this.
- 89vision 11y agoI've given up trying to use vcs links as dependencies with pip. I deploy everything to an internal devpi server and use --extra-index-url
- K0nserv 11y agoThe lack of separation between requested direct dependencies and a pinned resolved dependencies(including transitive) has been one of the great confusions for me learning Python. Having used Bundler, CocoaPods, Composer, and NPM before, which all have this separation builtin, pip feels broken. However there are a few project that tries to solves this, but the fact that the Python community has not decided on one cripples any initiative to fix it.
- mattnewton 11y agoI can second this, I wish every time I interact with pip that it was npm. I never appreciated the KISS approach npm uses of just dumping all the dependencies in local folder exhaustively until I visited Python versioning hell upon myself by not using a virtualenv (which is an ugly hack itself).
- true_religion 11y agoNot to mention that NPM can resolve different version dependencies so if A requires v2 and B requires v3 of a module, they both can live separate and happy lives. The above is possible for Python as well; I sketched out an implementation which patches __import__ to handle dependency resolution by version, but.... I'm afraid it's a bit unpythonic.
- StavrosK 11y agoHow does NPM do it? My requirements.txt looks like this and I've never had any problems: foo ~= 1.8.2 bar ~= 2.4.1 baz (These are only requested dependencies, resolved are not specified).
- K0nserv 11y agoI misspoke a little with regards to NPM. NPM has something called npm-shrinkwrap that allows you to lock resolved dependencies. It's used in many NPM projects and seems to be a standard chosen by the community. I am not sure what ~= means in requirments.txt, but I'm gonna guess it means something like ~> or ^. With as system like that if everyone follows semver correctly we are fairly okay. The problem is that not everyone does and you have no guarantee that deploying the same code at two points in time t1 and t2 will produce the same application since one of the dependencies might have released new code.
- amiraliakbari 11y agoInterestingly, the separation was the main problem I encountered deploying a Rails application for the first time. I still prefer the PIP's approach, as it promotes always using latest versions.
- akx 11y agoThe `pip-tools` utilities automate this nicely.
- thejash 11y agoThey automate that, plus more. Everyone interested in the original link should really check it out: https://github.com/nvie/pip-tools https://github.com/nvie/pip-tools
- nikolay 11y ago... but it's notorious for not working on latest Python and pip.
- njharman 11y agoA limitation you are not bound to cause they are open source.
- nikolay 11y agoSo, I should learn the inner-workings of pip just to use a basic tool that exists both in Node.js and Ruby land and a dozen others?
- njharman 11y agoYes. Or, wait until some one else does. This is how open source works. How do you think Node.js and Ruby got the capability? Do you imagine they sprout fully formed from hyperbole like "a dozen others".
- nikolay 11y agoI've been waiting, trust me. It's never been working with the latest pip, sorry!
- bobinator606 11y agolike it or hate it, this is exact how its supposed to be done.
- castis 11y agoThat doesn't mean its the way it should always be done though.
- sprayk 11y agoI was thinking about this the other day when I was helping a friend with his method #1-style requirements.txt, and how I wish there was something similar to composer's "lockfile". The author's proposed method is basically the same as how php's composer does it, with its composer.json and composer.lock. Specify your application requirements by hand in composer.json, run composer install, and composer.lock is generated. Check both in so you can have consistent deploys. When you want to upgrade to the latest versions within constraints set by hand in composer.json, run composer update to pull latest versions, updating composer.lock. Run tests, and commit the new composer.lock if you are satisfied.
- Legion 11y ago> The author's proposed method is basically the same as how php's composer does it Composer merely cloned Ruby's Bundler and its Gemfile/Gemfile.lock in that regard. Which is a good thing. It's beyond puzzling that Python has spawned multiple dependency managers, none of which have replicated the same golden path.
- servilio 11y agoThat it hasn't been adopted as a core functionality doesn't there isn't one, zc.buildout first stable release predates bundler's 0.3.0 by at least one year: https://pypi.python.org/pypi/zc.buildout/1.0.1 https://pypi.python.org/pypi/zc.buildout/1.0.1
- bpicolo 11y agoThis is sort of the same thing setup.py is intended for. The way I've seen this successful in practice across many projects: setup.py: specify top-level (i.e. used directly by the application) dependencies. No pinned deps as general practice, but fine to put a hard min/max version on them if it's for-sure known. requirements.txt: Pin all deps + sub-deps. This is your exactly known valid application state. As mentioned, a hasty deploy to production is not when you want to learn a dependency upgrade has broken your app. requirements-dev.txt: dev dependencies + include requirements.txt
- neuroid 11y agoThis. Using setup.py for specifying min versions (and occasionally max versions) is definitely the way to go. However, manually maintaining a requirements file in addition to setup.py is quite tedious in the long run. It is much better to freeze requirements during the build process and use the generated requirements file for deployments. But, decent test coverage is key here.
- falsedan 11y agoThis is easily the best option, barring how painful it is to write a setup.py.
- chishaku 11y agoFilling out setup.py and other package boilerplate used to annoy me as well but that all changed after I started using cookiecutter. https://github.com/audreyr/cookiecutter https://github.com/audreyr/cookiecutter
- falsedan 11y agoThanks! I use cookiecutter and I still have to tweak things like MANIFEST.in and adding the package to the path while installing, for automatically-created entry points.
- snuxoll 11y agoIt's really not that bad with setuptools - as an added bonus it becomes easy to package your application up in an RPM.
- crdoconnor 11y ago>While the Method #2 format for requirements.txt is best practice, it is a bit cumbersome. Namely, if I’m working on the codebase, and I want to $ pip install --upgrade some/all of the packages, I am unable to do so easily. $ pip freeze --local | grep -v '^\-e' | cut -d = -f 1 | xargs ./venv/bin/pip install -U Ought to be a first class command, though. After running that the first thing I do is run the tests. Then I freeze, commit and push. This is usually a good thing to do at the beginning of a new sprint so that more subtle bugs caused by upgrading packages can be teased out before releasing a new build. For my projects that I release on pypi I don't want to use pinned dependencies, but for them I run periodic tests that download all dependencies and run tests so that I'm (almost) instantaneously notified when a package I depend upon causes a bug in my code (e.g. by changing an API).
- tibbon 11y agoI'm probably going to get downvoted to hell on this; but as a Rubyist who has been working on a Python project, I've been finding pip to be really weak against rubygems/bundler. With RubyGems/Bundler I love the ability to point to github repos, lock versions (or allow minor/patch versions to update), have groups, etc. requiresments.txt and pip just feels, awkward and weird. Especially when combined with virtualenv, in comparison to Ruby this is just stiff and strange. I've had nothing but problems with more complex packages like opencv and opencl as well.
- deleted 11y ago[deleted]
- scandinavian 11y ago>With RubyGems/Bundler I love the ability to point to github repos You can do this with pip.
- mrharrison 11y ago> I've had nothing but problems with more complex packages I could say the same things about rubygems. With coding a lot of it is what you are familiar with. To me python and pip is clean and simple, ruby and rubygems is overly complex. But that's because I'm familiar with python, so yeah.
- infecto 11y agoI am fairly positive you can do everything you have described. Just doing a quick google search returns the answers I need. 1. Point to github repos. `pip install git+ssh://git@github.com/echweb/echweb-utils.git` http://stackoverflow.com/questions/4830856/is-it-possible-to-use-pip-to-install-a-package-from-a-private-github-repository http://stackoverflow.com/questions/4830856/is-it-possible-to... 2. Lock minor versions `Django>=1.3,<1.3.99` http://stackoverflow.com/questions/6047670/pip-specifying-minor-version http://stackoverflow.com/questions/6047670/pip-specifying-mi... What is a group? My google search did not turn up anything relevant.
- brbsix 11y ago
- blaze33 11y agoI would additionally recommend to use : pip freeze -r requirements-to-freeze.txt > requirements.txt instead of just: pip freeze > requirements.txt So you can keep your file structure with comments, nicely separating your dependencies from the dependencies of your dependencies. Actually I only used one requirements.txt file, removing everything below "## The following requirements were added by pip --freeze:" and regenerating it when I changed my dependencies. And sure, beware of git urls being replaced by egg names in the process.
- nikolay 11y agoNice! Didn't know this was even possible!
- nikolay 11y agoDuh! That's how I was using it along with the -l option!
- true_religion 11y agoAnother tool you might find useful is pipreq. pipreq --savepath gen.requirements.txt / The above will run through all your source code, and generate the requirements that you actually use via imports. https://github.com/bndr/pipreqs https://github.com/bndr/pipreqs Disclaimer: I contribute to it.
- StavrosK 11y agoDoes it work well? I can't imagine it would be very accurate for things like a Django project...
- true_religion 11y agoIt works pretty well for a Django project with the exception of anything that's only called from INSTALLED_APPs. I have a branch that can handle Django, but I kept it private because it's not really general purpose to introspect Django's settings.py then read out data.
- edavis 11y agoWith this workflow, what purpose does requirements.txt serve? Would the file ever be used directly? Only thing I can think of is you'd track top-level packages in requirements-to-freeze.txt during development, while your deploy would use requirements.txt to get a deterministic environment.
- fredsir 11y agoThat is indeed the idea.
- david-given 11y agoI still find myself automatically thinking of this PIP. http://www.shaels.net/index.php/cpm80-22-documents/using-cpm/6-pip-utility http://www.shaels.net/index.php/cpm80-22-documents/using-cpm...
- aristonmetron 11y agoshhh... don't tell anyone about wheels...
- Diederich 11y agoPinto (https://metacpan.org/pod/distribution/Pinto/lib/Pinto/Manual/Introduction.pod https://metacpan.org/pod/distribution/Pinto/lib/Pinto/Manual...) nails this problem in the Perl ecosystem: Pinto has two primary goals. First, Pinto seeks to address the problem of instability in the CPAN mirrors. Distribution archives are constantly added and removed from the CPAN, so if you use it to build a system or application, you may not get the same result twice. Second, Pinto seeks to encourage developers to use the CPAN toolchain for building, testing, and dependency management of their own local software, even if they never plan to release it to the CPAN. Pinto accomplishes these goals by providing tools for creating and managing your own custom repositories of distribution archives. These repositories can contain any distribution archives you like, and can be used with the standard CPAN toolchain. The tools also support various operations that enable you to deal with common problems that arise during the development process.
- goerz 11y agoI would argue that the information in "requirements-to-freeze.txt" could simply be expressed in the "install_requires" list in setup.py. That is, the "general" dependencies (with only vague version information) should be in setup.py, whereas requirements.txt should pin the exact "guaranteed to work" versions. For small/non-commerical projects, requirements.txt may not even be necessary.
- mixmastamyk 11y agoSo, you use `python setup.py` instead of pip?
- goerz 11y agoNo...? AFAIK (and based on my daily experience), pip processes setup.py, so it definitely installs the dependencies listed there.
- mixmastamyk 11y agoOk, so you just skip the requirements file.
- smnrchrds 11y agoWhat does "requests[security]" do? I don't remember ever using the bracket syntax.
- krallin 11y agoThis means "install the `requests` package with the `security` extras. In this particular case, this installs 'pyOpenSSL>=0.13', 'ndg-httpsclient', 'pyasn1'. See: https://github.com/kennethreitz/requests/blob/46184236dc177fb68c7863445609149d0ac243ea/setup.py#L94-L96 https://github.com/kennethreitz/requests/blob/46184236dc177f...
- antonagestam 11y agoFrom the docs: extras_require A dictionary mapping names of “extras” (optional features of your project) to strings or lists of strings specifying what other distributions must be installed to support those features. See the section below on Declaring Dependencies for details and examples of the format of this argument.
- luhn 11y agoBrackets are used to install recommended dependencies. See http://pythonhosted.org/setuptools/setuptools.html#declaring-extras-optional-features-with-their-own-dependencies http://pythonhosted.org/setuptools/setuptools.html#declaring... In the case of requests[security], it installs some extra packages that allow for more secure SSL. http://stackoverflow.com/questions/31811949/pip-install-requestssecurity-vs-pip-install-requests-difference http://stackoverflow.com/questions/31811949/pip-install-requ...
- deleted 11y ago[deleted]
- amiraliakbari 11y agoPractically, it is used to prevent newer versions of pip from complaining about "insecure platform" and have cleaner CI build logs.
- nikolay 11y agoI personally use requirements.txt for source and requirements.txt.lock, which I find more standard. Or maybe use the pip tools' standard: requirements.in for source and requirement.txt for compiled at least.
- njharman 11y agoInstead of doing that manually, use pip-tools https://github.com/nvie/pip-tools/ https://github.com/nvie/pip-tools/ Some posts on theory and use http://nvie.com/posts/pip-tools-10-released/ http://nvie.com/posts/pip-tools-10-released/
- blaze33 11y agoHe explicitly wrote: "I thought long and hard about building a tool to solve this problem. Others, like pip-tools, already have. But, I don’t want another tool in my toolchain; this should be possible with the tools available." so it's not like he's not aware of pip-tools.
- kenneth_reitz 11y ago/me has commit access to pip-tools.
- jbmorgado 11y agoI would also suggest using anaconda if you are using python for science: https://www.continuum.io/downloads https://www.continuum.io/downloads It makes managing the python packages and even the python versions quite easy.
- ekmartin 11y agoWe built Doppins (https://doppins.com https://doppins.com) to be able to use pinned PyPI dependencies and/or ranges and still keep them up-to-date continuously. Still another tool, but it's quite quick to enable on a repository and doesn't require any maintenance afterwards.
- 89vision 11y agoI've been using pip-tools to manage my requirements. It allows you to specify a top-level requirements file, and from that it builds you a requirements.txt with specific versions and all dependencies. It has really streamlined my pip workflow. https://github.com/nvie/pip-tools https://github.com/nvie/pip-tools
- conradev 11y agoYep, I'm surprised no one knows about pip-compile. It does exactly as the OP suggests, but with the ability to specify a range of versions.
- deepwalker 11y agoHey, I have wrote Pundle to solve this problem! Idea is simple — we just have package to install other packages with versions freeze. Pundle maintaine frozen.txt alongside with requirements.txt. What more — pundle does not use virtualenv and install all packages to user directory ~/.Pundledir and import frozen versions on demand. It have all nice commands like install, upgrade, info etc Check it https://github.com/Deepwalker/pundler https://github.com/Deepwalker/pundler
- darkerside 11y agoAnother method is to use pip constraints to enforce package versions https://pip.pypa.io/en/stable/user_guide/#constraints-files https://pip.pypa.io/en/stable/user_guide/#constraints-files
- itissid 11y agoI found myself using tox for managing the general case of this. But the spirit to the blog is in the same direction.
- p4wnc6 11y agoIf you don't mind committing to Anaconda Python distributions, then you should simply use conda. You can still pip-install things within a conda environment, and conda can manage more dependencies than just Python dependencies (a common use case is managing R dependencies for a Python statistical workflow). You can do conda list -e > requirements.txt then conda create -n newenv --file requirements.txt to create a Python environment from the frozen requirements. I believe that conda makes it easier to selectively update, but even if you don't enjoy those features of conda, the same two-file trick as in this post will work for conda as well, since you can use `conda update --file ...`. Conda's "dry-run" features are more useful than pip's as well. The perfect feature for conda to add is the ability to specify alternative Python distributions, either by a path to the executable, or by allowing alternative Python distributions to be hosted on Binstar. I can understand why Continuum wants conda to heavily influence everyone to use only Anaconda, but I think the goodwill of making conda work for any Python distribution would bring more to them than keeping it focused solely on Anaconda. (For example, I know some production environments that still use Python 2.6 and are prevented from updating to 2.7 -- and even if they did update, they'd need to keep around some managed environments for 2.6 for testing and legacy verification work).
- odonnellryan 11y agoI agree. Only used conda on a few projects just recently, so not too battle tested, but I've found it much easier to use, especially with "difficult" libraries.
- drcongo 11y agoVaguely related, I hacked together a little tool called Olaf[1] for people who like to pip freeze but also like to have multiple distinct requirements files like requirements-dev.txt – Needs a little work still but I've been using it on projects quite happily. [1] https://pypi.python.org/pypi/olaf https://pypi.python.org/pypi/olaf