5 ms·

Setup.py vs requirements.txt

64 points by donaldstufft 13y ago

severb 13y ago
This is just plain wrong. There is no reason not to use setup.py for your private "application" as he calls it. Even if it's not distributed trough the public index you may still benefit from having version numbers and "abstract" dependencies for your app. That way you may choose where to look for the dependencies at deploy time and use a private cheeseshop for delivering your private packages, not to mention that requirements.txt is a pip only thing.
- donaldstufft 13y ago
  Actually you've misread it or I didn't explain it well enough :) A private library/app can have abstract dependencies. As I said in the post it deals with the "deploy specific" side of a Python application. So each specific deployment would get it's own requirements.txt. And yes requirements.txt is a pip only thing but the concept applies to the similar file in zc.buildout whose name escapes me at the moment and any other similar file.
dwightgunning 13y ago
The setup.py approach that loosely couple dependencies is definitely useful as the author describes. However I think important advice to library developers that's missing in the article is to be explicit (somewhere, likely in documentation) about the version/source of each dependency that the library has been tested against. Libraries are distributed, applications are deployed.
- donaldstufft 13y ago
  Sure that'd be good too :) This wasn't really aiming to be a comprehensive best practices guide to distributing things. Mostly trying to explain the differences between what a dependency in setup.py means vs a requirements.txt like file. I have a hard time coming up with wording because sometimes applications are distributed as well and the "deployment" is just a config file + installing the dependencies. I hate using the term application because of that confusion but I haven't thought of a better way to word it yet :(
kevinastone 13y ago
Are you suggesting using `pip freeze > requirements.txt` as some kind of Gemfile.lock compared to setup.py#install_requires as the standard Gemfile?
- donaldstufft 13y ago
  ``setup.py#install_requires`` would be the gemspec file. Ideally We'd have requirements.txt and requirements.txt.lock which would correspond to Gemfile and Gemfile.lock. However right now that doesn't really happen. So you're kind of stuck having requirements.txt play a double duty where for things where you need exact exact dependencies you'd check in a very exact requirements.txt and things where you have a supported set of deps (maybe more restricted ranges) and you expect people to Deploy by cloning/unzipping a tarball etc you'd use a less restrictive requirements.txt. I would very much love to get a distinction between the two uses of requirements.txt so that we have a better correlation to Gemfile and Gemfile.lock
lifeisstillgood 13y ago
Is there really such a thing as an abstract dependency? There is such a thing as a tested system (ie two or more Independantly built packages that have been demonstrated to work together). This is the concrete - I have built these things and tested them - now take them both as they are and put them live. I now dislikes the idea that I will build packages (with pip literally from source) on production machines. No I don't quite get this - dependencies are just that - we depend on the code they provide to run. A library author cares as much as a sysadmin that they avoid a version of dependency that breaks the package. A better spec might be Requests=1.1,1.2,1.4 #oops 1.3 broke our tests And yes, requirements.txt and setup.py conflict. We don't have a clean packaging system in Python and we should accept it and fix it. None of this Precludes the useful ability to specify which location to retrieve which package from, but these are build instructions, not deployment instructions.
- donaldstufft 13y ago
  I call them abstract for lack of a better word. Referenced by name? It's not about providing versions of code that does or doesn't pass tests. It's about where that code is fetched from. If you reference the dependency by name then you can easily switch it out for something that looks like the code the author originally developed against, but actually has some changes (patches, bugfix, whatever). Where to retrieve packages from is both a build and a deployment instruction. After all you need to fetch the built packages from soemwhere yes? In general "reference by name and version/abstract" allows you to easily swap in a built package for a source package as well. Otherwise you'd be stuck with whatever formats the author published because the dependencies would be pointing to a specific url (likely outside your control). The very fact you can build a binary package and host it yourself works because of what I call abstract dependencies (for lack of a better name). If they wern't abstract then you'd have to fork the dependent package in order to point it at your built package/repository instead.
tehwalrus 13y ago
I have a lot of python code from my PhD which will be released when I pass my viva (with full version history, of course) - I'm not looking forward to working out how to deal with pypi after reading this! I have always tried to make self-contained packages with a simple "just run setup.py" install procedure for my smaller public projects on github, but my larger codebase uses lots of libraries, and specifying which ones people should install is going to be a nightmare...
- ubernostrum 13y ago
  I think you're really overcomplicating this if your use case is to make something installable from PyPI. Think of it like this: * "install_requires" in setup.py = "I don't really care how you get this, or possibly even what version of it you get, just get this". * requirements file = "I have a known-good, tested and functioning setup using these specific versions, from these specific URLs, of these specific packages; I cannot vouch for whether other versions or other distributions of those packages will work".
philipn 13y ago
You refer to dependency_links as a "misfeature," but I'm wondering why?
- donaldstufft 13y ago
  Because it makes you depend on urls instead of abstract name/version requirements. easy_install does provide some tooling to get around it but in general it shouldn't be used.
  - philipn 13y ago
    I guess this is where I'm confused. Isn't this the same as using the '-e' switch inside of a requirements.txt?
m0th87 13y ago
OK, the concept of "abstract dependencies" makes sense, but boy is it a complicated way to fix an edge case. How many times have you had to swap out a concrete dependency vs how many times have you bashed your head against the wall about the proper way to setup a python library? I know my experience skews heavily toward the latter. I wish python had an equivalent to npm. npm does a lot of things right: local packages by default, recursive dependency resolution, a simple declarative package specification (package.json), and the ability to run your own private npm registry. And it wraps up the logic of a bunch of tools: virtualenv + pip + (distutils / setuptools / distribute / distutils2).
- d23 13y ago
  > how many times have you bashed your head against the wall about the proper way to setup a python library Hmm, almost never actually. The only example I can think of is PIL. I usually have to manually edit the installer to get libjpeg and zlib and fooimagelib to install properly. Other than that, pip seems to have all the magic I crave.
  - m0th87 13y ago
    What about creating a new library? It's been a giant PITA for me, especially figuring out which of the 4 major distribution tools to use.
    - viraptor 13y ago
      The latest one. There was only the brief period when setuptools/disttools were confusing as far as I remember. Now, you can just use d2to1 and the new distribute config format.
      - lmm 13y ago
        So what's it called? "distribute"?
        donaldstufft 13y ago
ajanuary 13y ago
I've been working with Ivy which does dependency management for Java. It's got a similar split: an ivy.xml file for specifying what it depends on, and an ivysettings.xml file for specifying what repositories to look in for those dependencies.
teilo 13y ago
Yeah, I just can't agree with this. I maintain a private bitbucket repository for my company. It contains two things: Django apps, and specific projects which implement those Django apps. Each Django app is packaged using a standard setuptools/distribute package. All dependencies of each app is managed in setup.py. I place the projects under version control, in order to maintain the specific templates, url.py, fixtures, and configuration for each site. I use a requirements.txt file to manage the dependencies of the project. There is nothing abstract about either.
deleted 13y ago
[deleted]

64 points by donaldstufft 13y ago

This is just plain wrong. There is no reason not to use setup.py for your private "application" as he calls it. Even if it's not distributed trough the public index you may still benefit from having version numbers and "abstract" dependencies for your app. That way you may choose where to look for the dependencies at deploy time and use a private cheeseshop for delivering your private packages, not to mention that requirements.txt is a pip only thing.

donaldstufft 13y ago
Actually you've misread it or I didn't explain it well enough :) A private library/app can have abstract dependencies. As I said in the post it deals with the "deploy specific" side of a Python application. So each specific deployment would get it's own requirements.txt. And yes requirements.txt is a pip only thing but the concept applies to the similar file in zc.buildout whose name escapes me at the moment and any other similar file.

dwightgunning 13y ago

The setup.py approach that loosely couple dependencies is definitely useful as the author describes. However I think important advice to library developers that's missing in the article is to be explicit (somewhere, likely in documentation) about the version/source of each dependency that the library has been tested against. Libraries are distributed, applications are deployed.

donaldstufft 13y ago
Sure that'd be good too :) This wasn't really aiming to be a comprehensive best practices guide to distributing things. Mostly trying to explain the differences between what a dependency in setup.py means vs a requirements.txt like file. I have a hard time coming up with wording because sometimes applications are distributed as well and the "deployment" is just a config file + installing the dependencies. I hate using the term application because of that confusion but I haven't thought of a better way to word it yet :(

kevinastone 13y ago

Are you suggesting using `pip freeze > requirements.txt` as some kind of Gemfile.lock compared to setup.py#install_requires as the standard Gemfile?

donaldstufft 13y ago
``setup.py#install_requires`` would be the gemspec file. Ideally We'd have requirements.txt and requirements.txt.lock which would correspond to Gemfile and Gemfile.lock. However right now that doesn't really happen. So you're kind of stuck having requirements.txt play a double duty where for things where you need exact exact dependencies you'd check in a very exact requirements.txt and things where you have a supported set of deps (maybe more restricted ranges) and you expect people to Deploy by cloning/unzipping a tarball etc you'd use a less restrictive requirements.txt. I would very much love to get a distinction between the two uses of requirements.txt so that we have a better correlation to Gemfile and Gemfile.lock

lifeisstillgood 13y ago

Is there really such a thing as an abstract dependency? There is such a thing as a tested system (ie two or more Independantly built packages that have been demonstrated to work together). This is the concrete - I have built these things and tested them - now take them both as they are and put them live. I now dislikes the idea that I will build packages (with pip literally from source) on production machines. No I don't quite get this - dependencies are just that - we depend on the code they provide to run. A library author cares as much as a sysadmin that they avoid a version of dependency that breaks the package. A better spec might be Requests=1.1,1.2,1.4 #oops 1.3 broke our tests And yes, requirements.txt and setup.py conflict. We don't have a clean packaging system in Python and we should accept it and fix it. None of this Precludes the useful ability to specify which location to retrieve which package from, but these are build instructions, not deployment instructions.

donaldstufft 13y ago
I call them abstract for lack of a better word. Referenced by name? It's not about providing versions of code that does or doesn't pass tests. It's about where that code is fetched from. If you reference the dependency by name then you can easily switch it out for something that looks like the code the author originally developed against, but actually has some changes (patches, bugfix, whatever). Where to retrieve packages from is both a build and a deployment instruction. After all you need to fetch the built packages from soemwhere yes? In general "reference by name and version/abstract" allows you to easily swap in a built package for a source package as well. Otherwise you'd be stuck with whatever formats the author published because the dependencies would be pointing to a specific url (likely outside your control). The very fact you can build a binary package and host it yourself works because of what I call abstract dependencies (for lack of a better name). If they wern't abstract then you'd have to fork the dependent package in order to point it at your built package/repository instead.

tehwalrus 13y ago

I have a lot of python code from my PhD which will be released when I pass my viva (with full version history, of course) - I'm not looking forward to working out how to deal with pypi after reading this! I have always tried to make self-contained packages with a simple "just run setup.py" install procedure for my smaller public projects on github, but my larger codebase uses lots of libraries, and specifying which ones people should install is going to be a nightmare...

ubernostrum 13y ago
I think you're really overcomplicating this if your use case is to make something installable from PyPI. Think of it like this: * "install_requires" in setup.py = "I don't really care how you get this, or possibly even what version of it you get, just get this". * requirements file = "I have a known-good, tested and functioning setup using these specific versions, from these specific URLs, of these specific packages; I cannot vouch for whether other versions or other distributions of those packages will work".

philipn 13y ago

You refer to dependency_links as a "misfeature," but I'm wondering why?

donaldstufft 13y ago
Because it makes you depend on urls instead of abstract name/version requirements. easy_install does provide some tooling to get around it but in general it shouldn't be used.
- philipn 13y ago
  I guess this is where I'm confused. Isn't this the same as using the '-e' switch inside of a requirements.txt?

m0th87 13y ago

OK, the concept of "abstract dependencies" makes sense, but boy is it a complicated way to fix an edge case. How many times have you had to swap out a concrete dependency vs how many times have you bashed your head against the wall about the proper way to setup a python library? I know my experience skews heavily toward the latter. I wish python had an equivalent to npm. npm does a lot of things right: local packages by default, recursive dependency resolution, a simple declarative package specification (package.json), and the ability to run your own private npm registry. And it wraps up the logic of a bunch of tools: virtualenv + pip + (distutils / setuptools / distribute / distutils2).

d23 13y ago
> how many times have you bashed your head against the wall about the proper way to setup a python library Hmm, almost never actually. The only example I can think of is PIL. I usually have to manually edit the installer to get libjpeg and zlib and fooimagelib to install properly. Other than that, pip seems to have all the magic I crave.
- m0th87 13y ago
  What about creating a new library? It's been a giant PITA for me, especially figuring out which of the 4 major distribution tools to use.
  - viraptor 13y ago
    The latest one. There was only the brief period when setuptools/disttools were confusing as far as I remember. Now, you can just use d2to1 and the new distribute config format.
    - lmm 13y ago
      So what's it called? "distribute"?
      - donaldstufft 13y ago

ajanuary 13y ago

I've been working with Ivy which does dependency management for Java. It's got a similar split: an ivy.xml file for specifying what it depends on, and an ivysettings.xml file for specifying what repositories to look in for those dependencies.

teilo 13y ago

Yeah, I just can't agree with this. I maintain a private bitbucket repository for my company. It contains two things: Django apps, and specific projects which implement those Django apps. Each Django app is packaged using a standard setuptools/distribute package. All dependencies of each app is managed in setup.py. I place the projects under version control, in order to maintain the specific templates, url.py, fixtures, and configuration for each site. I use a requirements.txt file to manage the dependencies of the project. There is nothing abstract about either.

deleted 13y ago

[deleted]