4 ms·

This is sort of the same thing setup.py is intended for. The way I've seen this successful in practice across many projects: setup.py: specify top-level (i.e.

by bpicolo 11y ago

This is sort of the same thing setup.py is intended for. The way I've seen this successful in practice across many projects: setup.py: specify top-level (i.e. used directly by the application) dependencies. No pinned deps as general practice, but fine to put a hard min/max version on them if it's for-sure known. requirements.txt: Pin all deps + sub-deps. This is your exactly known valid application state. As mentioned, a hasty deploy to production is not when you want to learn a dependency upgrade has broken your app. requirements-dev.txt: dev dependencies + include requirements.txt

neuroid 11y ago
This. Using setup.py for specifying min versions (and occasionally max versions) is definitely the way to go. However, manually maintaining a requirements file in addition to setup.py is quite tedious in the long run. It is much better to freeze requirements during the build process and use the generated requirements file for deployments. But, decent test coverage is key here.
falsedan 11y ago
This is easily the best option, barring how painful it is to write a setup.py.
- chishaku 11y ago
  Filling out setup.py and other package boilerplate used to annoy me as well but that all changed after I started using cookiecutter. https://github.com/audreyr/cookiecutter https://github.com/audreyr/cookiecutter
  - falsedan 11y ago
    Thanks! I use cookiecutter and I still have to tweak things like MANIFEST.in and adding the package to the path while installing, for automatically-created entry points.
- snuxoll 11y ago
  It's really not that bad with setuptools - as an added bonus it becomes easy to package your application up in an RPM.
  - falsedan 11y ago
    setuptools? distutils you mean? I much prefer fpm for creating Linux distro packaged from a python distribution, since it can create debs.
    - snuxoll 11y ago
      No, I mean setuptools - it does everything distutils does but better. I don't trust random scripts to generate packages for me, writing an rpmspec or a debian control file is hardly a challenge and I'd encourage more people to take the 5 minutes to do so than relying on tools like FPM.
      - falsedan 11y ago
        … or tools like setuptools? Also, I assume you're talking about the bdist_rpm target that come from distutils.
        snuxoll 11y ago
        No, I write a proper RPM spec file - only takes a few minutes.
        falsedan 11y ago
        So, what's the added bonus of using setuptools then?
        snuxoll 11y ago
        setuptools is just less generally crufty than distutils, and things like entrypoints make writing installable console scripts feasible. Since it provides essentially the same interface the default rpmspec created by rpmdev-newspec works with it out of the box.
jsmthrowaway 11y ago
I've bookmarked Donald Stufft's opinion on this for a couple years, which contrasts interestingly with Kenneth's and your own (and which makes a lot of sense): https://caremad.io/2013/07/setup-vs-requirement/ https://caremad.io/2013/07/setup-vs-requirement/ I like the concrete-vs-abstract, library-vs-application ideas there. Most opinions on this overlook the distinction.
- bpicolo 11y ago
  I don't at all disagree with that, actually. I do consider that a part of setup.py, but it rarely matters in practice. The python ecosystem doesn't have similar programmatic api specs compared to, say, PHP where they get together and rigidly define these things: http://www.php-fig.org/psr/psr-7/ http://www.php-fig.org/psr/psr-7/ Not to say the PHP strategy is necessarily ideal (feels restrictive to me), rather just the comparison between the two. That's the sort of ecosystem that would make the abstract part matter more
  - talideon 11y ago
    No, no, Python very much does. PEP 0249 specifies DB API 2, PEP 0333 specifies WSGI, and PEP 3153 specifies the asyncio interface, for instance. If anything, I can think of more examples of it being done in the Python space than the PHP space.
ianbicking 11y ago
This in fact was my intention when creating this feature in pip: setup.py has the abstract dependencies, requirements.txt is a recipe for building something exact. And I quickly found requirements-dev was useful for adding tools (some people have done that with setup.py extras, but the ergonomics on that never seemed great). The one thing I wish was part of this is that there was a record of conflicts and purportedly successful combinations. That is, you distribute a package that says "I work with foo>=1.0" but you don't (and can't) know if the package works with foo 1.1 or 2.0. Semantic versioning feels like a fantasy that you could know, but you just can't. Understanding how versions work together is a discovery process, but we don't have a shared record of that discovery, and it doesn't belong to any one package version. This sense that package releases and known good combinations are separate things developed at separate paces is also part of the motivation of requirements.txt, and maybe why people often moved away from setup.py
- foxhop 11y ago
  Hi Ian, long time fan of your work. I'm supporting both using some silly logic to pull in requirements.txt and supply that to setup. https://github.com/russellballestrini/botoform https://github.com/russellballestrini/botoform
- e12e 11y ago
  A pacakage/system that handled this would be great. Zope and Plone have the notion of "known good sets" -- and it's not really a pleasure to use (but much better than nothing). As far as I can tell, with Plone 5 - the recommended way to install Plone is from the unified installer - leaving known good sets, pinning and buildout to manage plugins: http://docs.plone.org/manage/installing/installing_addons.html http://docs.plone.org/manage/installing/installing_addons.ht... For an example of what bootstrapping a full Plone 4 site via buildout entails, have a look at the (defunct) good-py project: http://good-py.appspot.com/release/plone/4.2rc2 http://good-py.appspot.com/release/plone/4.2rc2 http://www.martinaspeli.net/articles/hello-good-py-a-known-good-versions-manager http://www.martinaspeli.net/articles/hello-good-py-a-known-g... As long as one is able to manage to keep projects small, and in a virtual-env of it's own, managing "known good sets" (in buildout, or for pip) shouldn't really be too hard. But as projects grow, a real system for managing versions will be needed. As far as I know there are no good systems for this... yet. Ideally you'd want a list that people could update as they run into problems, so that if projectA runs fine with foo=1.0, and bar=1.1, maybe projectB discovers a bug in foo<=1.0.4rc5 and can update the requirement. It's not a trivial thing (see also: All the package managers, apt, yum, etc).

by bpicolo 11y ago

neuroid 11y ago

This. Using setup.py for specifying min versions (and occasionally max versions) is definitely the way to go. However, manually maintaining a requirements file in addition to setup.py is quite tedious in the long run. It is much better to freeze requirements during the build process and use the generated requirements file for deployments. But, decent test coverage is key here.

falsedan 11y ago

This is easily the best option, barring how painful it is to write a setup.py.

chishaku 11y ago
Filling out setup.py and other package boilerplate used to annoy me as well but that all changed after I started using cookiecutter. https://github.com/audreyr/cookiecutter https://github.com/audreyr/cookiecutter
- falsedan 11y ago
  Thanks! I use cookiecutter and I still have to tweak things like MANIFEST.in and adding the package to the path while installing, for automatically-created entry points.
snuxoll 11y ago
It's really not that bad with setuptools - as an added bonus it becomes easy to package your application up in an RPM.
- falsedan 11y ago
  setuptools? distutils you mean? I much prefer fpm for creating Linux distro packaged from a python distribution, since it can create debs.
  - snuxoll 11y ago
    No, I mean setuptools - it does everything distutils does but better. I don't trust random scripts to generate packages for me, writing an rpmspec or a debian control file is hardly a challenge and I'd encourage more people to take the 5 minutes to do so than relying on tools like FPM.
    - falsedan 11y ago
      … or tools like setuptools? Also, I assume you're talking about the bdist_rpm target that come from distutils.
      - snuxoll 11y ago
        No, I write a proper RPM spec file - only takes a few minutes.
        falsedan 11y ago
        So, what's the added bonus of using setuptools then?
        snuxoll 11y ago
        setuptools is just less generally crufty than distutils, and things like entrypoints make writing installable console scripts feasible. Since it provides essentially the same interface the default rpmspec created by rpmdev-newspec works with it out of the box.

jsmthrowaway 11y ago

I've bookmarked Donald Stufft's opinion on this for a couple years, which contrasts interestingly with Kenneth's and your own (and which makes a lot of sense): https://caremad.io/2013/07/setup-vs-requirement/ https://caremad.io/2013/07/setup-vs-requirement/ I like the concrete-vs-abstract, library-vs-application ideas there. Most opinions on this overlook the distinction.

bpicolo 11y ago
I don't at all disagree with that, actually. I do consider that a part of setup.py, but it rarely matters in practice. The python ecosystem doesn't have similar programmatic api specs compared to, say, PHP where they get together and rigidly define these things: http://www.php-fig.org/psr/psr-7/ http://www.php-fig.org/psr/psr-7/ Not to say the PHP strategy is necessarily ideal (feels restrictive to me), rather just the comparison between the two. That's the sort of ecosystem that would make the abstract part matter more
- talideon 11y ago
  No, no, Python very much does. PEP 0249 specifies DB API 2, PEP 0333 specifies WSGI, and PEP 3153 specifies the asyncio interface, for instance. If anything, I can think of more examples of it being done in the Python space than the PHP space.

ianbicking 11y ago

This in fact was my intention when creating this feature in pip: setup.py has the abstract dependencies, requirements.txt is a recipe for building something exact. And I quickly found requirements-dev was useful for adding tools (some people have done that with setup.py extras, but the ergonomics on that never seemed great). The one thing I wish was part of this is that there was a record of conflicts and purportedly successful combinations. That is, you distribute a package that says "I work with foo>=1.0" but you don't (and can't) know if the package works with foo 1.1 or 2.0. Semantic versioning feels like a fantasy that you could know, but you just can't. Understanding how versions work together is a discovery process, but we don't have a shared record of that discovery, and it doesn't belong to any one package version. This sense that package releases and known good combinations are separate things developed at separate paces is also part of the motivation of requirements.txt, and maybe why people often moved away from setup.py

foxhop 11y ago
Hi Ian, long time fan of your work. I'm supporting both using some silly logic to pull in requirements.txt and supply that to setup. https://github.com/russellballestrini/botoform https://github.com/russellballestrini/botoform
e12e 11y ago
A pacakage/system that handled this would be great. Zope and Plone have the notion of "known good sets" -- and it's not really a pleasure to use (but much better than nothing). As far as I can tell, with Plone 5 - the recommended way to install Plone is from the unified installer - leaving known good sets, pinning and buildout to manage plugins: http://docs.plone.org/manage/installing/installing_addons.html http://docs.plone.org/manage/installing/installing_addons.ht... For an example of what bootstrapping a full Plone 4 site via buildout entails, have a look at the (defunct) good-py project: http://good-py.appspot.com/release/plone/4.2rc2 http://good-py.appspot.com/release/plone/4.2rc2 http://www.martinaspeli.net/articles/hello-good-py-a-known-good-versions-manager http://www.martinaspeli.net/articles/hello-good-py-a-known-g... As long as one is able to manage to keep projects small, and in a virtual-env of it's own, managing "known good sets" (in buildout, or for pip) shouldn't really be too hard. But as projects grow, a real system for managing versions will be needed. As far as I know there are no good systems for this... yet. Ideally you'd want a list that people could update as they run into problems, so that if projectA runs fine with foo=1.0, and bar=1.1, maybe projectB discovers a bug in foo<=1.0.4rc5 and can update the requirement. It's not a trivial thing (see also: All the package managers, apt, yum, etc).