6 ms·

Guido van Rossum: People want CPAN (discussion on Python packages)

65 points by fserb 17y ago

staunch 17y ago
Both the CLI CPAN module/tool itself and http://search.cpan.org/ http://search.cpan.org/ are awesome to work with. I think every language should shamelessly copy them first, and try to improve upon them second. It almost seems like Ruby/Python have been consciously reluctant to do so.
- jrockway 17y ago
  CPAN's success, as with Debian's, is not due to the technical infrastructure but rather the social infrastructure. There are plenty of ways to make packages not work at all, it's just that the Perl people don't do that. (If you have ever read the CPAN source code, you would be surprised it works at all. Don't get me started on the various incompatible build systems, and what happens when your module's build system depends on a newer version of the build system.) Haskell's Cabal is the technical model to steal. Don't let modules execute their own code unless they actually need to. 99.9999% of modules do fine with some sort of declarative interface, rather than actual code to do those things.
  - chromatic 17y ago
    The CPAN works despite clunky code and competing implementations because the functional decomposition is sufficiently effective. (EUMM is some of the worst code in intent, design, and implementation I've ever read and used anyway.)
    - jrockway 17y ago
      The CPAN software infrastructure only works for things that it was intended to do, it is very hard to make it do arbitrary things. If you want to find out what file to download from the BackPAN to get Foo::Bar 0.2, that's too bad, you have to download the BackPAN and index it yourself. Of course, there is no way to index things without evalling code regexed out of every file. (And there is no way to predict if "make install" will actually install Foo::Bar.) Something CPAN couldn't easily do a few years ago was install to arbitrary directories. If you set the right environment variables, EUMM would sort of do the right thing. If you set different environment variables, sometimes MB would do the right thing. Eventually EUMM and MB were patched so that it almost always worked, and then local::lib was written to paper over the differences. And of course, nothing requires the use of EUMM or MB, so if a package doesn't use it, you can't install it to your home directory. Anyway, the EUMM is what you get when you write code to fix problems that people complain about. MB is what you get when you write specs to "fix" problems people complain about. Maybe someday we will have a build system that has a sane design and actually works.
  - marltod 17y ago
    having run into several vague errors from CPAN I agree
    - jrockway 17y ago
      CPAN.pm has some great errors. If it thinks your clock is set wrong (!), it will print a warning message.
  - plinkplonk 17y ago
    "Haskell's Cabal is the technical model to steal." I can't upvote this enough. CABAL is extremely well done.
  - cdavid 17y ago
    Yes, cabal is the thing to steal. Improving distutils is worthwhile, but a temporary stop gap, most fundamental issues cannot be solved from improving distutils. Distutils itself is only 10 000 LOC of bad code, without good API, and you can reuse most setup.py from a conversion step. I have started playing with a cabal-like project. http://github.com/cournape/toydist http://github.com/cournape/toydist It does not do much ATM, except for the conversion step: it can generates a static description of the package from existing setup.py. It reuses distutils to build package, but it does so from the declarative file, meaning that the whole thing is not tied to distutils anymore: distutils becomes an implementation detail.
- jcapote 17y ago
  Maybe python, but rubygems kicks ass.
  - patio11 17y ago
    Google is the de-facto rubygem search engine. If you don't know the real name of the gem, you Google for functionality or common name ([ruby prawn gem]), find a blog post, copy/paste the line that starts with "gem install" (gem install prawn), and then go back to whatever you were doing.
    - r11t 17y ago
      http://gemcutter.org http://gemcutter.org and the search feature of the website might reduce the need of googling for rubygems.
  - relme 17y ago
    Read the source: http://rubygems.rubyforge.org/svn/trunk/lib/ http://rubygems.rubyforge.org/svn/trunk/lib/
  - adw 17y ago
    rubygems and easy_install/.egg packages are more or less the same thing.
    - tvon 17y ago
      Eh, easy_install has been a sort of sticking point with me for a while. One of those things that makes me shake my fist at the sky and leaves me a bit baffled that I seem to be the only one pulling his hair out. So far as I can tell there is no way for easy_install to do anything but install packages. I see an "upgrade" command but it seems to only work if I specify the package name, and I don't see any way to list what version of what is installed. I also don't see any way to remove packages. In short, this means easy_install turns my system into a mystery grab bag of packages that I can't remove without fishing around in site-packages. Unless of course I'm missing something, but if I'm missing something it certainly isn't obvious.
ableal 17y ago
Debian's 'synaptic'. Help and leverage it to other platforms, if needed. Less pain all around.
- jeremymcanally 17y ago
  I sincerely doubt porting that to other platforms would be very easy, but perhaps you and I have different definitions of "pain." :) Plus, you'd have to worry about minor-yet-annoying namespace issues (e.g., a Python package and a non-Python package sharing a name). Am I the only one not offended by the idea of package managers for each programming language? They always work better when they're tailored to the language.
  - viraptor 17y ago
    There are cases when they don't work better. It happens when your non-python program requires something from python for scripting, or when a python module requires a 'classic' library. A global system is quite good in those cases. I'm not sure what you mean by the namespace issues. Everything that's installed as a python package in debian is prefixed with "python-"...
  - ableal 17y ago
    About the porting and the pain: I believe the hard parts are defining the metadata, and refining the actual program logic. In my experience over the last decade, 'synaptic' has been the most trouble-free system to use. I think it would be less work to clone/port the needed logic bits to Windows/whatever, and share most of the metadata defined for Debian/Ubuntu/etc, instead of redoing (and debugging) everything from scratch.
  - lucumo 17y ago
    > Am I the only one not offended by the idea of package managers for each programming language? Learning multiple tools to do a similar job is a bit of a nuisance. If you know one tool, you can learn its details over time. That's much harder with multiple tools. Does tool X remove configuration files when uninstalling? Can it even uninstall? Do I need to update some configuration files manually? There's also the part where you sometimes need integration with packages from other package manager systems. System libraries (libcurl, etc.), header files if it gets compiled at install time, make, a compiler, etc. But since the problem is getting solved by multiple tools already (cpan, pear, apt, yum, just plain ./configure+make, etc.), maybe we should work more on integrating those package managers and less on replacing the others. It seems unlikely to happen with so many incompatible personal preferences around...
amix 17y ago
A few weeks ago I argued that language distribution platforms should be based on a distributed version control system (like Mercurial or Git) and have an user-friendly web-interface (like BitBucket or GitHub). Reputation (like the one found in StacOverflow) should also be a part of the platform. Anyhow, read more here if you are interested: http://amix.dk/blog/viewEntry/19475 http://amix.dk/blog/viewEntry/19475
- steveklabnik 17y ago
  Nice post. I think this is a really cool idea in general. A few of us on one of my projects have been thinking myself recently about a package manager based on git, where you could literally merge in feature branches or patch branches as you wanted...
  - n8agrin 17y ago
    While a nice idea, it would be worth finding out why github abandoned this very concept in favor of gemcutter.org before diving in head first. I trust the github guys as being far more competent than most in all matters of distributed source control especially when it comes to package management.
    - inklesspen 17y ago
      Presumably because gemcutter was doing a better job?
    - pjhyett 17y ago
      The idea is sound, but it was a distraction for us, so we were more than happy to offload rubygems to a dedicated service that could give it the attention it deserves.
marcusbooster 17y ago
I find package systems fantastic for things I don't want to care about it, but get frustrated when it handles the things I do care about. Or maybe that's just my own experience using Common Lisp on Ubuntu.
wavesplash 17y ago
Perhaps I'm late to the game but why not take a good look at the Gem/Gemcutter Ruby packaging and distribution system? The Ruby folks have done a few iterations of packaging systems and have pretty much nailed it. Might be worth studying the whole 'gem' system (discovery, distributed publishing, versioning, dependencies, uninstall, etc). http://www.gemcutter.com http://www.gemcutter.com
- aceofspades19 17y ago
  s/com/org
adw 17y ago
The scientists he's talking about are smart people, but aren't really into computers and (what's more) have no patience at all for the amount of pain it takes to compile the dependencies Python packages need. It's not the Python side of things that's the problem. Numpy and Scipy (which nearly all Python scientific software depend on) is based on both bindings to C and to Fortran, and getting those library ducks in a row - especially on Windows or MacOS X, on Linux your package manager does it for you - is a pain for someone who knows what they're doing and almost impossible for someone who doesn't. Plus, well, most scientists outside physics use Windows. If you need a command-line tool you've already lost in that respect. What you're competing against is often Excel. The battle over Numpy being in core Python has been fought and lost, but short of that level of integration, I don't see that there's going to be much effective to do about this.
- illumen 17y ago
  Numpy is one of the best packaged python modules there is. Saying it is hard to install is completely wrong. It's even already packaged for many OSes. For example, it comes with OSX(and ubuntu, red hat, etc). There's a 3 click installer for windows. There are also distributions of python that have numpy(and many other sci modules) by default.
  - adw 17y ago
    ... and even so, I've seen people struggle to install them - Scipy in particular - countless times. I'm not sure how easy is easy enough, but it's going to have to be basically impossible to screw up. And I'm not sure that's possible short of bundling everything with Python, and of course that's a bad idea.
- cdavid 17y ago
  The problem is more complicated than just C code being more difficult to build. The whole distutils infrastructure is messy and badly designed. It takes care of everything from build up to installation and packaging, and all those parts are tighly coupled. It is also incredibly inflexible, and the way to extend it through subclassing leads to incompatible code (if package A subclass distutils, and package B subclass the same thing, how can you use A and B ?). Almost every design decision of distutils is wrong, and badly implemented. Numpy and scipy binaries are built for every release: actually, that's the platform we support the best in some sense since we can reliably build binaries, and that saddens me quite a bit.
blue1 17y ago
Common Lisp needs something well structured like CPAN too. The situation with clbuild, mudballs (deceased?), asdf-install etc. is rather confusing IMHO.

65 points by fserb 17y ago

Both the CLI CPAN module/tool itself and http://search.cpan.org/ http://search.cpan.org/ are awesome to work with. I think every language should shamelessly copy them first, and try to improve upon them second. It almost seems like Ruby/Python have been consciously reluctant to do so.

jrockway 17y ago
CPAN's success, as with Debian's, is not due to the technical infrastructure but rather the social infrastructure. There are plenty of ways to make packages not work at all, it's just that the Perl people don't do that. (If you have ever read the CPAN source code, you would be surprised it works at all. Don't get me started on the various incompatible build systems, and what happens when your module's build system depends on a newer version of the build system.) Haskell's Cabal is the technical model to steal. Don't let modules execute their own code unless they actually need to. 99.9999% of modules do fine with some sort of declarative interface, rather than actual code to do those things.
- chromatic 17y ago
  The CPAN works despite clunky code and competing implementations because the functional decomposition is sufficiently effective. (EUMM is some of the worst code in intent, design, and implementation I've ever read and used anyway.)
  - jrockway 17y ago
    The CPAN software infrastructure only works for things that it was intended to do, it is very hard to make it do arbitrary things. If you want to find out what file to download from the BackPAN to get Foo::Bar 0.2, that's too bad, you have to download the BackPAN and index it yourself. Of course, there is no way to index things without evalling code regexed out of every file. (And there is no way to predict if "make install" will actually install Foo::Bar.) Something CPAN couldn't easily do a few years ago was install to arbitrary directories. If you set the right environment variables, EUMM would sort of do the right thing. If you set different environment variables, sometimes MB would do the right thing. Eventually EUMM and MB were patched so that it almost always worked, and then local::lib was written to paper over the differences. And of course, nothing requires the use of EUMM or MB, so if a package doesn't use it, you can't install it to your home directory. Anyway, the EUMM is what you get when you write code to fix problems that people complain about. MB is what you get when you write specs to "fix" problems people complain about. Maybe someday we will have a build system that has a sane design and actually works.
- marltod 17y ago
  having run into several vague errors from CPAN I agree
  - jrockway 17y ago
    CPAN.pm has some great errors. If it thinks your clock is set wrong (!), it will print a warning message.
- plinkplonk 17y ago
  "Haskell's Cabal is the technical model to steal." I can't upvote this enough. CABAL is extremely well done.
- cdavid 17y ago
  Yes, cabal is the thing to steal. Improving distutils is worthwhile, but a temporary stop gap, most fundamental issues cannot be solved from improving distutils. Distutils itself is only 10 000 LOC of bad code, without good API, and you can reuse most setup.py from a conversion step. I have started playing with a cabal-like project. http://github.com/cournape/toydist http://github.com/cournape/toydist It does not do much ATM, except for the conversion step: it can generates a static description of the package from existing setup.py. It reuses distutils to build package, but it does so from the declarative file, meaning that the whole thing is not tied to distutils anymore: distutils becomes an implementation detail.
jcapote 17y ago
Maybe python, but rubygems kicks ass.
- patio11 17y ago
  Google is the de-facto rubygem search engine. If you don't know the real name of the gem, you Google for functionality or common name ([ruby prawn gem]), find a blog post, copy/paste the line that starts with "gem install" (gem install prawn), and then go back to whatever you were doing.
  - r11t 17y ago
    http://gemcutter.org http://gemcutter.org and the search feature of the website might reduce the need of googling for rubygems.
- relme 17y ago
  Read the source: http://rubygems.rubyforge.org/svn/trunk/lib/ http://rubygems.rubyforge.org/svn/trunk/lib/
- adw 17y ago
  rubygems and easy_install/.egg packages are more or less the same thing.
  - tvon 17y ago
    Eh, easy_install has been a sort of sticking point with me for a while. One of those things that makes me shake my fist at the sky and leaves me a bit baffled that I seem to be the only one pulling his hair out. So far as I can tell there is no way for easy_install to do anything but install packages. I see an "upgrade" command but it seems to only work if I specify the package name, and I don't see any way to list what version of what is installed. I also don't see any way to remove packages. In short, this means easy_install turns my system into a mystery grab bag of packages that I can't remove without fishing around in site-packages. Unless of course I'm missing something, but if I'm missing something it certainly isn't obvious.

ableal 17y ago

Debian's 'synaptic'. Help and leverage it to other platforms, if needed. Less pain all around.

jeremymcanally 17y ago
I sincerely doubt porting that to other platforms would be very easy, but perhaps you and I have different definitions of "pain." :) Plus, you'd have to worry about minor-yet-annoying namespace issues (e.g., a Python package and a non-Python package sharing a name). Am I the only one not offended by the idea of package managers for each programming language? They always work better when they're tailored to the language.
- viraptor 17y ago
  There are cases when they don't work better. It happens when your non-python program requires something from python for scripting, or when a python module requires a 'classic' library. A global system is quite good in those cases. I'm not sure what you mean by the namespace issues. Everything that's installed as a python package in debian is prefixed with "python-"...
- ableal 17y ago
  About the porting and the pain: I believe the hard parts are defining the metadata, and refining the actual program logic. In my experience over the last decade, 'synaptic' has been the most trouble-free system to use. I think it would be less work to clone/port the needed logic bits to Windows/whatever, and share most of the metadata defined for Debian/Ubuntu/etc, instead of redoing (and debugging) everything from scratch.
- lucumo 17y ago
  > Am I the only one not offended by the idea of package managers for each programming language? Learning multiple tools to do a similar job is a bit of a nuisance. If you know one tool, you can learn its details over time. That's much harder with multiple tools. Does tool X remove configuration files when uninstalling? Can it even uninstall? Do I need to update some configuration files manually? There's also the part where you sometimes need integration with packages from other package manager systems. System libraries (libcurl, etc.), header files if it gets compiled at install time, make, a compiler, etc. But since the problem is getting solved by multiple tools already (cpan, pear, apt, yum, just plain ./configure+make, etc.), maybe we should work more on integrating those package managers and less on replacing the others. It seems unlikely to happen with so many incompatible personal preferences around...

amix 17y ago

A few weeks ago I argued that language distribution platforms should be based on a distributed version control system (like Mercurial or Git) and have an user-friendly web-interface (like BitBucket or GitHub). Reputation (like the one found in StacOverflow) should also be a part of the platform. Anyhow, read more here if you are interested: http://amix.dk/blog/viewEntry/19475 http://amix.dk/blog/viewEntry/19475

steveklabnik 17y ago
Nice post. I think this is a really cool idea in general. A few of us on one of my projects have been thinking myself recently about a package manager based on git, where you could literally merge in feature branches or patch branches as you wanted...
- n8agrin 17y ago
  While a nice idea, it would be worth finding out why github abandoned this very concept in favor of gemcutter.org before diving in head first. I trust the github guys as being far more competent than most in all matters of distributed source control especially when it comes to package management.
  - inklesspen 17y ago
    Presumably because gemcutter was doing a better job?
  - pjhyett 17y ago
    The idea is sound, but it was a distraction for us, so we were more than happy to offload rubygems to a dedicated service that could give it the attention it deserves.

marcusbooster 17y ago

I find package systems fantastic for things I don't want to care about it, but get frustrated when it handles the things I do care about. Or maybe that's just my own experience using Common Lisp on Ubuntu.

wavesplash 17y ago

Perhaps I'm late to the game but why not take a good look at the Gem/Gemcutter Ruby packaging and distribution system? The Ruby folks have done a few iterations of packaging systems and have pretty much nailed it. Might be worth studying the whole 'gem' system (discovery, distributed publishing, versioning, dependencies, uninstall, etc). http://www.gemcutter.com http://www.gemcutter.com

aceofspades19 17y ago
s/com/org

adw 17y ago

The scientists he's talking about are smart people, but aren't really into computers and (what's more) have no patience at all for the amount of pain it takes to compile the dependencies Python packages need. It's not the Python side of things that's the problem. Numpy and Scipy (which nearly all Python scientific software depend on) is based on both bindings to C and to Fortran, and getting those library ducks in a row - especially on Windows or MacOS X, on Linux your package manager does it for you - is a pain for someone who knows what they're doing and almost impossible for someone who doesn't. Plus, well, most scientists outside physics use Windows. If you need a command-line tool you've already lost in that respect. What you're competing against is often Excel. The battle over Numpy being in core Python has been fought and lost, but short of that level of integration, I don't see that there's going to be much effective to do about this.

illumen 17y ago
Numpy is one of the best packaged python modules there is. Saying it is hard to install is completely wrong. It's even already packaged for many OSes. For example, it comes with OSX(and ubuntu, red hat, etc). There's a 3 click installer for windows. There are also distributions of python that have numpy(and many other sci modules) by default.
- adw 17y ago
  ... and even so, I've seen people struggle to install them - Scipy in particular - countless times. I'm not sure how easy is easy enough, but it's going to have to be basically impossible to screw up. And I'm not sure that's possible short of bundling everything with Python, and of course that's a bad idea.
cdavid 17y ago
The problem is more complicated than just C code being more difficult to build. The whole distutils infrastructure is messy and badly designed. It takes care of everything from build up to installation and packaging, and all those parts are tighly coupled. It is also incredibly inflexible, and the way to extend it through subclassing leads to incompatible code (if package A subclass distutils, and package B subclass the same thing, how can you use A and B ?). Almost every design decision of distutils is wrong, and badly implemented. Numpy and scipy binaries are built for every release: actually, that's the platform we support the best in some sense since we can reliably build binaries, and that saddens me quite a bit.

blue1 17y ago

Common Lisp needs something well structured like CPAN too. The situation with clbuild, mudballs (deceased?), asdf-install etc. is rather confusing IMHO.