19 ms·

Python Packaging: Hate, hate, hate everywhere

191 points by uggedal 14y ago

userulluipeste 14y ago
Hate, hate, hate on specific platform, not everywhere! All that trouble that he complains about is only on *nix. For Windows is quite simple. Install Phyton, then look here for other packages: http://www.lfd.uci.edu/~gohlke/pythonlibs/ http://www.lfd.uci.edu/~gohlke/pythonlibs/ Edit: I should have said ”For Windows and ReactOS”
- wesm 14y ago
  Programmatic deployment on Windows is still a massive problem (especially for extension modules-- e.g. the entire scientific Python ecosystem).
  - userulluipeste 14y ago
    Actually I've tried it and got no problem. Have you tried it? Installing packages on NT-kernel OS is different than on *nix, BTW! I guess that's what it was with "programmatic".
    - wesm 14y ago
      Could you explain how Christoph Gohlke's .exe installers solve the programmatic deployment problem (clicking through an .exe installer is not "programmatic deployment")?
      - userulluipeste 14y ago
        Everything can be wrong when measured with a wrong reference. That, BTW, is used in marketing and PR to gain a better perception for one's "different" design. You have to do it "programmatic" on nix because that's how it's done there. Easer alternatives just aren't developed enough because those are't in accordance with the principles in place, thus aren't well received. Some like it, other come to hate it, like this guy did. You don't have to use "programatic deployment" on NT-kernel systems once all you want to do is get the job done. Stop thinking nix all the time.
        calpaterson 14y ago
        Being able to build in one step is a common practice in software engineering everywhere, not just on unix. If you've ever heard the term "Continuous Deployment", then build in one step is being done (and hopefully more besides).
      - BijPnwkkoy 14y ago
        Compared to the effort to build the binaries it is trivial to repackage the content of those installers into EGG, MSI, NSI or other programmatically deployable formats. In fact the Pythonxy and Enthought Python distributions repackage some of those installers into their formats. The exe installers created by Python distutils are the least common distribution/packaging option that works on Python 2.4 to 3.3, 32 and 64 bit. The installers are valid ZIP files and it takes a few lines of command line or Python script to extract the content to any target location. It is the sheer availability of those binaries, and the feedback (incl. patches) to the package authors about Windows build and runtime issues that is the value of the project. It does not solve the Python packaging or deployment problems.
    - andybak 14y ago
      How do those nice .exe files work with virtualenvs? You are using virtualenv, aren't you?
      - userulluipeste 14y ago
        http://www.lfd.uci.edu/~gohlke/pythonlibs/#virtualenv http://www.lfd.uci.edu/~gohlke/pythonlibs/#virtualenv
        andybak 14y ago
        I was asking about whether the installers respect existing virtualenvs or whether they install globally.
  - samwillis 14y ago
    I have used Pip, Virtualenv and virtualenvwrapper-win on windows with very little trouble... except for anything that isn't pure Python. In order to install PIL in a virtualenv on windows I ultimately ended up downloading the PIL windows installer, extracting the required files and placing them in the virtualenvs site packages directory. This is what needs fixing! Pip with access to compiled windows binaries. Install Pip on windows: http://stackoverflow.com/a/4921215 http://stackoverflow.com/a/4921215 Virtualenv Win: https://github.com/davidmarble/virtualenvwrapper-win https://github.com/davidmarble/virtualenvwrapper-win
    - maxerickson 14y ago
      A workaround is to edit the registry to make the virtualenv available as an install target: http://nedbatchelder.com/blog/201007/installing_python_packages_from_windows_installers_into.html http://nedbatchelder.com/blog/201007/installing_python_packa...
      - samwillis 14y ago
        Thanks for the link! That is interesting. I followed though to the SO thread and discovered that you can install windows .exe installers using easy_install into virtualenvs. That is really usefull, its a shame that PyPi doesn't also link to the windows installers so that you wouldn't have to put the path to a windows initialler in. -- Install windows binary packages into virtualenvs with easy_install - http://stackoverflow.com/a/5442340 http://stackoverflow.com/a/5442340
    - megaman821 14y ago
      Inside your virtualenv you can just easy_install pil_installer.exe. Edit: I think I should expand on the limitations of this. You can do this to any exe that was built with distutils, which you can check by changing the file extension to zip and seeing if it opens up.
    - the_mitsuhiko 14y ago
      > This is what needs fixing! Pip with access to compiled windows binaries. That's the same issue really. Pip can't install (by design) binary distributions. Neither eggs nor .exes. So you are limited to easy_install on windows in many cases.
- Joeboy 14y ago
  That's great as long as it's a common package, and you want to install it globally, and you don't care what version you get. But if that's all you want it's easier to type eg. "apt-get install python-reportlab" on *nix.
  - BijPnwkkoy 14y ago
    Among those common packages are pil-py3, numpy-mkl, scipy w/ umfpack, curses, videocapture, orange, twainmodule, simpleitk, pymol, vigra, cellprofiler, cellcognition, pythonmagick, cgkit, scikits.vectorplot, nlopt, slycot, casuarius, ilastik, iocbio, vlfd, mmlib, pysfml2, pywcs.
    - jlgreco 14y ago
      It appears as though in order to get Flask this way, I have to either also get Pylons, googlemaps, etc... or I have to unzip an exe and install it manually. This really does not look pleasant.
      - BijPnwkkoy 14y ago
        The "Base distribution" is a distribution of packages for some specific needs. If you need individual installers for Pylons, googlemaps etc. you need to look elsewhere.
        jlgreco 14y ago
        Yes, that is what I gathered. I mist can't see this being something I would want to use.
- gte910h 14y ago
  I hate package dependencies on windows.
prodigal_erik 14y ago
I would have liked to see a rationale for using virtualenv and all this python-only stuff, when rpm/apt/msi had more mature tools and would have given sysadmins a unified sanely manageable view of what's deployed and their cross-language dependencies.
- slurgfest 14y ago
  I am going to guess you are coming at this entirely from the perspective of a sysadmin who doesn't do a lot with python. virtualenv allows you to easily isolate dependencies so that multiple versions can coexist in different projects without fighting. If you sysadmin significant amounts of Python stuff then you would have already seen that benefit. It also allows you to control PYTHONPATH more elegantly than setting it in every script. This lets you write clean sensible imports rather than using fragile path tricks or wrestling with relative imports. Your project dependencies are often not exactly the same as what the distribution gives. On one hand, distributions can be mind-bogglingly slow about releasing updates to python libraries. On the other hand, you may not want to force your project to use a new update of something because it may introduce bugs. It is pretty essential to have some lead time to port if your app is anything important. So you need control of the versions of your dependencies. So if your project dependencies are not exactly the same as what the distribution uses, then you will need some kind of isolation mechanism in order to allow your project to work as a distribution package. You could do that, but you could also save time and just use virtualenv. Creating platform-specific packages is nontrivial, also it is platform-specific. If I write a library for Python I only want to specify the packaging metadata/scripts once. I am certainly not interested in trying to get it accepted in repos for every distribution.
  - rlpb 14y ago
    > Creating platform-specific packages is nontrivial, also it is platform-specific. Unfortunately, Python-specific packages are Python-specific. I don't want to manage one package manager for Python, one for Ruby, one for Perl and yet another one for system libraries. And I'd prefer to have to trust only one vendor for security updates for everything. And anything that is not Python depends on system libraries, which are supplied by apt or yum and live in an distro-specific namespace. How does setuptools declare and pull in a dependency on a system library? I understand the need for Python, Ruby etc. to have their own packaging for cross-platform use, including on non-Unix. But system packaging tools have their uses too. There is no easy answer, and certainly no One True Answer.
slurgfest 14y ago
Yes, setuptools is broken. Proposal: don't use setuptools! It's funny that the author doesn't make any mention of this, which is not new news: http://docs.python.org/dev/library/packaging.html http://docs.python.org/dev/library/packaging.html Maybe not every use case anyone ever proposed needs to be standardized.
- varikin 14y ago
  That isn't the takeaway I got. It was that Setuptools is broken, but so is pip and distutils2. Pip lost very worthwhile functionality that exists in setuptools, namely binary eggs.
  - slurgfest 14y ago
    Binary eggs aren't a good reason to trash pip, which otherwise works great. He really didn't say how distutils2 was broken, just expressed some sort of prejudice against it.
    - masklinn 14y ago
      > aren't a good reason to trash pip Good thing that never happened then.
      - slurgfest 14y ago
        pip isn't broken just because it doesn't support a particular use case from setuptools.
        masklinn 14y ago
        Again, it would be nice if you stopped making shit up on the spot, TFA didn't say anywhere that pip is "broken" or anything even remotely close to that.
  - the_mitsuhiko 14y ago
    > That isn't the takeaway I got. It was that Setuptools is broken, but so is pip and distutils2. Pip lost very worthwhile functionality that exists in setuptools, namely binary eggs. The takeaway should be that it's very easy to miss usecases by accident when replacing tools with other tools.
edwinnathaniel 14y ago
Despite all the hate, Maven solves a lot of problems in the Java land and I couldn't be more happier.
- gvalkov 14y ago
  There is really no exact equivalent, but you could give buildout[1] a go if you find yourself in need of a maven-like tool in Python land. [1] http://www.buildout.org/ http://www.buildout.org/
  - edwinnathaniel 14y ago
    Sweet, thanks for pointing out that BO is closer to Maven! I've heard a few Python SCM tools (setuptools, distutils, fabric, scons, buildout) but haven't gone deep into the Python land yet.
adambyrtek 14y ago
Another post on the same subject written a few years ago: http://www.b-list.org/weblog/2008/dec/14/packaging/ http://www.b-list.org/weblog/2008/dec/14/packaging/
warmwaffles 14y ago
Most languages are in the same boat. PHP, Java, C, etc...
jnoller 14y ago
sigh If you want to help, check out the Python-dev thread where this is being hashed out ad-infinitum rather than flamed to death: http://mail.python.org/pipermail/python-dev/2012-June/120430.html http://mail.python.org/pipermail/python-dev/2012-June/120430... - for the tl;dr crowd - yeah, things have problems, want to fix.
- gitarr 14y ago
  Yes! Every developer should know one of the very helpful and working ways of doing things in open source software development: "Stop complaining, fix it." It's one thing to complain about closed source software that you have to use and probably paid for as well. But a very different one to complain about free and open source software that you want to use. Stop the bickering, start the work on fixing the problem instead.
  - jnoller 14y ago
    I just want people to know there are avenues to help, and that core developers and many others aren't blind to the mistake, missteps and other things wrong. I'm exhausted by the vitriol; I'd ignore a bug report (and have) that described things as idiotic, and there are avenues for people to help chime in and guide the future. Nothing is unfixable.
    - rfugger 14y ago
      I didn't see any vitriol in the article.
      - the_mitsuhiko 14y ago
        > I didn't see any vitriol in the article. I hope there was not because that was not my intention at all. It's more an observation that replacing things with other things is a dangerous thing because you can easily dismiss parts of the design as “useless” by accident. I had the feeling that happened when ditching setuptools and that's a mistake that problem should not be repeated. I made that mistake in the past already at one point where I tried to replace the Python logging library and I missde perfectly valid usecases in the process.
defen 14y ago
Non-python programmer here. The author refers to "Python's idiotic import system" - can someone explain why it's idiotic? How does it differ from Ruby or Perl (both of which I am familiar with)?
- kingkilr 14y ago
  I'd say the biggest difference between Python and Ruby's is that Python's has a syntax, whereas Ruby's is a function call, this makes Ruby's more flexible since you can always replace it, add new arguments, or in other ways manipulate it. I'm not sure if that's what Armin is referring to though.
  - mrud 14y ago
    JFTR python also has a function call to import modules http://docs.python.org/library/functions.html#__import__ http://docs.python.org/library/functions.html#__import__
  - lloeki 14y ago
    > I'd say the biggest difference between Python and Ruby's is that Python's has a syntax, whereas Ruby's is a function call This is not true on two fronts. First, the import statement is a convenient, hookable wrapper around the __import__ function. It's quite customizable already. See PEP302 which references and allows implementation of PEP273 (importing modules from Zip archives). Other points of reference include [0], [1] and [2] Also, the biggest difference is that when you 'require', you require a filename (i.e the arbitrary content of a file in the filesystem), but when you 'import', you import objects from a namespace into a scope. The namespace is resolved as an item living in the filesystem. In Ruby I could require 'foo/bar' and it could create the constants Foo::Bar and Qux::Quux. I have no way to require Qux::Quux. Rails for example tries to autoload constants based on their names, but for all I know, the second I reference Foo::Bar, it could actually define Qux::Quux. Ruby encourages this style of programming, spreading extendable classes and modules around in multiple files, while Python decides that things should be contained and well-behaving, and not trivially pollute unrelated module namespaces. My analysis is that by definition you just can't have both, and that either one has a set of benefits and drawbacks. [0] http://docs.python.org/library/importlib.html#importlib.import_module http://docs.python.org/library/importlib.html#importlib.impo... [1] http://docs.python.org/library/imputil.html http://docs.python.org/library/imputil.html [2] http://docs.python.org/py3k/library/importlib.html?highlight=importlib#importlib http://docs.python.org/py3k/library/importlib.html?highlight...
socratic 14y ago
Does anyone know if namespace packages actually work? I've been wanting to put up a few utility libraries on Github. I don't think they're good enough for PyPI, and I mostly just want to be able to pip install them and have access to a few minor functions that I use periodically. However, I don't think they deserve their own top level library name, ideally, they'd all be under "socratic.*" or similar. However, there seems to be some sort of mess with init in the top level of the package, some pkgutil fix, some sort of incomplete PEP (maybe 402?), etc. Should I just give up and name my internal libraries "socratic_{name}" and be done with it? Or does having multiple packages with the same namespace work?
- the_mitsuhiko 14y ago
  > Does anyone know if namespace packages actually work? They used to work somewhat. I had flaskext.* registered as a namespace package but unfortunately it conflicts with pip which is why new Flask extensions name their package `flask_bar` instead of `flaskext.bar`. The exact problem is that setuptools uses pth magic to put libraries into a namespace package which conflicts with pip's idea of installing packages flat into site-packages.
- zzzeek 14y ago
  it has some rough edges, sometimes when I try to build the docs for one it has a hard time importing both packages until I kick it a few times, but they work. I'm doing it for these two modules: http://pypi.python.org/pypi/dogpile.core/ http://pypi.python.org/pypi/dogpile.core/ http://pypi.python.org/pypi/dogpile.cache/ http://pypi.python.org/pypi/dogpile.cache/ edit: also they work fine with pip so not sure what armin's issue here is.
  - the_mitsuhiko 14y ago
    > edit: also they work fine with pip so not sure what armin's issue here is. See this issue which ultimately made me ditch them: https://github.com/pypa/pip/issues/3 https://github.com/pypa/pip/issues/3
- ianb 14y ago
  It gets bad if you use pip and easy_install with namespace packages. If you stick to one or the other you should generally be okay. But yes, just use socratic_{name} – it makes everything easier, and "." and "_" are just string differences. The only good reason IMHO for using namespace packages is because you are breaking up an existing package into multiple packages and you want to keep the dotted names. And even then I might just prefer a compatibility package that does the mapping and move things to entirely new package names.
dice 14y ago
As a sysadmin pip and its ilk annoy me as well, although for a reason which is not mentioned in the article: It creates an entire package management system which is not the distribution's package manager. Ruby's gems tick me off for the same reason. As a sysadmin you need to decide to either manage Python and Ruby entirely outside of the OS's native package management or try and wrap every single Python egg and Ruby gem in an RPM/DPKG/whatever. Mixing the management between two packaging systems is just going to cause trouble. There's a similar problem with Perl's CPAN, which is what I think all of these projects are aiming to emulate, but the nice thing about CPAN is that repositories like rpmforge or EPEL already have a large number of commonly used CPAN packages all wrapped up in RPMs already. It would be nice to see similar community efforts centered around Python and Ruby packages. Perhaps one day I will have enough free time to start one.
- SourPatch 14y ago
  I don't agree with that approach. It sounds nice for a sysadmin, but as a programmer I don't want to have to wait for you or the distributor to approve and package the libs I need to work with.
  - dice 14y ago
    >I don't agree with that approach. It sounds nice for a sysadmin, but as a programmer I don't want to have to wait for you or the distributor to approve and package the libs I need to work with. Which is why development, QA, staging and production are all separate environments. Do what you want in your dev environment: when the code is ready for testing let the ops team know what the dependencies are. We'll take care of wrapping everything in packages and updating chef/puppet/whatever to deploy it.
- quanticle 14y ago
  The answer, as far as I can tell, is that it makes it easier for Python (and Ruby and Perl) developers to share code and stay on the "latest and greatest" versions of their libraries. I mean, as a library author, what would you rather do? Just create a single egg, gem, CPAN package, or whatever? Or would you rather create three or four versions of the same package (one for Debian-based systems, one for Fedora-based systems, one for Gentoo, one for Arch, etc.)? I understand that it can be frustrating from a sysadmin perspective (pip, especially, isn't sysadmin friendly at all), but from the perspective of a library author, eggs and gems are much preferable to debs and rpms because I know that they'll work in a reasonably distribution independent fashion. EDIT: I know that's not a satisfying answer. But, as far as I can tell, it is the primary reason that runtimes like Python and Ruby include their own package management rather than falling back on the package management that the OS provides.
smagch 14y ago
I'm a python-learning guy who uses npm and homebrew. Comparing to npm and homebrew, it seems to me python's package manager is quite tricky.
berntb 14y ago
Why doesn't the Python world just copy CPAN and update it for Python? You can e.g. literally find stuff in CPAN with hundreds of dependencies -- which is a pain, but even that works. At least CPAN testers? Please... If Python people insist that there is only one way to do it -- and the answer is Python, then rewrite CPAN in Python. Copy stuff that works...
- slurgfest 14y ago
  "Only one way to do it" is a point of language design, not an assertion that Python is the one true way. Python does have PyPI. Works fine. Use pip to install and uninstall packages. Use virtualenv to isolate dependencies/PYTHONPATH, as needed. Can you be specific about what aspect of CPAN you think should be ported?
  - berntb 14y ago
    Specific? I already mentioned CPAN Testers. And that CPAN works... (I've seen people with years of Python experience that had problem doing simple things. Edit: Including uninstall of packages, which you mention.) Edit: Here you see a result of CPAN Testers for a module with complex dependencies. Note statistics on where/how dependencies are run. Also click on a dependency icon to see the matrix of perl/os versions. http://deps.cpantesters.org/?module=Catalyst;perl=latest http://deps.cpantesters.org/?module=Catalyst;perl=latest
    - slurgfest 14y ago
      I don't have any problem doing uninstall of packages. Use pip rather than easy_install. You are done. I'm not convinced that CPAN is better
      - berntb 14y ago
        Good for you, re uninstall... (I really didn't expect Python users to admit that anything in any other environment could be better... :-) E.g. running tests at install by default is obviously bad? :-) ) So no comment about CPAN Testers -- automatically installing dependencies, running the test suites on different perl/os versions and then getting neat reports? What is the better way in Python?
vishaldpatel 14y ago
Title of the post reminds me of the Player Hater's Ball.
danbmil99 14y ago
It's all baby stuff compared to autotools.
aaronh 14y ago
Ugh...python packaging, it makes my head hurt every time I have to figure out how it works again. This is something that Ruby got very right, how could Python be such a mess? I wish it was just a simple 3-step process to fix http://www.youtube.com/watch?v=HR_5QFZ-kbk http://www.youtube.com/watch?v=HR_5QFZ-kbk
agravier 14y ago
I'm using bento, which keeps backward compatibility with setuptools/distribute, but aims at fixing the traditional Python packaging tools: http://cournape.github.com/Bento/ http://cournape.github.com/Bento/ It is stable and usable.

191 points by uggedal 14y ago

Hate, hate, hate on specific platform, not everywhere! All that trouble that he complains about is only on *nix. For Windows is quite simple. Install Phyton, then look here for other packages: http://www.lfd.uci.edu/~gohlke/pythonlibs/ http://www.lfd.uci.edu/~gohlke/pythonlibs/ Edit: I should have said ”For Windows and ReactOS”

wesm 14y ago
Programmatic deployment on Windows is still a massive problem (especially for extension modules-- e.g. the entire scientific Python ecosystem).
- userulluipeste 14y ago
  Actually I've tried it and got no problem. Have you tried it? Installing packages on NT-kernel OS is different than on *nix, BTW! I guess that's what it was with "programmatic".
  - wesm 14y ago
    Could you explain how Christoph Gohlke's .exe installers solve the programmatic deployment problem (clicking through an .exe installer is not "programmatic deployment")?
    - userulluipeste 14y ago
      Everything can be wrong when measured with a wrong reference. That, BTW, is used in marketing and PR to gain a better perception for one's "different" design. You have to do it "programmatic" on nix because that's how it's done there. Easer alternatives just aren't developed enough because those are't in accordance with the principles in place, thus aren't well received. Some like it, other come to hate it, like this guy did. You don't have to use "programatic deployment" on NT-kernel systems once all you want to do is get the job done. Stop thinking nix all the time.
      - calpaterson 14y ago
        Being able to build in one step is a common practice in software engineering everywhere, not just on unix. If you've ever heard the term "Continuous Deployment", then build in one step is being done (and hopefully more besides).
    - BijPnwkkoy 14y ago
      Compared to the effort to build the binaries it is trivial to repackage the content of those installers into EGG, MSI, NSI or other programmatically deployable formats. In fact the Pythonxy and Enthought Python distributions repackage some of those installers into their formats. The exe installers created by Python distutils are the least common distribution/packaging option that works on Python 2.4 to 3.3, 32 and 64 bit. The installers are valid ZIP files and it takes a few lines of command line or Python script to extract the content to any target location. It is the sheer availability of those binaries, and the feedback (incl. patches) to the package authors about Windows build and runtime issues that is the value of the project. It does not solve the Python packaging or deployment problems.
  - andybak 14y ago
    How do those nice .exe files work with virtualenvs? You are using virtualenv, aren't you?
    - userulluipeste 14y ago
      http://www.lfd.uci.edu/~gohlke/pythonlibs/#virtualenv http://www.lfd.uci.edu/~gohlke/pythonlibs/#virtualenv
      - andybak 14y ago
        I was asking about whether the installers respect existing virtualenvs or whether they install globally.
- samwillis 14y ago
  I have used Pip, Virtualenv and virtualenvwrapper-win on windows with very little trouble... except for anything that isn't pure Python. In order to install PIL in a virtualenv on windows I ultimately ended up downloading the PIL windows installer, extracting the required files and placing them in the virtualenvs site packages directory. This is what needs fixing! Pip with access to compiled windows binaries. Install Pip on windows: http://stackoverflow.com/a/4921215 http://stackoverflow.com/a/4921215 Virtualenv Win: https://github.com/davidmarble/virtualenvwrapper-win https://github.com/davidmarble/virtualenvwrapper-win
  - maxerickson 14y ago
    A workaround is to edit the registry to make the virtualenv available as an install target: http://nedbatchelder.com/blog/201007/installing_python_packages_from_windows_installers_into.html http://nedbatchelder.com/blog/201007/installing_python_packa...
    - samwillis 14y ago
      Thanks for the link! That is interesting. I followed though to the SO thread and discovered that you can install windows .exe installers using easy_install into virtualenvs. That is really usefull, its a shame that PyPi doesn't also link to the windows installers so that you wouldn't have to put the path to a windows initialler in. -- Install windows binary packages into virtualenvs with easy_install - http://stackoverflow.com/a/5442340 http://stackoverflow.com/a/5442340
  - megaman821 14y ago
    Inside your virtualenv you can just easy_install pil_installer.exe. Edit: I think I should expand on the limitations of this. You can do this to any exe that was built with distutils, which you can check by changing the file extension to zip and seeing if it opens up.
  - the_mitsuhiko 14y ago
    > This is what needs fixing! Pip with access to compiled windows binaries. That's the same issue really. Pip can't install (by design) binary distributions. Neither eggs nor .exes. So you are limited to easy_install on windows in many cases.
Joeboy 14y ago
That's great as long as it's a common package, and you want to install it globally, and you don't care what version you get. But if that's all you want it's easier to type eg. "apt-get install python-reportlab" on *nix.
- BijPnwkkoy 14y ago
  Among those common packages are pil-py3, numpy-mkl, scipy w/ umfpack, curses, videocapture, orange, twainmodule, simpleitk, pymol, vigra, cellprofiler, cellcognition, pythonmagick, cgkit, scikits.vectorplot, nlopt, slycot, casuarius, ilastik, iocbio, vlfd, mmlib, pysfml2, pywcs.
  - jlgreco 14y ago
    It appears as though in order to get Flask this way, I have to either also get Pylons, googlemaps, etc... or I have to unzip an exe and install it manually. This really does not look pleasant.
    - BijPnwkkoy 14y ago
      The "Base distribution" is a distribution of packages for some specific needs. If you need individual installers for Pylons, googlemaps etc. you need to look elsewhere.
      - jlgreco 14y ago
        Yes, that is what I gathered. I mist can't see this being something I would want to use.
gte910h 14y ago
I hate package dependencies on windows.

prodigal_erik 14y ago

I would have liked to see a rationale for using virtualenv and all this python-only stuff, when rpm/apt/msi had more mature tools and would have given sysadmins a unified sanely manageable view of what's deployed and their cross-language dependencies.

slurgfest 14y ago
I am going to guess you are coming at this entirely from the perspective of a sysadmin who doesn't do a lot with python. virtualenv allows you to easily isolate dependencies so that multiple versions can coexist in different projects without fighting. If you sysadmin significant amounts of Python stuff then you would have already seen that benefit. It also allows you to control PYTHONPATH more elegantly than setting it in every script. This lets you write clean sensible imports rather than using fragile path tricks or wrestling with relative imports. Your project dependencies are often not exactly the same as what the distribution gives. On one hand, distributions can be mind-bogglingly slow about releasing updates to python libraries. On the other hand, you may not want to force your project to use a new update of something because it may introduce bugs. It is pretty essential to have some lead time to port if your app is anything important. So you need control of the versions of your dependencies. So if your project dependencies are not exactly the same as what the distribution uses, then you will need some kind of isolation mechanism in order to allow your project to work as a distribution package. You could do that, but you could also save time and just use virtualenv. Creating platform-specific packages is nontrivial, also it is platform-specific. If I write a library for Python I only want to specify the packaging metadata/scripts once. I am certainly not interested in trying to get it accepted in repos for every distribution.
- rlpb 14y ago
  > Creating platform-specific packages is nontrivial, also it is platform-specific. Unfortunately, Python-specific packages are Python-specific. I don't want to manage one package manager for Python, one for Ruby, one for Perl and yet another one for system libraries. And I'd prefer to have to trust only one vendor for security updates for everything. And anything that is not Python depends on system libraries, which are supplied by apt or yum and live in an distro-specific namespace. How does setuptools declare and pull in a dependency on a system library? I understand the need for Python, Ruby etc. to have their own packaging for cross-platform use, including on non-Unix. But system packaging tools have their uses too. There is no easy answer, and certainly no One True Answer.

slurgfest 14y ago

Yes, setuptools is broken. Proposal: don't use setuptools! It's funny that the author doesn't make any mention of this, which is not new news: http://docs.python.org/dev/library/packaging.html http://docs.python.org/dev/library/packaging.html Maybe not every use case anyone ever proposed needs to be standardized.

varikin 14y ago
That isn't the takeaway I got. It was that Setuptools is broken, but so is pip and distutils2. Pip lost very worthwhile functionality that exists in setuptools, namely binary eggs.
- slurgfest 14y ago
  Binary eggs aren't a good reason to trash pip, which otherwise works great. He really didn't say how distutils2 was broken, just expressed some sort of prejudice against it.
  - masklinn 14y ago
    > aren't a good reason to trash pip Good thing that never happened then.
    - slurgfest 14y ago
      pip isn't broken just because it doesn't support a particular use case from setuptools.
      - masklinn 14y ago
        Again, it would be nice if you stopped making shit up on the spot, TFA didn't say anywhere that pip is "broken" or anything even remotely close to that.
- the_mitsuhiko 14y ago
  > That isn't the takeaway I got. It was that Setuptools is broken, but so is pip and distutils2. Pip lost very worthwhile functionality that exists in setuptools, namely binary eggs. The takeaway should be that it's very easy to miss usecases by accident when replacing tools with other tools.

edwinnathaniel 14y ago

Despite all the hate, Maven solves a lot of problems in the Java land and I couldn't be more happier.

gvalkov 14y ago
There is really no exact equivalent, but you could give buildout[1] a go if you find yourself in need of a maven-like tool in Python land. [1] http://www.buildout.org/ http://www.buildout.org/
- edwinnathaniel 14y ago
  Sweet, thanks for pointing out that BO is closer to Maven! I've heard a few Python SCM tools (setuptools, distutils, fabric, scons, buildout) but haven't gone deep into the Python land yet.

adambyrtek 14y ago

Another post on the same subject written a few years ago: http://www.b-list.org/weblog/2008/dec/14/packaging/ http://www.b-list.org/weblog/2008/dec/14/packaging/

warmwaffles 14y ago

Most languages are in the same boat. PHP, Java, C, etc...

jnoller 14y ago

sigh If you want to help, check out the Python-dev thread where this is being hashed out ad-infinitum rather than flamed to death: http://mail.python.org/pipermail/python-dev/2012-June/120430.html http://mail.python.org/pipermail/python-dev/2012-June/120430... - for the tl;dr crowd - yeah, things have problems, want to fix.

gitarr 14y ago
Yes! Every developer should know one of the very helpful and working ways of doing things in open source software development: "Stop complaining, fix it." It's one thing to complain about closed source software that you have to use and probably paid for as well. But a very different one to complain about free and open source software that you want to use. Stop the bickering, start the work on fixing the problem instead.
- jnoller 14y ago
  I just want people to know there are avenues to help, and that core developers and many others aren't blind to the mistake, missteps and other things wrong. I'm exhausted by the vitriol; I'd ignore a bug report (and have) that described things as idiotic, and there are avenues for people to help chime in and guide the future. Nothing is unfixable.
  - rfugger 14y ago
    I didn't see any vitriol in the article.
    - the_mitsuhiko 14y ago
      > I didn't see any vitriol in the article. I hope there was not because that was not my intention at all. It's more an observation that replacing things with other things is a dangerous thing because you can easily dismiss parts of the design as “useless” by accident. I had the feeling that happened when ditching setuptools and that's a mistake that problem should not be repeated. I made that mistake in the past already at one point where I tried to replace the Python logging library and I missde perfectly valid usecases in the process.

defen 14y ago

Non-python programmer here. The author refers to "Python's idiotic import system" - can someone explain why it's idiotic? How does it differ from Ruby or Perl (both of which I am familiar with)?

kingkilr 14y ago
I'd say the biggest difference between Python and Ruby's is that Python's has a syntax, whereas Ruby's is a function call, this makes Ruby's more flexible since you can always replace it, add new arguments, or in other ways manipulate it. I'm not sure if that's what Armin is referring to though.
- mrud 14y ago
  JFTR python also has a function call to import modules http://docs.python.org/library/functions.html#__import__ http://docs.python.org/library/functions.html#__import__
- lloeki 14y ago
  > I'd say the biggest difference between Python and Ruby's is that Python's has a syntax, whereas Ruby's is a function call This is not true on two fronts. First, the import statement is a convenient, hookable wrapper around the __import__ function. It's quite customizable already. See PEP302 which references and allows implementation of PEP273 (importing modules from Zip archives). Other points of reference include [0], [1] and [2] Also, the biggest difference is that when you 'require', you require a filename (i.e the arbitrary content of a file in the filesystem), but when you 'import', you import objects from a namespace into a scope. The namespace is resolved as an item living in the filesystem. In Ruby I could require 'foo/bar' and it could create the constants Foo::Bar and Qux::Quux. I have no way to require Qux::Quux. Rails for example tries to autoload constants based on their names, but for all I know, the second I reference Foo::Bar, it could actually define Qux::Quux. Ruby encourages this style of programming, spreading extendable classes and modules around in multiple files, while Python decides that things should be contained and well-behaving, and not trivially pollute unrelated module namespaces. My analysis is that by definition you just can't have both, and that either one has a set of benefits and drawbacks. [0] http://docs.python.org/library/importlib.html#importlib.import_module http://docs.python.org/library/importlib.html#importlib.impo... [1] http://docs.python.org/library/imputil.html http://docs.python.org/library/imputil.html [2] http://docs.python.org/py3k/library/importlib.html?highlight=importlib#importlib http://docs.python.org/py3k/library/importlib.html?highlight...

socratic 14y ago

Does anyone know if namespace packages actually work? I've been wanting to put up a few utility libraries on Github. I don't think they're good enough for PyPI, and I mostly just want to be able to pip install them and have access to a few minor functions that I use periodically. However, I don't think they deserve their own top level library name, ideally, they'd all be under "socratic.*" or similar. However, there seems to be some sort of mess with init in the top level of the package, some pkgutil fix, some sort of incomplete PEP (maybe 402?), etc. Should I just give up and name my internal libraries "socratic_{name}" and be done with it? Or does having multiple packages with the same namespace work?

the_mitsuhiko 14y ago
> Does anyone know if namespace packages actually work? They used to work somewhat. I had flaskext.* registered as a namespace package but unfortunately it conflicts with pip which is why new Flask extensions name their package `flask_bar` instead of `flaskext.bar`. The exact problem is that setuptools uses pth magic to put libraries into a namespace package which conflicts with pip's idea of installing packages flat into site-packages.
zzzeek 14y ago
it has some rough edges, sometimes when I try to build the docs for one it has a hard time importing both packages until I kick it a few times, but they work. I'm doing it for these two modules: http://pypi.python.org/pypi/dogpile.core/ http://pypi.python.org/pypi/dogpile.core/ http://pypi.python.org/pypi/dogpile.cache/ http://pypi.python.org/pypi/dogpile.cache/ edit: also they work fine with pip so not sure what armin's issue here is.
- the_mitsuhiko 14y ago
  > edit: also they work fine with pip so not sure what armin's issue here is. See this issue which ultimately made me ditch them: https://github.com/pypa/pip/issues/3 https://github.com/pypa/pip/issues/3
ianb 14y ago
It gets bad if you use pip and easy_install with namespace packages. If you stick to one or the other you should generally be okay. But yes, just use socratic_{name} – it makes everything easier, and "." and "_" are just string differences. The only good reason IMHO for using namespace packages is because you are breaking up an existing package into multiple packages and you want to keep the dotted names. And even then I might just prefer a compatibility package that does the mapping and move things to entirely new package names.

dice 14y ago

As a sysadmin pip and its ilk annoy me as well, although for a reason which is not mentioned in the article: It creates an entire package management system which is not the distribution's package manager. Ruby's gems tick me off for the same reason. As a sysadmin you need to decide to either manage Python and Ruby entirely outside of the OS's native package management or try and wrap every single Python egg and Ruby gem in an RPM/DPKG/whatever. Mixing the management between two packaging systems is just going to cause trouble. There's a similar problem with Perl's CPAN, which is what I think all of these projects are aiming to emulate, but the nice thing about CPAN is that repositories like rpmforge or EPEL already have a large number of commonly used CPAN packages all wrapped up in RPMs already. It would be nice to see similar community efforts centered around Python and Ruby packages. Perhaps one day I will have enough free time to start one.

SourPatch 14y ago
I don't agree with that approach. It sounds nice for a sysadmin, but as a programmer I don't want to have to wait for you or the distributor to approve and package the libs I need to work with.
- dice 14y ago
  >I don't agree with that approach. It sounds nice for a sysadmin, but as a programmer I don't want to have to wait for you or the distributor to approve and package the libs I need to work with. Which is why development, QA, staging and production are all separate environments. Do what you want in your dev environment: when the code is ready for testing let the ops team know what the dependencies are. We'll take care of wrapping everything in packages and updating chef/puppet/whatever to deploy it.
quanticle 14y ago
The answer, as far as I can tell, is that it makes it easier for Python (and Ruby and Perl) developers to share code and stay on the "latest and greatest" versions of their libraries. I mean, as a library author, what would you rather do? Just create a single egg, gem, CPAN package, or whatever? Or would you rather create three or four versions of the same package (one for Debian-based systems, one for Fedora-based systems, one for Gentoo, one for Arch, etc.)? I understand that it can be frustrating from a sysadmin perspective (pip, especially, isn't sysadmin friendly at all), but from the perspective of a library author, eggs and gems are much preferable to debs and rpms because I know that they'll work in a reasonably distribution independent fashion. EDIT: I know that's not a satisfying answer. But, as far as I can tell, it is the primary reason that runtimes like Python and Ruby include their own package management rather than falling back on the package management that the OS provides.

smagch 14y ago

I'm a python-learning guy who uses npm and homebrew. Comparing to npm and homebrew, it seems to me python's package manager is quite tricky.

berntb 14y ago

Why doesn't the Python world just copy CPAN and update it for Python? You can e.g. literally find stuff in CPAN with hundreds of dependencies -- which is a pain, but even that works. At least CPAN testers? Please... If Python people insist that there is only one way to do it -- and the answer is Python, then rewrite CPAN in Python. Copy stuff that works...

slurgfest 14y ago
"Only one way to do it" is a point of language design, not an assertion that Python is the one true way. Python does have PyPI. Works fine. Use pip to install and uninstall packages. Use virtualenv to isolate dependencies/PYTHONPATH, as needed. Can you be specific about what aspect of CPAN you think should be ported?
- berntb 14y ago
  Specific? I already mentioned CPAN Testers. And that CPAN works... (I've seen people with years of Python experience that had problem doing simple things. Edit: Including uninstall of packages, which you mention.) Edit: Here you see a result of CPAN Testers for a module with complex dependencies. Note statistics on where/how dependencies are run. Also click on a dependency icon to see the matrix of perl/os versions. http://deps.cpantesters.org/?module=Catalyst;perl=latest http://deps.cpantesters.org/?module=Catalyst;perl=latest
  - slurgfest 14y ago
    I don't have any problem doing uninstall of packages. Use pip rather than easy_install. You are done. I'm not convinced that CPAN is better
    - berntb 14y ago
      Good for you, re uninstall... (I really didn't expect Python users to admit that anything in any other environment could be better... :-) E.g. running tests at install by default is obviously bad? :-) ) So no comment about CPAN Testers -- automatically installing dependencies, running the test suites on different perl/os versions and then getting neat reports? What is the better way in Python?

vishaldpatel 14y ago

Title of the post reminds me of the Player Hater's Ball.

danbmil99 14y ago

It's all baby stuff compared to autotools.

aaronh 14y ago

Ugh...python packaging, it makes my head hurt every time I have to figure out how it works again. This is something that Ruby got very right, how could Python be such a mess? I wish it was just a simple 3-step process to fix http://www.youtube.com/watch?v=HR_5QFZ-kbk http://www.youtube.com/watch?v=HR_5QFZ-kbk

agravier 14y ago

I'm using bento, which keeps backward compatibility with setuptools/distribute, but aims at fixing the traditional Python packaging tools: http://cournape.github.com/Bento/ http://cournape.github.com/Bento/ It is stable and usable.