9 ms·

Musings about Debian and Python

108 points by plessthanpt05 13y ago

TheSwordsman 13y ago
I think I agree with the blog post in its entirety, as a fellow Pythonista / Debianite. The Debian packages that are released always seem to be top-notch, stable, and fit for production. (Let's forget the OpenSSL incident, shall we?) I don't fear that the new version is going to break my currently installed software, and I'm not worried about dependencies somehow having been missed causing havoc. It just works. (And when it doesn't, it's because I did something deep within dpkg/apt I should not have.) I can't say I've always had the same experience with other distributions which is the reason why I moved to Debian in the first place. Truthfully, I can't ramble any specific scenarios off from the top of my head. Any time I've had an issue with a Debian software package, the bug threads have always been constructive with proponents for both sides explaining why it should be one way over the other. Eventually, the best decision is made (even if I personally disagree). However, the sacrifice for this stability is the fact packages can become a bit 'stale' when it comes to new versions. I don't mind sacrifices like this. And if you need newer stuff, that's why backports exist. If you want the latest and the greatest use pip. But for the love of all things, couple your pip usage with a virtual environment. Hell, even if you aren't using pip get in the habit of using a venv. My only bad experience with virtualenvs was a recent Python security update (related to /dev/random). The system libraries changed, the Python executable in the venv did not, sadness ensued. Even then, once I figured out the issue, it was a quick fix. Just re-init the virtualenv, and it fixed it for me. No need to move code around. In short, if I want a stable system I don't need to babysit, I go with Debian. I trust the people maintaining the packages. I use virtualenvs for 90% of my Python development, and use pip inside of those venvs. Hasn't failed me, yet. Probably shouldn't jinx myself...
zdw 13y ago
More background on Python's packaging methods and history: http://aosabook.org/en/packaging.html http://aosabook.org/en/packaging.html I'd also add that native packages tend to be far more repeatable when you have to deploy and update a lot of systems in any environment. Note that with tools like fpm or other similar quick and dirty package generation tools, in general don't have to stick to the upstream system's development guidelines, but you get the benefits of using the native package tools.
don_draper 13y ago
"I prefer virtualenv based setups for development" I'm a casual python user and I find virtualenv to be very helpful.
st3fan 13y ago
Or do what I did: give up on Python packaging and use Go or a JVM based language that produce single artifact deployables. I have been in this Python mess far too long. Now I just make a .war file or tell Go to compile for my target platform. Both result in single files that can be deployed without post-processing, installing/compiling dependencies or whatever ... it simplifies things for everyone. Developers, CI, packagers, ops, security folks.
- deleted 13y ago
  [deleted]
- the_mitsuhiko 13y ago
  > or tell Go to compile for my target platform JFTR: the absolutely same is possible in Python.
  - mattip 13y ago
    what is the py2exe equivalent for linux /OSX ?
    - oinksoft 13y ago
      py2pp for the Mac. It appears there is a py2deb and py2rpm for Linux, and I imagine users of Arch, Slack, etc. would be expected to download a .tgz and `setup.py install` it.
      - ginko 13y ago
        > I imagine users of Arch, Slack, etc. would be expected to download a .tgz and `setup.py install` it. In Arch you would wrap this in a PKGBUILD script to generate a Pacman package. You can find PKGBUILD's for most things in the AUR.
        oinksoft 13y ago
        Ah, of course, you would want to create a slackpkg for Slack, and push to the AUR for Arch. It's still good to offer the .tgz as a baseline for users of systems you might have forgotten or not have time to package for though, (BSD, etc.).
ekr 13y ago
I don't want to start any argument, but there is a reason why Stallman recommends writing GNU packages in C (so that users don't need to acquire and install all kinds of dependencies, and not just them). I certainly think again about installing something when I see that the software is written in Java or Python, mostly because I don't like to bloat my system.
- gaius 13y ago
  Realistically, Python is as ubiquitous as Perl these days.
- pekk 13y ago
  What actual concrete problem do you have with 'bloat' due to languages such as perl?
- null_ptr 13y ago
  So many Ruby/Python programs are dependent on several frameworks or libraries, nothing seems to be self-contained anymore. That's putting me off, too, however I think that's in part because of my outdated expectations (as much as I hate to admit it). Things are trending towards user programs that focus exclusively on high-level problem solving and delegate all building blocks to 3rd party modules, and these modules may in turn do the same.
Siecje 13y ago
virtualenv and pip for your own project and use the package to satisfy requirements of installed programs.
fingerprinter 13y ago
I didn't realize there was a fight going on. It seems like this is where everyone settled and is a good solution: Virtualenv + pip for development system wide (apt, dpkg) python etc for system tools and packages. This makes sense to me. Keep your personal code sandboxed from your system tools. As an aside, I know there are quite a few tools and programs in both Debian and Ubuntu written in Python. I wonder if we'll look back in 5-10 years and realize/think this was a mistake. Particularly for UI code. I love me some Python, but I cringe every time I have to interact with a Python GUI application.
- idank 13y ago
  And what do you do if the dpkg'd version is a year older than the one you'd like to use and is available in pypi? Install just that one via pip? But what if that package has dependencies newer than what's installable with apt? In the end you find yourself installing as much as possible with pip, and resort to apt for the really heavy stuff like numpy.
- justincormack 13y ago
  And what for production? A lot of the argument seems to be around the dev/ops divide.
  - fingerprinter 13y ago
    There's probably a bunch of different ways to do this, but I just use virtualenv + pip still. Here's some documentation on Django (as an example): https://docs.djangoproject.com/en/dev/howto/deployment/wsgi/modwsgi/#using-a-virtualenv https://docs.djangoproject.com/en/dev/howto/deployment/wsgi/... I think a more interesting question is not around the python side of things, but the infrastructure. Do you compile all your infra tools from source or do you use the system packages? Like, if you want to use RabbitMQ, if debian/Ubuntu provides it, would you use that or grab the source? I personally prefer to use the debian/ubuntu ones in these cases unless they are horribly outdated, which isn't generally the case. I can understand, however, if someone thinks this is a terrible idea.
- nandhp 13y ago
emillon 13y ago
The background of this post is a thread on the debian-python mailing list, relative to a new PEP: http://lists.debian.org/debian-python/2013/09/msg00049.html http://lists.debian.org/debian-python/2013/09/msg00049.html
pedrocr 13y ago
Where I like to draw the line between apt and rubygems/pip is between "things that are base system" and "things that are app deployment". So in a typical rails stack that would be apt installs the system-wide stack (apache, ruby, passenger, rubygems) and then capistrano is used to deploy my webapps with their gems from my dev machine to their app-specific directory in the production server. This way I can have my sysadmin hat on while setting up the server and depend on debian/ubuntu to handle security upgrades and generally create a consistent system. Then I can put my devops hat on and use capistrano and bundler to manage the security/dependencies of my own code. But I see where this breaks down. If the base stack is moving at a much faster pace than the distributions (e.g., right now the version of passenger in Ubuntu LTS is incredibly old) it's attractive to just ignore the system packages and install everything from original sources (e.g., install ruby from source with rbenv). But doing that is just throwing away the integration work the distribution has done. I'd much rather include some extra repositories to get updated versions that integrate through apt for the few things that I care to upgrade faster than Ubuntu LTS allows me. Right now that's puppet, passenger and a few more.
- rlpb 13y ago
  I agree with you. The pain occurs when this breaks down. Take Vagrant, for example. Brilliant tool. Depends on gems that weren't distro packaged for years. This made Vagrant unusable for anyone not deploying a ruby-based application stack, unless sysadmins accepted that they were going to have to deal with the extra work associated with multiple vendors and update mechanisms for each language on the system. So those of us using other stacks could use Vagrant, but only if we adopted Ruby's app-level packaging system. See https://news.ycombinator.com/item?id=2059964 https://news.ycombinator.com/item?id=2059964 as an example of life being made difficult for distro packagers by the Ruby community. Since Rubyists tend to insist that distro packages are bad, so this is how it tends to stay. Thankfully the packaging of Vagrant and dependencies has now happened (finally; after years), and I can "apt-get install vagrant" now. But I hope this illustrates the problem. In the common use case, there is a dichotomy between distro and app-level dependencies, packaging and development. But in the grand scheme of things we all want to share our work across communities, and this dichotomy is a barrier to this ideal. Dependencies don't just go in one direction with language module repositories like pypi and rubygems at the top. Sometimes we want parts of the system to depend on them, too.
zipfle 13y ago
The design of this blog is so good that I almost didn't notice how good it is.
- ams6110 13y ago
  I'm guessing it was produced using LaTeX
  - RamiK 13y ago
    
- paultag 13y ago
  (author here) D'aw, thanks so much. I really appreciate that. I identify as a OS-level hacker, and very focused on code, not design, so it means a lot to hear such kind words when I put a bit of effort into it. Thank you!
  - pyre 13y ago
    It was really confusing trying to see how you did the large letters at the start of the paragraphs because Firefox's inspector didn't show the css rules for :first-character since the DOM element selected was the <p>.
auvrw 13y ago
well-written (and -typeset) article, although i think there is probably less controversy around this than the tone of the article suggests? anyway, side note: switched from nm-applet to wicd-client and haven't looked back.
- paultag 13y ago
  (author here) Thanks! Pairing the fonts took a little while, and fiddling with it took a bit of time - so thank you! :) Yeah. The goal here isn't to stir the pot in the middle of a massive flame-war, but it's the sort of thing where I see a part of one of these viewpoints now and again. The hope here is that this can be something that will be given out when someone expresses a viewpoint that starts to stir up this same-old flame again, as well as make people think a bit more seriously about issues regarding integration between language (and native) package managers (on both sides).
jamtan 13y ago
I like to get the best of both pip and the distro's native package manager ... https://github.com/jordansissel/fpm https://github.com/jordansissel/fpm
jrochkind1 13y ago
interesting that debian and perl have come to the same impasse as debian and ruby.
lmm 13y ago
Debian set a bad precedent with their tightly-integrated approach to packaging perl. Users and developers (not unreasonably) expected python to work the same way, with pypi/apt interoperating just like cpan/apt.
focusaurus 13y ago
My post on this topic, which is in agreement with this post. http://peterlyons.com/problog/2012/09/managing-per-project-interpreters-and-the-path http://peterlyons.com/problog/2012/09/managing-per-project-i...
hcarvalhoalves 13y ago
In my opinion package management is a non-solution. It doesn't solve many obvious problems (all your software is tied inside release cycles, tied to shared libraries and you can't have multiple versions of the same package installed), plus it operates under assumptions that don't even make sense anymore (we're not distributing 3rd party software bundled in floppy disks anymore to tie dependencies together). Installing applications as containerized packages (e.g., Gobolinux, OS X .app) should be the norm. Both apt and pip get it wrong because they don't solve any of the above problems by themselves (hopefully you can use pip+virtualenv to achieve containerization, and more recently, LXC).
- dec0dedab0de 13y ago
  So your point is that there should be no shared dependencies, or centralized collections of trusted software?
  - hcarvalhoalves 13y ago
    > So your point is that there should be no shared dependencies You can have both, where it makes sense, instead of everything being shared by default (e.g., you can provide all your software a base OpenSSL version). See how FreeBSD solves that with jails and sharedfs. > or centralized collections of trusted software? All you need for that is app signing (e.g., App Store). You can still update apps independently, there's no dependency management.
Steltek 13y ago
I have a slight tangent to this topic: Why does every language need their own package manager? I can understand avoiding a specific distro's PM but do we need to reinvent the wheel for every single language? Do we need to poorly reinvent Make just so the build syntax matches the source? It seems like a better solution would be to declare a common interface, both to users and to tools, for each language's package system. At least that way there's some guarantee (or at least a strong hint to the author!) that any given tool would support some nice-to-have extra functionality. A use case for me is that I see a really neat thing Foo written in an unfamiliar language. Install instructions go something like: "Just run 'simple_make foo' to try it out!" or perhaps "Just run 'curl http://get-foo-bar.baz/rootme http://get-foo-bar.baz/rootme | sh -'". Okay, now perhaps it's installed. It might even be installed where I want but it probably won't be. But what happens if I need to update it? How can I keep all of these pet projects up to date? I have to go and learn easy_install, gem, leiningen, sbt, maven, npm, go, quicklisp, cabal, etc. I mean, I'm not too lazy to read up on each tool but it adds up quick and not every tool supports an Autoconf-like --prefix or apt-get {update,upgrade}.
- est 13y ago
  Because everyone hates centralized package info manager, for instance, Windows Registry.

108 points by plessthanpt05 13y ago

I think I agree with the blog post in its entirety, as a fellow Pythonista / Debianite. The Debian packages that are released always seem to be top-notch, stable, and fit for production. (Let's forget the OpenSSL incident, shall we?) I don't fear that the new version is going to break my currently installed software, and I'm not worried about dependencies somehow having been missed causing havoc. It just works. (And when it doesn't, it's because I did something deep within dpkg/apt I should not have.) I can't say I've always had the same experience with other distributions which is the reason why I moved to Debian in the first place. Truthfully, I can't ramble any specific scenarios off from the top of my head. Any time I've had an issue with a Debian software package, the bug threads have always been constructive with proponents for both sides explaining why it should be one way over the other. Eventually, the best decision is made (even if I personally disagree). However, the sacrifice for this stability is the fact packages can become a bit 'stale' when it comes to new versions. I don't mind sacrifices like this. And if you need newer stuff, that's why backports exist. If you want the latest and the greatest use pip. But for the love of all things, couple your pip usage with a virtual environment. Hell, even if you aren't using pip get in the habit of using a venv. My only bad experience with virtualenvs was a recent Python security update (related to /dev/random). The system libraries changed, the Python executable in the venv did not, sadness ensued. Even then, once I figured out the issue, it was a quick fix. Just re-init the virtualenv, and it fixed it for me. No need to move code around. In short, if I want a stable system I don't need to babysit, I go with Debian. I trust the people maintaining the packages. I use virtualenvs for 90% of my Python development, and use pip inside of those venvs. Hasn't failed me, yet. Probably shouldn't jinx myself...

zdw 13y ago

More background on Python's packaging methods and history: http://aosabook.org/en/packaging.html http://aosabook.org/en/packaging.html I'd also add that native packages tend to be far more repeatable when you have to deploy and update a lot of systems in any environment. Note that with tools like fpm or other similar quick and dirty package generation tools, in general don't have to stick to the upstream system's development guidelines, but you get the benefits of using the native package tools.

don_draper 13y ago

"I prefer virtualenv based setups for development" I'm a casual python user and I find virtualenv to be very helpful.

st3fan 13y ago

Or do what I did: give up on Python packaging and use Go or a JVM based language that produce single artifact deployables. I have been in this Python mess far too long. Now I just make a .war file or tell Go to compile for my target platform. Both result in single files that can be deployed without post-processing, installing/compiling dependencies or whatever ... it simplifies things for everyone. Developers, CI, packagers, ops, security folks.

deleted 13y ago
[deleted]
the_mitsuhiko 13y ago
> or tell Go to compile for my target platform JFTR: the absolutely same is possible in Python.
- mattip 13y ago
  what is the py2exe equivalent for linux /OSX ?
  - oinksoft 13y ago
    py2pp for the Mac. It appears there is a py2deb and py2rpm for Linux, and I imagine users of Arch, Slack, etc. would be expected to download a .tgz and `setup.py install` it.
    - ginko 13y ago
      > I imagine users of Arch, Slack, etc. would be expected to download a .tgz and `setup.py install` it. In Arch you would wrap this in a PKGBUILD script to generate a Pacman package. You can find PKGBUILD's for most things in the AUR.
      - oinksoft 13y ago
        Ah, of course, you would want to create a slackpkg for Slack, and push to the AUR for Arch. It's still good to offer the .tgz as a baseline for users of systems you might have forgotten or not have time to package for though, (BSD, etc.).

ekr 13y ago

I don't want to start any argument, but there is a reason why Stallman recommends writing GNU packages in C (so that users don't need to acquire and install all kinds of dependencies, and not just them). I certainly think again about installing something when I see that the software is written in Java or Python, mostly because I don't like to bloat my system.

gaius 13y ago
Realistically, Python is as ubiquitous as Perl these days.
pekk 13y ago
What actual concrete problem do you have with 'bloat' due to languages such as perl?
null_ptr 13y ago
So many Ruby/Python programs are dependent on several frameworks or libraries, nothing seems to be self-contained anymore. That's putting me off, too, however I think that's in part because of my outdated expectations (as much as I hate to admit it). Things are trending towards user programs that focus exclusively on high-level problem solving and delegate all building blocks to 3rd party modules, and these modules may in turn do the same.

Siecje 13y ago

virtualenv and pip for your own project and use the package to satisfy requirements of installed programs.

fingerprinter 13y ago

I didn't realize there was a fight going on. It seems like this is where everyone settled and is a good solution: Virtualenv + pip for development system wide (apt, dpkg) python etc for system tools and packages. This makes sense to me. Keep your personal code sandboxed from your system tools. As an aside, I know there are quite a few tools and programs in both Debian and Ubuntu written in Python. I wonder if we'll look back in 5-10 years and realize/think this was a mistake. Particularly for UI code. I love me some Python, but I cringe every time I have to interact with a Python GUI application.

idank 13y ago
And what do you do if the dpkg'd version is a year older than the one you'd like to use and is available in pypi? Install just that one via pip? But what if that package has dependencies newer than what's installable with apt? In the end you find yourself installing as much as possible with pip, and resort to apt for the really heavy stuff like numpy.
justincormack 13y ago
And what for production? A lot of the argument seems to be around the dev/ops divide.
- fingerprinter 13y ago
  There's probably a bunch of different ways to do this, but I just use virtualenv + pip still. Here's some documentation on Django (as an example): https://docs.djangoproject.com/en/dev/howto/deployment/wsgi/modwsgi/#using-a-virtualenv https://docs.djangoproject.com/en/dev/howto/deployment/wsgi/... I think a more interesting question is not around the python side of things, but the infrastructure. Do you compile all your infra tools from source or do you use the system packages? Like, if you want to use RabbitMQ, if debian/Ubuntu provides it, would you use that or grab the source? I personally prefer to use the debian/ubuntu ones in these cases unless they are horribly outdated, which isn't generally the case. I can understand, however, if someone thinks this is a terrible idea.
nandhp 13y ago

emillon 13y ago

The background of this post is a thread on the debian-python mailing list, relative to a new PEP: http://lists.debian.org/debian-python/2013/09/msg00049.html http://lists.debian.org/debian-python/2013/09/msg00049.html

pedrocr 13y ago

Where I like to draw the line between apt and rubygems/pip is between "things that are base system" and "things that are app deployment". So in a typical rails stack that would be apt installs the system-wide stack (apache, ruby, passenger, rubygems) and then capistrano is used to deploy my webapps with their gems from my dev machine to their app-specific directory in the production server. This way I can have my sysadmin hat on while setting up the server and depend on debian/ubuntu to handle security upgrades and generally create a consistent system. Then I can put my devops hat on and use capistrano and bundler to manage the security/dependencies of my own code. But I see where this breaks down. If the base stack is moving at a much faster pace than the distributions (e.g., right now the version of passenger in Ubuntu LTS is incredibly old) it's attractive to just ignore the system packages and install everything from original sources (e.g., install ruby from source with rbenv). But doing that is just throwing away the integration work the distribution has done. I'd much rather include some extra repositories to get updated versions that integrate through apt for the few things that I care to upgrade faster than Ubuntu LTS allows me. Right now that's puppet, passenger and a few more.

rlpb 13y ago
I agree with you. The pain occurs when this breaks down. Take Vagrant, for example. Brilliant tool. Depends on gems that weren't distro packaged for years. This made Vagrant unusable for anyone not deploying a ruby-based application stack, unless sysadmins accepted that they were going to have to deal with the extra work associated with multiple vendors and update mechanisms for each language on the system. So those of us using other stacks could use Vagrant, but only if we adopted Ruby's app-level packaging system. See https://news.ycombinator.com/item?id=2059964 https://news.ycombinator.com/item?id=2059964 as an example of life being made difficult for distro packagers by the Ruby community. Since Rubyists tend to insist that distro packages are bad, so this is how it tends to stay. Thankfully the packaging of Vagrant and dependencies has now happened (finally; after years), and I can "apt-get install vagrant" now. But I hope this illustrates the problem. In the common use case, there is a dichotomy between distro and app-level dependencies, packaging and development. But in the grand scheme of things we all want to share our work across communities, and this dichotomy is a barrier to this ideal. Dependencies don't just go in one direction with language module repositories like pypi and rubygems at the top. Sometimes we want parts of the system to depend on them, too.

zipfle 13y ago

The design of this blog is so good that I almost didn't notice how good it is.

ams6110 13y ago
I'm guessing it was produced using LaTeX
- RamiK 13y ago
paultag 13y ago
(author here) D'aw, thanks so much. I really appreciate that. I identify as a OS-level hacker, and very focused on code, not design, so it means a lot to hear such kind words when I put a bit of effort into it. Thank you!
- pyre 13y ago
  It was really confusing trying to see how you did the large letters at the start of the paragraphs because Firefox's inspector didn't show the css rules for :first-character since the DOM element selected was the <p>.

auvrw 13y ago

well-written (and -typeset) article, although i think there is probably less controversy around this than the tone of the article suggests? anyway, side note: switched from nm-applet to wicd-client and haven't looked back.

paultag 13y ago
(author here) Thanks! Pairing the fonts took a little while, and fiddling with it took a bit of time - so thank you! :) Yeah. The goal here isn't to stir the pot in the middle of a massive flame-war, but it's the sort of thing where I see a part of one of these viewpoints now and again. The hope here is that this can be something that will be given out when someone expresses a viewpoint that starts to stir up this same-old flame again, as well as make people think a bit more seriously about issues regarding integration between language (and native) package managers (on both sides).

jamtan 13y ago

I like to get the best of both pip and the distro's native package manager ... https://github.com/jordansissel/fpm https://github.com/jordansissel/fpm

jrochkind1 13y ago

interesting that debian and perl have come to the same impasse as debian and ruby.

lmm 13y ago

Debian set a bad precedent with their tightly-integrated approach to packaging perl. Users and developers (not unreasonably) expected python to work the same way, with pypi/apt interoperating just like cpan/apt.

focusaurus 13y ago

My post on this topic, which is in agreement with this post. http://peterlyons.com/problog/2012/09/managing-per-project-interpreters-and-the-path http://peterlyons.com/problog/2012/09/managing-per-project-i...

hcarvalhoalves 13y ago

In my opinion package management is a non-solution. It doesn't solve many obvious problems (all your software is tied inside release cycles, tied to shared libraries and you can't have multiple versions of the same package installed), plus it operates under assumptions that don't even make sense anymore (we're not distributing 3rd party software bundled in floppy disks anymore to tie dependencies together). Installing applications as containerized packages (e.g., Gobolinux, OS X .app) should be the norm. Both apt and pip get it wrong because they don't solve any of the above problems by themselves (hopefully you can use pip+virtualenv to achieve containerization, and more recently, LXC).

dec0dedab0de 13y ago
So your point is that there should be no shared dependencies, or centralized collections of trusted software?
- hcarvalhoalves 13y ago
  > So your point is that there should be no shared dependencies You can have both, where it makes sense, instead of everything being shared by default (e.g., you can provide all your software a base OpenSSL version). See how FreeBSD solves that with jails and sharedfs. > or centralized collections of trusted software? All you need for that is app signing (e.g., App Store). You can still update apps independently, there's no dependency management.

Steltek 13y ago

I have a slight tangent to this topic: Why does every language need their own package manager? I can understand avoiding a specific distro's PM but do we need to reinvent the wheel for every single language? Do we need to poorly reinvent Make just so the build syntax matches the source? It seems like a better solution would be to declare a common interface, both to users and to tools, for each language's package system. At least that way there's some guarantee (or at least a strong hint to the author!) that any given tool would support some nice-to-have extra functionality. A use case for me is that I see a really neat thing Foo written in an unfamiliar language. Install instructions go something like: "Just run 'simple_make foo' to try it out!" or perhaps "Just run 'curl http://get-foo-bar.baz/rootme http://get-foo-bar.baz/rootme | sh -'". Okay, now perhaps it's installed. It might even be installed where I want but it probably won't be. But what happens if I need to update it? How can I keep all of these pet projects up to date? I have to go and learn easy_install, gem, leiningen, sbt, maven, npm, go, quicklisp, cabal, etc. I mean, I'm not too lazy to read up on each tool but it adds up quick and not every tool supports an Autoconf-like --prefix or apt-get {update,upgrade}.

est 13y ago
Because everyone hates centralized package info manager, for instance, Windows Registry.