5 ms·

Why developers should care about system packages

48 points by garethr 16y ago

njharman 16y ago
devs, esp polygot/startup devs, need sysadmin xp/knowledge.
tedunangst 16y ago
I have never shipped unix software that used the system package manager, for a number of reasons. If you want it to be cross platform, you need to build/test/ship the fallback tgz manual install anyway. Packages don't save any work, they only increase the size of your build farm and testing matrix. The software was usually built against specific versions (often modified) of 3rd party libraries. We can't rely on the system to have them, so we need to ship them too, so it's not like relying on system packages would reduce our download size at all. All told, the number of customers who care is very small, either because people just don't care or because the people who care don't pay for software. That was for commercial software of course, but even for open source software, I'm not sure why it's the author's responsibility to go setup a dozen different VMs. That really is the package maintainer's job.
- alxp 16y ago
  > I'm not sure why it's the author's responsibility to go setup a dozen different VMs. That really is the package maintainer's job. If you're in the enviable position of having lots of people willing to put in the time to maintain your package, and a distro that thinks your software is sufficiently noteworthy to include in their distro. But if your software is relatively new and you want to encourage wide adoption, one way to help is to make the install process friendly and familiar for sysadmins as well as fellow developers. And if you speak their language that goes a long way towards building that trust.
- garethr 16y ago
  In lots of cases the separate package maintainer's do a good but ultimately thankless task. The example I gave between the rabbitmq packages which are built as part of the development effort and made available on the rabbitmq site and the solr packages which are sometimes out of date and not referenced by the site highlights the point I'm trying to make.
- zdw 16y ago
  * The software was usually built against specific versions (often modified) of 3rd party libraries. * That sounds like a disaster waiting to happen... maintaining a different version of libraries outside of their main development trunk means that: 1. Extra steps when a new main version comes out - you have to repatch your differences in. 2. #1 incurs a delay, thus a longer time that your customer are running without patches, which may be a security issue. 3. You won't do #1 at all. Thus you become reliant on old, unmaintained, buggy, security-holey libraries. The only time I can think this is acceptable is to fix problems that upstream won't fix, or are platform specific (ie, not with your custom software, but with the OS underneath everything).
  - X-Istence 16y ago
    I'm not the OP, but I will respond anyway: 1. We have made changes to the 3rd party source code that we want to ship now, not when they finally let the patch into their source tree. 2. We get new updates from the latest release, patch in our stuff and we are set, quick test on the CI servers and we can ship. Yes, customers may be without the fix for a little while longer... 3. It is actually in our best interest to keep 3rd party libraries at their latest release, if we want secure software. Which is why it is part of our release documents and steps that are required to be followed before we release.
    - saurik 16y ago
      Then what you have is a fork of the library. There is no reason you can't distribute forks of the library (to either your own paths or with your own modified names) using the system package manager. If you are expecting the person taking your patches to install them to the system, however, that's a whole 'nother ball game, and gets back to the original point: as a system administrator, I have no reason to believe your "better" versions of those libraries are going to interoperate with everything else on the system, and I am then going to have to do a bunch of work doing that isolation myself.
  - FooBarWidget 16y ago
    And if you allow the customer to upgrade libraries themselves (or install any version they desire) you'll risk breaking the application. That can be even worse.
- viraptor 16y ago
  > I'm not sure why it's the author's responsibility to go setup a dozen different VMs To make sure it compiles and works under dozen different distributions (in practice centos + ubuntu/debian usually cover most cases). If you don't test it, how do you know it's usable at all? Besides, as long as you don't invent some crazy custom build system, the only thing you really have to write is metadata and dependencies (rules file can be 3 lines long for autotools software). Also, this way you can just push your software to a build farm like launchpad ppa and let them do the dirty work. If you include self-contained tests with the sources, you can even run them after the build to verify correctness for that environment. This is much better than doing it by hand. Also keep in mind that the first person to deploy your software automatically to a couple of places will either use your/distro's package or will have to make their own. Building from source works only for one-person uncontrolled environment.
  - X-Istence 16y ago
    Please add FreeBSD to the list of OS's you use for testing of compilation. You have no idea how frustrating it is to have an open source project claim to run on unix like operating systems and not compile correctly on FreeBSD or Mac OS X...
    - tedunangst 16y ago
      try using openbsd sometime... :)
    - viraptor 16y ago
      As much as I agree... I'm not sure how much it's worth the effort until someone points out that your project definitely doesn't work. Even looking at silly stats like: http://distrowatch.com/stats.php?section=popularity http://distrowatch.com/stats.php?section=popularity - covering one popular rpm and one deb distribution, you're covering majority already (with only slight modifications if necessary). Mac users will expect their own packages anyways and you can't run Mac OSX legally without owning the hardware, which many developers don't own. Regarding FreeBSD... lack of any good result in the first 3 pages of "freebsd packaging guide" search should explain a lot.
- thwarted 16y ago
  I'm not sure why it's the author's responsibility to go setup a dozen different VMs. That really is the package maintainer's job. Agreed. I do, however, expect upstream to accept reasonable patches from package maintainers to fix building bugs or make it easier for people to build packages themselves. For the few times I've done it, I've had no trouble getting upstream to accept an updated .spec file (or even a barebones .spec file) that they can distribute in their targz to make it easier for the people who want to build their software as a package if their distribution doesn't provide one.
PKeeble 16y ago
One of the reasons software gets released in scripts rather than packages is because the root package installation mechanism is root accessible only. Most companies restrict root access and package installation is blocked by proxies. The combination of which means that scripts that can be downloaded separately and run are much easier for enterprise developers to use and get started with. Until the average enterprise realises that packaging and installation needs to be done better we will always have the need for mechanisms that hack around the problem that system admins cause for the users of the servers.
- ldng 16y ago
  I agree, but developers have enough on their plate and knowing how to administer your workstation isn't the same as administrating a server. That said, allowing user system wide installation could be possible. I think, at least under Linux, combining selinux and cgroup to provide sandboxed system wide user install could be possible but you'd possibly need to bypass classic unix directory ownership and permissions. That might require to much work or not even be impossible. Anyone here with enough knowledge to confirm or refute my feeling ?
joevandyk 16y ago
Is there a decently written guide for creating ubuntu packages? The package maintainers guide confuses the heck out of me and it's way too long and disorganized.
- rmc 16y ago
  Theres a great programme called checkinstall that gets you 90% there. Rather than running 'make install' run 'checkinstall', or rather than 'sudo python installscript.pt', run 'sudo checkinstall python installscript.py'
  - garethr 16y ago
    Checkinstall is brilliant: http://checkinstall.izto.org/ http://checkinstall.izto.org/ It's also guilty of being unattractive to the developers I'd love to see use it. The site says "Recent News: Dec 26th, 2009" and then concentrates on a NOTE TO SLACKWARE 8.0 USERS. Showing the hello world of checkinstall would be much better imho, and it's what developers from the web world expect.
    - regularfry 16y ago
      Checkinstall of Python and VirtualEnv is the only sane way I've found to do python development and deployment when you're developing on Maverick and deploying to Lenny.
- viraptor 16y ago
  https://wiki.ubuntu.com/PackagingGuide https://wiki.ubuntu.com/PackagingGuide - a little bit better than Debian's official one. Unfortunately the more you do it, the more you actually need to refer to the Debian's guide and see why it was written that way... I also recommend checking out https://wiki.ubuntu.com/PbuilderHowto https://wiki.ubuntu.com/PbuilderHowto - pbuilder is "the" way to build packages, which I unfortunately discovered very late.
- garethr 16y ago
  To my knowledge no. I think this is one of the main problems in getting people to create system packages. I think this fosters the split between package maintainers and developers. It's not that the documentation is all bad, it's just not aimed at the audience I think it should be. System packages just aren't what web savvy developers grew up on and the documentation feels out of step. But the capabilities and tools around system packages are awesome. I might take a go a writing a simple guide, but I'd be more than happy to see other people have a go that know more than I do. Let me know if this sounds of interest.
joevandyk 16y ago
Is there a decently written guide for creating ubuntu packages? The package maintainers guide confuses the heck out of me and it's way too long and disorganized.
jolan 16y ago
Your post boils down to something like "developers should ship Debian packages because I use Debian and all my favorite software packages are out of date or non-existent". Maybe you should move away from a distro that freezes for 6 months to roll a stable release which won't get anything more than security/crash fixes for 2+ years.
- viraptor 16y ago
  Many Debian developers agree: http://lwn.net/Articles/406301/ http://lwn.net/Articles/406301/ OTOH, it's nice to know that you can run in the same environment without changes for 2 years and not care about incompatible updates.
- garethr 16y ago
  That wasn't my aim. In fact I'm not using Debian at the moment as I indicated, I just know the commands better to give examples. This isn't about a specific distro, it's about the huge benefits of system packages and their toolchains in general. I'm not even really concerned with official packages and the official blessed package repos.
  - jolan 16y ago
    I've created hundreds of packages for OpenBSD and you just seem pretty gullible about the whole process. You can't just write about the packaging process and hope people come forward to volunteer to do the actual work. There's a large amount of thankless sweat equity when you provide packages. People whine when things are broken and rarely send praise when it just works as it should. If you really want to change the world, start with creating a Ruby apt repository for Debian/Ubuntu/whatever you use that people can actually deploy in production.
regularfry 16y ago
For my money, here's what needs to happen (from a Ruby perspective; salt to taste): 1. Debian and Ubuntu stop packaging any Ruby libraries which aren't dependencies of an actual end-user application. If rubygems itself is a library dependency, package it but take away the gem command. 2. Make a public policy saying "If you want to do development, use RVM," and have metapackages with the dependencies for building each RVM ruby (so ruby-dev-mri, ruby-dev-jruby, and so on). Make sure that users know that if they think they need to run "gem install foo", then they need RVM. It'd be nice for RVM to be able to build system packages, but I don't think it's necessary: there are already systems (nascent, but improving) for converting gems to system packages which handle dependencies nicely. Is there anything wrong with this picture?
- garethr 16y ago
  RVM is great. But by design it's a solution to compiling software on every machine. This simply doesn't work for lots of sysadmins, especially with lots of machines - it means installing a complete build environment which opens up much more evil.
  - regularfry 16y ago
    Well, yes. And that's the effective situation we've got at the moment. What I'm proposing is a full acknowledgement of that: if you want to live on the bleeding edge, then that should be supported as well as possible, but a very clear delineation should be drawn between the responsibilities of the distribution and those of the developer/sysadmin. RVM does that very cleanly, even if you only use it with the system ruby.
- jamesbritt 16y ago
  "If rubygems itself is a library dependency, package it but take away the gem command." Might be better to un-gem any required gems and package them with the app. Then add some code that fakes out any calls in the app to 'gem' or 'rubygems'. OTOH, rubygems is part of ruby now, so leaving it out might be a problem.
  - 16y ago
SwellJoe 16y ago
I consider it a significant black mark on a product, open source or otherwise, if it doesn't have native packages for the most popular Linux distros. It's even better if they provide a yum or apt repository. It is a challenge to do so, but if we can do it for our products (which have vast and complicated dependency chains and need to perform a huge amount of post-configuration in order to be easy for non-technical users to use), damned near any other project ought to be able to do it. The vast majority of really ugly security issues I've seen have been due to people having installed something from source, sometimes years ago, and not realizing they're running exploitable software because their package manager tells them they're up to date. This reason alone should be enough to keep people using the native package manager for as much as possible. But there are many other good reasons, a few of which have been touched on in this article.
- mapleoin 16y ago
  Collaborating with distribution packagers is a lot better than maintaining your own repositories. Packagers should always be up to speed with the latest guidelines and best practices for their distribution. I have often seen badly built packages by upstreams.
  - SwellJoe 16y ago
    That's not feasible in a lot of cases. Commercial software, for instance, is not generally welcome in distribution repositories.
    - regularfry 16y ago
      No, but running a public repository for a given distribution is the way forward in that case, and that does involve knowing the lay of the land.
      - SwellJoe 16y ago
        Which is exactly what I said in my previous comment.

48 points by garethr 16y ago

devs, esp polygot/startup devs, need sysadmin xp/knowledge.

I have never shipped unix software that used the system package manager, for a number of reasons. If you want it to be cross platform, you need to build/test/ship the fallback tgz manual install anyway. Packages don't save any work, they only increase the size of your build farm and testing matrix. The software was usually built against specific versions (often modified) of 3rd party libraries. We can't rely on the system to have them, so we need to ship them too, so it's not like relying on system packages would reduce our download size at all. All told, the number of customers who care is very small, either because people just don't care or because the people who care don't pay for software. That was for commercial software of course, but even for open source software, I'm not sure why it's the author's responsibility to go setup a dozen different VMs. That really is the package maintainer's job.

alxp 16y ago
> I'm not sure why it's the author's responsibility to go setup a dozen different VMs. That really is the package maintainer's job. If you're in the enviable position of having lots of people willing to put in the time to maintain your package, and a distro that thinks your software is sufficiently noteworthy to include in their distro. But if your software is relatively new and you want to encourage wide adoption, one way to help is to make the install process friendly and familiar for sysadmins as well as fellow developers. And if you speak their language that goes a long way towards building that trust.
garethr 16y ago
In lots of cases the separate package maintainer's do a good but ultimately thankless task. The example I gave between the rabbitmq packages which are built as part of the development effort and made available on the rabbitmq site and the solr packages which are sometimes out of date and not referenced by the site highlights the point I'm trying to make.
zdw 16y ago
* The software was usually built against specific versions (often modified) of 3rd party libraries. * That sounds like a disaster waiting to happen... maintaining a different version of libraries outside of their main development trunk means that: 1. Extra steps when a new main version comes out - you have to repatch your differences in. 2. #1 incurs a delay, thus a longer time that your customer are running without patches, which may be a security issue. 3. You won't do #1 at all. Thus you become reliant on old, unmaintained, buggy, security-holey libraries. The only time I can think this is acceptable is to fix problems that upstream won't fix, or are platform specific (ie, not with your custom software, but with the OS underneath everything).
- X-Istence 16y ago
  I'm not the OP, but I will respond anyway: 1. We have made changes to the 3rd party source code that we want to ship now, not when they finally let the patch into their source tree. 2. We get new updates from the latest release, patch in our stuff and we are set, quick test on the CI servers and we can ship. Yes, customers may be without the fix for a little while longer... 3. It is actually in our best interest to keep 3rd party libraries at their latest release, if we want secure software. Which is why it is part of our release documents and steps that are required to be followed before we release.
  - saurik 16y ago
    Then what you have is a fork of the library. There is no reason you can't distribute forks of the library (to either your own paths or with your own modified names) using the system package manager. If you are expecting the person taking your patches to install them to the system, however, that's a whole 'nother ball game, and gets back to the original point: as a system administrator, I have no reason to believe your "better" versions of those libraries are going to interoperate with everything else on the system, and I am then going to have to do a bunch of work doing that isolation myself.
- FooBarWidget 16y ago
  And if you allow the customer to upgrade libraries themselves (or install any version they desire) you'll risk breaking the application. That can be even worse.
viraptor 16y ago
> I'm not sure why it's the author's responsibility to go setup a dozen different VMs To make sure it compiles and works under dozen different distributions (in practice centos + ubuntu/debian usually cover most cases). If you don't test it, how do you know it's usable at all? Besides, as long as you don't invent some crazy custom build system, the only thing you really have to write is metadata and dependencies (rules file can be 3 lines long for autotools software). Also, this way you can just push your software to a build farm like launchpad ppa and let them do the dirty work. If you include self-contained tests with the sources, you can even run them after the build to verify correctness for that environment. This is much better than doing it by hand. Also keep in mind that the first person to deploy your software automatically to a couple of places will either use your/distro's package or will have to make their own. Building from source works only for one-person uncontrolled environment.
- X-Istence 16y ago
  Please add FreeBSD to the list of OS's you use for testing of compilation. You have no idea how frustrating it is to have an open source project claim to run on unix like operating systems and not compile correctly on FreeBSD or Mac OS X...
  - tedunangst 16y ago
    try using openbsd sometime... :)
  - viraptor 16y ago
    As much as I agree... I'm not sure how much it's worth the effort until someone points out that your project definitely doesn't work. Even looking at silly stats like: http://distrowatch.com/stats.php?section=popularity http://distrowatch.com/stats.php?section=popularity - covering one popular rpm and one deb distribution, you're covering majority already (with only slight modifications if necessary). Mac users will expect their own packages anyways and you can't run Mac OSX legally without owning the hardware, which many developers don't own. Regarding FreeBSD... lack of any good result in the first 3 pages of "freebsd packaging guide" search should explain a lot.
thwarted 16y ago
I'm not sure why it's the author's responsibility to go setup a dozen different VMs. That really is the package maintainer's job. Agreed. I do, however, expect upstream to accept reasonable patches from package maintainers to fix building bugs or make it easier for people to build packages themselves. For the few times I've done it, I've had no trouble getting upstream to accept an updated .spec file (or even a barebones .spec file) that they can distribute in their targz to make it easier for the people who want to build their software as a package if their distribution doesn't provide one.

PKeeble 16y ago

One of the reasons software gets released in scripts rather than packages is because the root package installation mechanism is root accessible only. Most companies restrict root access and package installation is blocked by proxies. The combination of which means that scripts that can be downloaded separately and run are much easier for enterprise developers to use and get started with. Until the average enterprise realises that packaging and installation needs to be done better we will always have the need for mechanisms that hack around the problem that system admins cause for the users of the servers.

ldng 16y ago
I agree, but developers have enough on their plate and knowing how to administer your workstation isn't the same as administrating a server. That said, allowing user system wide installation could be possible. I think, at least under Linux, combining selinux and cgroup to provide sandboxed system wide user install could be possible but you'd possibly need to bypass classic unix directory ownership and permissions. That might require to much work or not even be impossible. Anyone here with enough knowledge to confirm or refute my feeling ?

joevandyk 16y ago

Is there a decently written guide for creating ubuntu packages? The package maintainers guide confuses the heck out of me and it's way too long and disorganized.

rmc 16y ago
Theres a great programme called checkinstall that gets you 90% there. Rather than running 'make install' run 'checkinstall', or rather than 'sudo python installscript.pt', run 'sudo checkinstall python installscript.py'
- garethr 16y ago
  Checkinstall is brilliant: http://checkinstall.izto.org/ http://checkinstall.izto.org/ It's also guilty of being unattractive to the developers I'd love to see use it. The site says "Recent News: Dec 26th, 2009" and then concentrates on a NOTE TO SLACKWARE 8.0 USERS. Showing the hello world of checkinstall would be much better imho, and it's what developers from the web world expect.
  - regularfry 16y ago
    Checkinstall of Python and VirtualEnv is the only sane way I've found to do python development and deployment when you're developing on Maverick and deploying to Lenny.
viraptor 16y ago
https://wiki.ubuntu.com/PackagingGuide https://wiki.ubuntu.com/PackagingGuide - a little bit better than Debian's official one. Unfortunately the more you do it, the more you actually need to refer to the Debian's guide and see why it was written that way... I also recommend checking out https://wiki.ubuntu.com/PbuilderHowto https://wiki.ubuntu.com/PbuilderHowto - pbuilder is "the" way to build packages, which I unfortunately discovered very late.
garethr 16y ago
To my knowledge no. I think this is one of the main problems in getting people to create system packages. I think this fosters the split between package maintainers and developers. It's not that the documentation is all bad, it's just not aimed at the audience I think it should be. System packages just aren't what web savvy developers grew up on and the documentation feels out of step. But the capabilities and tools around system packages are awesome. I might take a go a writing a simple guide, but I'd be more than happy to see other people have a go that know more than I do. Let me know if this sounds of interest.

joevandyk 16y ago

Is there a decently written guide for creating ubuntu packages? The package maintainers guide confuses the heck out of me and it's way too long and disorganized.

jolan 16y ago

Your post boils down to something like "developers should ship Debian packages because I use Debian and all my favorite software packages are out of date or non-existent". Maybe you should move away from a distro that freezes for 6 months to roll a stable release which won't get anything more than security/crash fixes for 2+ years.

viraptor 16y ago
Many Debian developers agree: http://lwn.net/Articles/406301/ http://lwn.net/Articles/406301/ OTOH, it's nice to know that you can run in the same environment without changes for 2 years and not care about incompatible updates.
garethr 16y ago
That wasn't my aim. In fact I'm not using Debian at the moment as I indicated, I just know the commands better to give examples. This isn't about a specific distro, it's about the huge benefits of system packages and their toolchains in general. I'm not even really concerned with official packages and the official blessed package repos.
- jolan 16y ago
  I've created hundreds of packages for OpenBSD and you just seem pretty gullible about the whole process. You can't just write about the packaging process and hope people come forward to volunteer to do the actual work. There's a large amount of thankless sweat equity when you provide packages. People whine when things are broken and rarely send praise when it just works as it should. If you really want to change the world, start with creating a Ruby apt repository for Debian/Ubuntu/whatever you use that people can actually deploy in production.

regularfry 16y ago

For my money, here's what needs to happen (from a Ruby perspective; salt to taste): 1. Debian and Ubuntu stop packaging any Ruby libraries which aren't dependencies of an actual end-user application. If rubygems itself is a library dependency, package it but take away the gem command. 2. Make a public policy saying "If you want to do development, use RVM," and have metapackages with the dependencies for building each RVM ruby (so ruby-dev-mri, ruby-dev-jruby, and so on). Make sure that users know that if they think they need to run "gem install foo", then they need RVM. It'd be nice for RVM to be able to build system packages, but I don't think it's necessary: there are already systems (nascent, but improving) for converting gems to system packages which handle dependencies nicely. Is there anything wrong with this picture?

garethr 16y ago
RVM is great. But by design it's a solution to compiling software on every machine. This simply doesn't work for lots of sysadmins, especially with lots of machines - it means installing a complete build environment which opens up much more evil.
- regularfry 16y ago
  Well, yes. And that's the effective situation we've got at the moment. What I'm proposing is a full acknowledgement of that: if you want to live on the bleeding edge, then that should be supported as well as possible, but a very clear delineation should be drawn between the responsibilities of the distribution and those of the developer/sysadmin. RVM does that very cleanly, even if you only use it with the system ruby.
jamesbritt 16y ago
"If rubygems itself is a library dependency, package it but take away the gem command." Might be better to un-gem any required gems and package them with the app. Then add some code that fakes out any calls in the app to 'gem' or 'rubygems'. OTOH, rubygems is part of ruby now, so leaving it out might be a problem.
- 16y ago

SwellJoe 16y ago

I consider it a significant black mark on a product, open source or otherwise, if it doesn't have native packages for the most popular Linux distros. It's even better if they provide a yum or apt repository. It is a challenge to do so, but if we can do it for our products (which have vast and complicated dependency chains and need to perform a huge amount of post-configuration in order to be easy for non-technical users to use), damned near any other project ought to be able to do it. The vast majority of really ugly security issues I've seen have been due to people having installed something from source, sometimes years ago, and not realizing they're running exploitable software because their package manager tells them they're up to date. This reason alone should be enough to keep people using the native package manager for as much as possible. But there are many other good reasons, a few of which have been touched on in this article.

mapleoin 16y ago
Collaborating with distribution packagers is a lot better than maintaining your own repositories. Packagers should always be up to speed with the latest guidelines and best practices for their distribution. I have often seen badly built packages by upstreams.
- SwellJoe 16y ago
  That's not feasible in a lot of cases. Commercial software, for instance, is not generally welcome in distribution repositories.
  - regularfry 16y ago
    No, but running a public repository for a given distribution is the way forward in that case, and that does involve knowing the lay of the land.
    - SwellJoe 16y ago
      Which is exactly what I said in my previous comment.