11 ms·

Pkgr.io: Easy way to get a Debian package out of any app

184 points by crohr 13y ago

blinry 13y ago
I recently discovered the OpenSUSE build service, an automated build service for RPM and DEB packages: https://build.opensuse.org/ https://build.opensuse.org/ It builds the packages on virtual machines for many different distributions and 32+64 bit CPUs.
- baldfat 13y ago
  The number one under used service on the planet. It is amazing and should have revolutionized packaging years ago!!! Hope people will just "get it" soon!
  - gingerlime 13y ago
    I took a look on the website as it does sound promising, but couldn't figure out what's going on (even with the precursor info on this HN thread). I could totally "get" why it's the number one under used service. I wish I could just as easily grasp how to use this service. pkgr.io does a way better job explaining what the service provide, why it's a good idea to use it, and even a quick overview of how to use it.
- comice 13y ago
  https://launchpad.net/ https://launchpad.net/ does this too (though just for Ubuntu distros). It's quite nice and a pretty impressive service, considering it's provided for free.
bguthrie 13y ago
Gist of the thing: As a Rails developer, I've become tired of messing with ruby versions, gem dependencies, bundle installations, deploy-time asset precompilation on my servers, etc. ... Basically, what I wanted was a Heroku-like workflow that generates and hosts debian packages, which is what exactly what PKGR provides. As a customer of Heroku's excellent but rather expensive services, and a fan of deployment via package management, I'm certainly intrigued.
- wting 13y ago
  I use Dokku (self hosted Heroku) for this purpose: https://github.com/progrium/dokku https://github.com/progrium/dokku
krupan 13y ago
Does this only work for Ruby projects?
- crohr 13y ago
  At launch, only Ruby and Node.js projects are officially supported. But you can always specify another buildpack in your app's configuration to test with other runtimes (https://pkgr.io/doc#buildpack https://pkgr.io/doc#buildpack).
jlafon 13y ago
Well done. All of my questions were answered by browsing the site. +1 for using SSL everywhere.
qbonnard 13y ago
How does it compare to Docker for example?
- pandemicsyn 13y ago
  It doesn't really compare. It's: "The easiest way to get a debian package out of any app" Basically you push up to the service and it takes care of building a debian package for you and provides an apt repo to pull from. Usually you end up having to do some permutation of https://wiki.debian.org/HowToPackageForDebian https://wiki.debian.org/HowToPackageForDebian
- crohr 13y ago
  OP here. For all the goodness offered by Docker to deploy apps, sometimes a good old debian package is all that's needed. The format is widely supported and battle-tested, while Docker requires specificities under the form of kernel version, additional software to install, and additional operational knowledge. In short, if you already have a working Docker installation, you probably don't need PKGR, though nothing prevents you from building your containers by simply apt-get install'ing the app instead of having 10's of RUN commands to install the dependencies.
  - mwcampbell 13y ago
    I could see pkgr being useful for creating an on-premises "enterprise" version of a web app, in situations where the target customers would like to deploy it on their own Debian or Ubuntu machines. Docker is a bit too cutting edge for that scenario.
    - jpdlla 13y ago
      This is exactly the same scenario I had in mind. This would allow projects to have an easier way to distribute some kind of Enterprise version of a web app. Will definitely take a look.
- jafaku 13y ago
  Yeah, I don't think I will ever need stuff like what OP created now that we have Docker. Containerize all the things!
mtrn 13y ago
Packaging is a real pain, kind of a schlep. One project, which solved a lot of problems for me was https://github.com/jordansissel/fpm https://github.com/jordansissel/fpm. One edge-case, where fpm cannot help by itself, occurs in the python (w/ C-bindings) world, when your dev and prod environments use different libc versions. I got around that issue by running fpm inside a VM, which matches the libc version of the target system. But it's still not as easy as it can be - so pkgr.io looks promising.
- crohr 13y ago
  OP here, thanks for the comment. fpm is a great tool: PKGR is based on https://github.com/crohr/pkgr https://github.com/crohr/pkgr, which itself uses fpm. That being said, you indeed need to make sure your build system is close to your target system, and it is not that easy to package complex apps with specific dependencies. Generally, (Web) Application packaging does not have the same needs as OS packaging.
crymer11 13y ago
How does this work with assets that need to be served externally, e.g. uploaded to S3?
- crohr 13y ago
  You have the possibility to add custom before/after hooks, and thus upload freshly precompiled assets from them, however there is currently no secure way to encrypt secrets (in your case, S3 credentials). This may come later on (a la Travis-CI, http://docs.travis-ci.com/user/encryption-keys/ http://docs.travis-ci.com/user/encryption-keys/), but is not a priority right now.
  - crymer11 13y ago
    Are the hooks not run on each instance the package is being deployed to?
    - crohr 13y ago
      No, the hooks currently supported are scripts that are run before and after the slug compilation phase. I don't think you would want to make your package installation dependent on pre/post install scripts that access the network.
baldfat 13y ago
Not interested because it only works for deb.
jjgreen 13y ago
Since a .deb, shouldn't this be installed into /usr rather than /opt?
- phaer 13y ago
  No, why? It should be installed in /usr and so on if it conforms to debians packaging guidelines, otherwise it should stay in /opt.
  - nailer 13y ago
    Good point. I guess random apps might, for example, put their binaries outside bin, or their docs outside of share/docs/myapp
- YokoZar 13y ago
  No, as a non distro-official package it should install into its own prefix in /opt.
  - jjgreen 13y ago
    According to https://wiki.debian.org/FilesystemHierarchyStandard https://wiki.debian.org/FilesystemHierarchyStandard /opt is for "Pre-compiled, non '.deb' binary distributions", and then using the standard bin/ share/ .. subdirectories. The "own prefix" convention is for RH variants IIRC
    - YokoZar 13y ago
      These aren't "proper" debs though -- they're not provided by the distribution, don't use the standard build tools, and so on. The closest analogy I can think of is the new packages you get from the paid portion of the Ubuntu App Store (ie, not universe), and those indeed conform to the /opt/vendor standard (with a few exceptions for things like a .desktop file).
jamesgeck0 13y ago
My first thought was that there was no way I'd want to make my deploys dependent on yet another random webservice. But it looks like there's a standalone version, too. https://github.com/crohr/pkgr https://github.com/crohr/pkgr
comice 13y ago
"Uses Heroku buildpacks to embed all your dependencies within the debian package. That way, no need to mess with stale system dependencies" That does mean, however, that it's now your job to rebuild your package whenever a library gets a security fix. Possibly worth the sacrifice, but an important one I think people need to understand.
- crohr 13y ago
  OP here. Totally right, the final goal being that a new version of the package gets automatically rebuilt whenever a buildpack gets updated with a security fix.
maslam 13y ago
This solves a real problem. Nicely done. Can't wait to try it out.
comice 13y ago
Somewhat related is gem2deb: https://github.com/lnussbaum/gem2deb https://github.com/lnussbaum/gem2deb Missing the neat config interface of pkgr (and the build service obviously!) but it's fully supported by Debian/Ubuntu now (used to build almost all of their ruby app and library packages).
pekk 13y ago
Installing dependencies for web apps at the system level is usually the wrong thing. You may have a deb but now you are prone to conflicts between that deb and the distro's debs.
- mwcampbell 13y ago
  There are no conflicts, because the application debs put everything in e.g. /opt/discourse
tomswartz07 13y ago
This looks very useful for .deb systems. Does anyone have any suggestions for other platforms? Perhaps Windows, OSX, etc?
brokenparser 13y ago
Do the generated packages pass lintian?
otterley 13y ago
Any plans to do the same for RPM packages? Most enterprises are running a RHEL variant, not Debian/Ubuntu.
takeoutweight 13y ago
To offer an alternative, I've personally found checkinstall to be useful for putting together ad-hoc .deb packages for deployment. http://manpages.ubuntu.com/manpages/trusty/man8/checkinstall.8.html http://manpages.ubuntu.com/manpages/trusty/man8/checkinstall...
- regularfry 13y ago
  Unfortunately checkinstall has an inconvenient bug on (at least) Debian Wheezy which makes sudo required for package building under some circumstances.
  - infinity0 13y ago
    How so? What goes wrong with `make && sudo checkinstall make install`?
    - regularfry 13y ago
      Filesystem translation has a bug: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=717778 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=717778 https://bugs.launchpad.net/ubuntu/+source/checkinstall/+bug/78455 https://bugs.launchpad.net/ubuntu/+source/checkinstall/+bug/... This manifests itself by any "mkdir -p" which requires root in the `make install` phase failing.
drdaeman 13y ago
Not sure whenever this is a good service or not (proper Debian packaging is 10% writing scripts 90% reading and conforming to policies), but I believe it would be really better if it'd be able to produce and maintain "debian" branch and tags for pbuilder/git-buildpackage, so standard package building practices would apply.
- derefr 13y ago
  Pkgr (and fpm, the program it uses) is not for shipping a package that people can install and depend upon as part of their own OS distribution. It's for using apt-get as your orchestration system, instead of puppet/chef/etc. You package your application and its dependencies as .debs, create an apt-mirror containing those debs, add that apt repo to your deploy-targets' sources.list.d's, "apt-get install yourcorp-system-release", and relax.
  - regularfry 13y ago
    We just use apt-get as the deployment and versioning mechanism, driven from puppet. This works well.
  - infinity0 13y ago
    They should add a disclaimer that they are not a replacement for proper Debian packaging then. Lots of whiny people complain about the complexity of Debian packaging when really they don't understand the requirements for integrating nicely with every other installed program or potentially-installable program.

184 points by crohr 13y ago

I recently discovered the OpenSUSE build service, an automated build service for RPM and DEB packages: https://build.opensuse.org/ https://build.opensuse.org/ It builds the packages on virtual machines for many different distributions and 32+64 bit CPUs.

baldfat 13y ago
The number one under used service on the planet. It is amazing and should have revolutionized packaging years ago!!! Hope people will just "get it" soon!
- gingerlime 13y ago
  I took a look on the website as it does sound promising, but couldn't figure out what's going on (even with the precursor info on this HN thread). I could totally "get" why it's the number one under used service. I wish I could just as easily grasp how to use this service. pkgr.io does a way better job explaining what the service provide, why it's a good idea to use it, and even a quick overview of how to use it.
comice 13y ago
https://launchpad.net/ https://launchpad.net/ does this too (though just for Ubuntu distros). It's quite nice and a pretty impressive service, considering it's provided for free.

bguthrie 13y ago

Gist of the thing: As a Rails developer, I've become tired of messing with ruby versions, gem dependencies, bundle installations, deploy-time asset precompilation on my servers, etc. ... Basically, what I wanted was a Heroku-like workflow that generates and hosts debian packages, which is what exactly what PKGR provides. As a customer of Heroku's excellent but rather expensive services, and a fan of deployment via package management, I'm certainly intrigued.

wting 13y ago
I use Dokku (self hosted Heroku) for this purpose: https://github.com/progrium/dokku https://github.com/progrium/dokku

krupan 13y ago

Does this only work for Ruby projects?

crohr 13y ago
At launch, only Ruby and Node.js projects are officially supported. But you can always specify another buildpack in your app's configuration to test with other runtimes (https://pkgr.io/doc#buildpack https://pkgr.io/doc#buildpack).

jlafon 13y ago

Well done. All of my questions were answered by browsing the site. +1 for using SSL everywhere.

qbonnard 13y ago

How does it compare to Docker for example?

pandemicsyn 13y ago
It doesn't really compare. It's: "The easiest way to get a debian package out of any app" Basically you push up to the service and it takes care of building a debian package for you and provides an apt repo to pull from. Usually you end up having to do some permutation of https://wiki.debian.org/HowToPackageForDebian https://wiki.debian.org/HowToPackageForDebian
crohr 13y ago
OP here. For all the goodness offered by Docker to deploy apps, sometimes a good old debian package is all that's needed. The format is widely supported and battle-tested, while Docker requires specificities under the form of kernel version, additional software to install, and additional operational knowledge. In short, if you already have a working Docker installation, you probably don't need PKGR, though nothing prevents you from building your containers by simply apt-get install'ing the app instead of having 10's of RUN commands to install the dependencies.
- mwcampbell 13y ago
  I could see pkgr being useful for creating an on-premises "enterprise" version of a web app, in situations where the target customers would like to deploy it on their own Debian or Ubuntu machines. Docker is a bit too cutting edge for that scenario.
  - jpdlla 13y ago
    This is exactly the same scenario I had in mind. This would allow projects to have an easier way to distribute some kind of Enterprise version of a web app. Will definitely take a look.
jafaku 13y ago
Yeah, I don't think I will ever need stuff like what OP created now that we have Docker. Containerize all the things!

mtrn 13y ago

Packaging is a real pain, kind of a schlep. One project, which solved a lot of problems for me was https://github.com/jordansissel/fpm https://github.com/jordansissel/fpm. One edge-case, where fpm cannot help by itself, occurs in the python (w/ C-bindings) world, when your dev and prod environments use different libc versions. I got around that issue by running fpm inside a VM, which matches the libc version of the target system. But it's still not as easy as it can be - so pkgr.io looks promising.

crohr 13y ago
OP here, thanks for the comment. fpm is a great tool: PKGR is based on https://github.com/crohr/pkgr https://github.com/crohr/pkgr, which itself uses fpm. That being said, you indeed need to make sure your build system is close to your target system, and it is not that easy to package complex apps with specific dependencies. Generally, (Web) Application packaging does not have the same needs as OS packaging.

crymer11 13y ago

How does this work with assets that need to be served externally, e.g. uploaded to S3?

crohr 13y ago
You have the possibility to add custom before/after hooks, and thus upload freshly precompiled assets from them, however there is currently no secure way to encrypt secrets (in your case, S3 credentials). This may come later on (a la Travis-CI, http://docs.travis-ci.com/user/encryption-keys/ http://docs.travis-ci.com/user/encryption-keys/), but is not a priority right now.
- crymer11 13y ago
  Are the hooks not run on each instance the package is being deployed to?
  - crohr 13y ago
    No, the hooks currently supported are scripts that are run before and after the slug compilation phase. I don't think you would want to make your package installation dependent on pre/post install scripts that access the network.

baldfat 13y ago

Not interested because it only works for deb.

jjgreen 13y ago

Since a .deb, shouldn't this be installed into /usr rather than /opt?

phaer 13y ago
No, why? It should be installed in /usr and so on if it conforms to debians packaging guidelines, otherwise it should stay in /opt.
- nailer 13y ago
  Good point. I guess random apps might, for example, put their binaries outside bin, or their docs outside of share/docs/myapp
YokoZar 13y ago
No, as a non distro-official package it should install into its own prefix in /opt.
- jjgreen 13y ago
  According to https://wiki.debian.org/FilesystemHierarchyStandard https://wiki.debian.org/FilesystemHierarchyStandard /opt is for "Pre-compiled, non '.deb' binary distributions", and then using the standard bin/ share/ .. subdirectories. The "own prefix" convention is for RH variants IIRC
  - YokoZar 13y ago
    These aren't "proper" debs though -- they're not provided by the distribution, don't use the standard build tools, and so on. The closest analogy I can think of is the new packages you get from the paid portion of the Ubuntu App Store (ie, not universe), and those indeed conform to the /opt/vendor standard (with a few exceptions for things like a .desktop file).

jamesgeck0 13y ago

My first thought was that there was no way I'd want to make my deploys dependent on yet another random webservice. But it looks like there's a standalone version, too. https://github.com/crohr/pkgr https://github.com/crohr/pkgr

comice 13y ago

"Uses Heroku buildpacks to embed all your dependencies within the debian package. That way, no need to mess with stale system dependencies" That does mean, however, that it's now your job to rebuild your package whenever a library gets a security fix. Possibly worth the sacrifice, but an important one I think people need to understand.

crohr 13y ago
OP here. Totally right, the final goal being that a new version of the package gets automatically rebuilt whenever a buildpack gets updated with a security fix.

maslam 13y ago

This solves a real problem. Nicely done. Can't wait to try it out.

comice 13y ago

Somewhat related is gem2deb: https://github.com/lnussbaum/gem2deb https://github.com/lnussbaum/gem2deb Missing the neat config interface of pkgr (and the build service obviously!) but it's fully supported by Debian/Ubuntu now (used to build almost all of their ruby app and library packages).

pekk 13y ago

Installing dependencies for web apps at the system level is usually the wrong thing. You may have a deb but now you are prone to conflicts between that deb and the distro's debs.

mwcampbell 13y ago
There are no conflicts, because the application debs put everything in e.g. /opt/discourse

tomswartz07 13y ago

This looks very useful for .deb systems. Does anyone have any suggestions for other platforms? Perhaps Windows, OSX, etc?

brokenparser 13y ago

Do the generated packages pass lintian?

otterley 13y ago

Any plans to do the same for RPM packages? Most enterprises are running a RHEL variant, not Debian/Ubuntu.

takeoutweight 13y ago

To offer an alternative, I've personally found checkinstall to be useful for putting together ad-hoc .deb packages for deployment. http://manpages.ubuntu.com/manpages/trusty/man8/checkinstall.8.html http://manpages.ubuntu.com/manpages/trusty/man8/checkinstall...

regularfry 13y ago
Unfortunately checkinstall has an inconvenient bug on (at least) Debian Wheezy which makes sudo required for package building under some circumstances.
- infinity0 13y ago
  How so? What goes wrong with `make && sudo checkinstall make install`?
  - regularfry 13y ago
    Filesystem translation has a bug: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=717778 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=717778 https://bugs.launchpad.net/ubuntu/+source/checkinstall/+bug/78455 https://bugs.launchpad.net/ubuntu/+source/checkinstall/+bug/... This manifests itself by any "mkdir -p" which requires root in the `make install` phase failing.

drdaeman 13y ago

Not sure whenever this is a good service or not (proper Debian packaging is 10% writing scripts 90% reading and conforming to policies), but I believe it would be really better if it'd be able to produce and maintain "debian" branch and tags for pbuilder/git-buildpackage, so standard package building practices would apply.

derefr 13y ago
Pkgr (and fpm, the program it uses) is not for shipping a package that people can install and depend upon as part of their own OS distribution. It's for using apt-get as your orchestration system, instead of puppet/chef/etc. You package your application and its dependencies as .debs, create an apt-mirror containing those debs, add that apt repo to your deploy-targets' sources.list.d's, "apt-get install yourcorp-system-release", and relax.
- regularfry 13y ago
  We just use apt-get as the deployment and versioning mechanism, driven from puppet. This works well.
- infinity0 13y ago
  They should add a disclaimer that they are not a replacement for proper Debian packaging then. Lots of whiny people complain about the complexity of Debian packaging when really they don't understand the requirements for integrating nicely with every other installed program or potentially-installable program.