4 ms·

Unfortunately, it looks like it may not be something will be able to work with. Instead of an ABI like Apache uses, Nginx went with signature checking. To load

by cbr 11y ago

Unfortunately, it looks like it may not be something will be able to work with. Instead of an ABI like Apache uses, Nginx went with signature checking. To load a precompiled module, the signature has to match. This means we would have to compile many different binary modules for every configuration people might have, and for every version of Nginx. Like several dozen. As it stands, though, the module system would be very helpful for OS packagers, because they don't have to worry about signature mismatch.

falcolas 11y ago
I would think that containers would help obviate such concerns, since it would be rather trivial to keep even a hundred images in place for an automated compilation run. Of course, serving the output would be harder, but still not impossible.
- cbr 11y ago
  Generating all those binaries would still be a pain, but you're right that containers could help. We'd also need to write a script that looked at your nginx install and figured out what would be the right precompiled module to download.
- xorcist 11y ago
  How does containers help? Building a hundred different nginx's shouldn't be neither harder nor easier if you stuff the binary in a container or not.
  - falcolas 11y ago
    It gives you one image per configuration, instead of one build machine, or one stored VM. It makes it easier for a single machine to iterate over dozens or hundreds of build environments with a minimum of overhead. For example, one Ubuntu machine with Docker can create build environments for every Linux based distro (and version thereof) that exists, and iterate through all of them when it comes time to build. The setup is not trivial, though it is easier than creating a similar environment with VMs or physical machines.
    - snuxoll 11y ago
      Every distribution on the planet handles this with chroots for doing builds and it works out just fine without needing to do gymnastics with docker. Fedora, for example, has a farm of build hosts that are architecture specific (so there is no cross-compiling outside of bootstrapping a new build target), from there a new chroot gets created for every build and the build runs isolated within it. They've been using this same system to maintain packages for up to 7 different releases at once (counting the EPEL repositories) with no issues, I fail to see any benefit docker brings to the table here - and really, nginx should just provide a stable ABI.
LukeShu 11y ago
So does nginx still have to know about all of the modules at compile-time? Or can I compile nginx, then later compile a module for that nginx?
- cbr 11y ago
  That's fine, and is something you'd be able to do with PageSpeed. (But I'd really love a way to distribute binary modules anyone could just use, with no additional compilation.)
snuxoll 11y ago
Actually, it's still a pain for package maintainers. Take a look at mod_wsgi for apache as an example, it's maintained as a completely separate package (the rpmspec and dist-git trees are totally independent from apache) - when the httpd package gets upgraded mod_wsgi doesn't have to be rebuilt at all. This is important, in fact, because rebuilding a package means updating the spec file, bumping the release, and submitting a new build to koji - not something you want to do for loadable modules every time httpd is updated. This is going to create headaches for anyone wanting to package nginx, either they will have to compile all the modules when building nginx (creating many sub-packages) or there will have to be coordination between package maintainers to do a mass rebuild of all loadable modules every time the nginx package gets bumped. Neither situation is desirable, and I hope they release a stable ABI because until then nobody is going to waste time packaging loadable modules.
- Florin_Andrei 11y ago
  Wow. That basically means their current module scheme is useless for package maintainers.
  - snuxoll 11y ago
    Pretty much. Like I said, technically you could just have the build for nginx also include all of the modules and just split them up into sub-packages, but that defeats the entire point of having loadable modules that can be compiled independently.
    - voltagex_ 11y ago
      This is off topic, but I think there should be more tutorials for developers along the lines of "how to make life easier for packagers". Sidenote: creating Debian packages is an intimidating task.
      - snuxoll 11y ago
        1. Don't use a million dependencies that aren't already included in the majority of distributions. As a packager, I have no desire to maintain 20 complicated libraries you decided to depend on just to include your application in the package collection. A couple is fine, and the flatter your dependency tree is the less painful it is for a package maintainer to deal with. If you must include a dependency then please keep the transitive ones under control! 2. Use a standard build/packaging system. autotools, cmake, scons, setuptools/distutils, maven, gem (read: not bundler), cpan, etc. Whatever is considered a "standard" way of building and releasing software in the language you are writing it in is acceptable, using hacked together Makefiles and build scripts is generally a quick way to make packagers hate you and not want to waste time getting your software built. 3. Don't vendor dependencies - if you do then support the use of system-installed libraries in their place. I won't package software that insists on vending dependencies, I'm happy to package an extra library or two but I don't have that option if you don't support it. 4. Please support a standard installation mechanism if at all possible. I can relocate your build artifacts to the buildroot if necessary, but then I have to maintain it as the upstream changes - and distributions may be inconsistent with each other. If you are using a standard build system this should come for practically free unless you've done some nasty hacks, in which case please reconsider the modifications you have made. Debian packaging is pretty simple but I find the decision to use Makefile's for debian/rules to be a little annoying. rpmspec's are a lot clearer and rpm macro's are a lot easier to intuit since you can easily use rpmspec -e to see what they expand to, versus the blackbox that is the debhelper scripts sometimes. If you are new to packaging seriously take a look at making RPM's your first stop, rpmdev-newspec has a lot of templates for spec files that make getting started with any project that uses a standard build system super-easy (automake, cmake, python, ruby, perl, etc).

by cbr 11y ago

falcolas 11y ago

I would think that containers would help obviate such concerns, since it would be rather trivial to keep even a hundred images in place for an automated compilation run. Of course, serving the output would be harder, but still not impossible.

cbr 11y ago
Generating all those binaries would still be a pain, but you're right that containers could help. We'd also need to write a script that looked at your nginx install and figured out what would be the right precompiled module to download.
xorcist 11y ago
How does containers help? Building a hundred different nginx's shouldn't be neither harder nor easier if you stuff the binary in a container or not.
- falcolas 11y ago
  It gives you one image per configuration, instead of one build machine, or one stored VM. It makes it easier for a single machine to iterate over dozens or hundreds of build environments with a minimum of overhead. For example, one Ubuntu machine with Docker can create build environments for every Linux based distro (and version thereof) that exists, and iterate through all of them when it comes time to build. The setup is not trivial, though it is easier than creating a similar environment with VMs or physical machines.
  - snuxoll 11y ago
    Every distribution on the planet handles this with chroots for doing builds and it works out just fine without needing to do gymnastics with docker. Fedora, for example, has a farm of build hosts that are architecture specific (so there is no cross-compiling outside of bootstrapping a new build target), from there a new chroot gets created for every build and the build runs isolated within it. They've been using this same system to maintain packages for up to 7 different releases at once (counting the EPEL repositories) with no issues, I fail to see any benefit docker brings to the table here - and really, nginx should just provide a stable ABI.

LukeShu 11y ago

So does nginx still have to know about all of the modules at compile-time? Or can I compile nginx, then later compile a module for that nginx?

cbr 11y ago
That's fine, and is something you'd be able to do with PageSpeed. (But I'd really love a way to distribute binary modules anyone could just use, with no additional compilation.)

snuxoll 11y ago

Actually, it's still a pain for package maintainers. Take a look at mod_wsgi for apache as an example, it's maintained as a completely separate package (the rpmspec and dist-git trees are totally independent from apache) - when the httpd package gets upgraded mod_wsgi doesn't have to be rebuilt at all. This is important, in fact, because rebuilding a package means updating the spec file, bumping the release, and submitting a new build to koji - not something you want to do for loadable modules every time httpd is updated. This is going to create headaches for anyone wanting to package nginx, either they will have to compile all the modules when building nginx (creating many sub-packages) or there will have to be coordination between package maintainers to do a mass rebuild of all loadable modules every time the nginx package gets bumped. Neither situation is desirable, and I hope they release a stable ABI because until then nobody is going to waste time packaging loadable modules.

Florin_Andrei 11y ago
Wow. That basically means their current module scheme is useless for package maintainers.
- snuxoll 11y ago
  Pretty much. Like I said, technically you could just have the build for nginx also include all of the modules and just split them up into sub-packages, but that defeats the entire point of having loadable modules that can be compiled independently.
  - voltagex_ 11y ago
    This is off topic, but I think there should be more tutorials for developers along the lines of "how to make life easier for packagers". Sidenote: creating Debian packages is an intimidating task.
    - snuxoll 11y ago
      1. Don't use a million dependencies that aren't already included in the majority of distributions. As a packager, I have no desire to maintain 20 complicated libraries you decided to depend on just to include your application in the package collection. A couple is fine, and the flatter your dependency tree is the less painful it is for a package maintainer to deal with. If you must include a dependency then please keep the transitive ones under control! 2. Use a standard build/packaging system. autotools, cmake, scons, setuptools/distutils, maven, gem (read: not bundler), cpan, etc. Whatever is considered a "standard" way of building and releasing software in the language you are writing it in is acceptable, using hacked together Makefiles and build scripts is generally a quick way to make packagers hate you and not want to waste time getting your software built. 3. Don't vendor dependencies - if you do then support the use of system-installed libraries in their place. I won't package software that insists on vending dependencies, I'm happy to package an extra library or two but I don't have that option if you don't support it. 4. Please support a standard installation mechanism if at all possible. I can relocate your build artifacts to the buildroot if necessary, but then I have to maintain it as the upstream changes - and distributions may be inconsistent with each other. If you are using a standard build system this should come for practically free unless you've done some nasty hacks, in which case please reconsider the modifications you have made. Debian packaging is pretty simple but I find the decision to use Makefile's for debian/rules to be a little annoying. rpmspec's are a lot clearer and rpm macro's are a lot easier to intuit since you can easily use rpmspec -e to see what they expand to, versus the blackbox that is the debhelper scripts sometimes. If you are new to packaging seriously take a look at making RPM's your first stop, rpmdev-newspec has a lot of templates for spec files that make getting started with any project that uses a standard build system super-easy (automake, cmake, python, ruby, perl, etc).