4 ms·

Bypassing the developers say "version 1.15 is out now" instead of "version 1.15 is out: 1) Now + however many years until the next Debian stable 2) Now + howe

Bypassing the developers say "version 1.15 is out now" instead of "version 1.15 is out: 1) Now + however many years until the next Debian stable 2) Now + however many months until the next Ubuntu 3) Now for Arch users 4) Now + whatever for Fedora 5) Now + a different whatever for SUSE etc etc" Package-the-world-and-freeze distribution schemes are a black hole of suck for development tool projects that have their own ecosystem and want to iterate at a consistent pace. "Of course," you could say, "the devs should just run their own secondary repos instead of submitting to the official ones". So now instead of one bash script you just have to trust not to rootkit you, the devs distribute 10 different incompatible package formats you just have to trust not to rootkit you, at only 10x the maintenance, build script, and sysadmin burden as you run all these bespoke repos.

derefr 11y ago
Note that "package-the-world-and-freeze" distributions (Debian, RHEL, whoever else) have an entire infrastructure in place to creating their own line of packages from an "upstream's" basic-tarball releases. When Debian determines to, e.g., support Linux 3.6 for the next five years, it's not suddenly Linus's responsibility to backport kernel patches to Linux 3.6. Likewise, it's not your responsibility to maintain a stable branch of your software so that Debian can do less work packaging it. The other kind of model, of Ubuntu PPAs and such, where they're expected to track the developer's releases, frequently incentivize the developer to get those packages created as part of their own build infrastructure. But there's still no expectation that they will. The culture of the "distribution" is set up to allow developers to just develop, and distributors to be the ones to think about distribution.
- msbarnett 11y ago
  > Note that "package-the-world-and-freeze" distributions (Debian, RHEL, whoever else) have an entire infrastructure in place to creating their own line of packages from an "upstream's" basic-tarball releases. > When Debian determines to, e.g., support Linux 3.6 for the next five years, it's not suddenly Linus's responsibility to backport kernel patches to Linux 3.6. Likewise, it's not your responsibility to maintain a stable branch of your software so that Debian can do less work packaging it. That's great and all, but if I'm a Ruby dev my life isn't any better just because the burden of packaging a 4-year-out-of-date version of the runtime isn't on me. The Rails community wants to move to the latest Rails version, which wants to user modern Ruby syntax, which depends on a modern Ruby version, and the community doesn't just want to bifurcate and fork every single gem for half a decade because Debian's dragging it's feet about the next release. And I can't just order my users to use Debian Sid in a production environment. The fundamental problem here is that Distro-packaging is optimized to distribute software for an ecosystem comprised of that Distro's users. The problem a lot of dev tool projects run into is that they serve a cross-cutting community of many different OS's users, all of whom want to stay in sync with the rest of the cross-cutting community, and who couldn't give two shits about what some other person's OS's package repo's timelines are like. That's the niche that NPM, RVM, et al are serving and it's a niche that distro packages are completely unsuited to handle. And RVM et al tend to "package" as scripts because we're right back in this mess if Debian RVM users are 4 years behind OS X RVM users.
  - derefr 11y ago
    I think you're under a mistaken view of what OS distro packages exist for. The "system ruby" and other such packages aren't for end-user (or developer) consumption; they exist for the sake of other system packages, in case a piece of system software needs to run some Ruby code. "Application software"—the kind you deploy to a system using Chef or stick in a Docker container or somesuch—is expected to use vendored libraries and a vendored platform, the way e.g. Erlang releases do. Not just in development, but in production. This is why /opt exists. It's only once your software becomes "system software"—once it becomes something people expect to be a stable, black-box part of the OS rather than something they integrate into their solution—that you must pay attention to the versions of other "system software." OSes demand stability from system software for one simple reason: the ability to silently auto-update "the OS" in production to take out crashers and security vulnerabilities, without materially affecting the application software running atop it. If you want your package to successfully participate in the OS's zero-downtime auto-update magic, then your package needs to be stable and have stable deps. No new features, no breaking API changes, nothing that will break other system software that people are depending on in production. But if you don't have that requirement—if ABI-incompatible updates to your package just affect the developers consuming your library/tool, rather than users whose OSes suddenly break—then there's no need to integrate with the OS. Note that rvm, and rubygems or npm, and so forth, aren't "secondary OS package managers." They're developer tools, for developing application software. The end result that real operations engineers will expect out of you is a blob (maybe a private package, maybe a container, maybe just a tarball) containing the platform, the libraries, and your code, baked together. Production systems should not be running rvm, or gem/npm install, to get your software's deps installed! --- An interesting consequence of this philosophy, though, is that there are many "leaf packages" in most distros that shouldn't be system packages at all, but should rather be floating above the OS, consumed via a secondary "application software manager" that pulls down full pieces of self-contained vendored-deps software. OSX almost has the right idea by dividing the world between a low-level Unix system (all system software; all involving stable-version interdependency), and "application bundles" that embed their own vendored Frameworks. But they don't go far enough; there's no such thing as a "CLI application bundle" which gets probed on Spotlight discovery to provide a new bin/ directory for your $PATH, the way GUI application bundles get probed to provide new file-type associations. If there was, they could move a lot of their own "leaf nodes" out into CLI bundles. (The "Command Line Tools" package containing gcc et al. is a prime candidate for a CLI bundle. It's not system software; it's its own vendored build environment!) Despite what I said above, it might be perfectly sensible to run something like gem/npm on a production system—but only for requesting the "leaf packages" you'll actually be using on that machine—both applications and utilities. Such an "application package manager" could theoretically even do dependency-resolution—but only insofar as it would be doing it to optimize download time by breaking out things like "foo-1.0/vendor/[someframework]-1.1.13-[exactsha]" into its own slug so different apps' downloads could reuse it from cache. The result of running "app install [foo]" would still have to be a copy of foo-1.0 with its [someframework] vendored inside it, not a symlink or other such reference from foo to someframework. (Basically, if you get the idea, I'm talking about a cross between Homebrew Casks and Nix.) If you twist your mind just right, container solutions like Docker are the "application package manager" I'm talking about. But they actually provide too much isolation to run a lot of important application software (games, for example), because they're built under the assumption that you're running someone else's untrusted application software, and also on the assumption that the environment above the container isn't itself a secure/ephemeral sandbox like a VM. It'd be pretty easy, though, to take the ideas from container management, and use them to create a one-true-tool for creating, deploying, and managing the lifecycle of "releases" of application software in an OS-neutral way. While rvm(1) shouldn't be installed in production, appvm(1) could very well be.
brazzledazzle 11y ago
I think the point was less about a rootkit and more about maintainability. That said, a secondary repo hosted on SSL with trusted keys is a lot more secure than piping a curl'd script into bash. If you trust the application developer but not the infrastructure in-between anyway.

derefr 11y ago

Note that "package-the-world-and-freeze" distributions (Debian, RHEL, whoever else) have an entire infrastructure in place to creating their own line of packages from an "upstream's" basic-tarball releases. When Debian determines to, e.g., support Linux 3.6 for the next five years, it's not suddenly Linus's responsibility to backport kernel patches to Linux 3.6. Likewise, it's not your responsibility to maintain a stable branch of your software so that Debian can do less work packaging it. The other kind of model, of Ubuntu PPAs and such, where they're expected to track the developer's releases, frequently incentivize the developer to get those packages created as part of their own build infrastructure. But there's still no expectation that they will. The culture of the "distribution" is set up to allow developers to just develop, and distributors to be the ones to think about distribution.

msbarnett 11y ago
> Note that "package-the-world-and-freeze" distributions (Debian, RHEL, whoever else) have an entire infrastructure in place to creating their own line of packages from an "upstream's" basic-tarball releases. > When Debian determines to, e.g., support Linux 3.6 for the next five years, it's not suddenly Linus's responsibility to backport kernel patches to Linux 3.6. Likewise, it's not your responsibility to maintain a stable branch of your software so that Debian can do less work packaging it. That's great and all, but if I'm a Ruby dev my life isn't any better just because the burden of packaging a 4-year-out-of-date version of the runtime isn't on me. The Rails community wants to move to the latest Rails version, which wants to user modern Ruby syntax, which depends on a modern Ruby version, and the community doesn't just want to bifurcate and fork every single gem for half a decade because Debian's dragging it's feet about the next release. And I can't just order my users to use Debian Sid in a production environment. The fundamental problem here is that Distro-packaging is optimized to distribute software for an ecosystem comprised of that Distro's users. The problem a lot of dev tool projects run into is that they serve a cross-cutting community of many different OS's users, all of whom want to stay in sync with the rest of the cross-cutting community, and who couldn't give two shits about what some other person's OS's package repo's timelines are like. That's the niche that NPM, RVM, et al are serving and it's a niche that distro packages are completely unsuited to handle. And RVM et al tend to "package" as scripts because we're right back in this mess if Debian RVM users are 4 years behind OS X RVM users.
- derefr 11y ago
  I think you're under a mistaken view of what OS distro packages exist for. The "system ruby" and other such packages aren't for end-user (or developer) consumption; they exist for the sake of other system packages, in case a piece of system software needs to run some Ruby code. "Application software"—the kind you deploy to a system using Chef or stick in a Docker container or somesuch—is expected to use vendored libraries and a vendored platform, the way e.g. Erlang releases do. Not just in development, but in production. This is why /opt exists. It's only once your software becomes "system software"—once it becomes something people expect to be a stable, black-box part of the OS rather than something they integrate into their solution—that you must pay attention to the versions of other "system software." OSes demand stability from system software for one simple reason: the ability to silently auto-update "the OS" in production to take out crashers and security vulnerabilities, without materially affecting the application software running atop it. If you want your package to successfully participate in the OS's zero-downtime auto-update magic, then your package needs to be stable and have stable deps. No new features, no breaking API changes, nothing that will break other system software that people are depending on in production. But if you don't have that requirement—if ABI-incompatible updates to your package just affect the developers consuming your library/tool, rather than users whose OSes suddenly break—then there's no need to integrate with the OS. Note that rvm, and rubygems or npm, and so forth, aren't "secondary OS package managers." They're developer tools, for developing application software. The end result that real operations engineers will expect out of you is a blob (maybe a private package, maybe a container, maybe just a tarball) containing the platform, the libraries, and your code, baked together. Production systems should not be running rvm, or gem/npm install, to get your software's deps installed! --- An interesting consequence of this philosophy, though, is that there are many "leaf packages" in most distros that shouldn't be system packages at all, but should rather be floating above the OS, consumed via a secondary "application software manager" that pulls down full pieces of self-contained vendored-deps software. OSX almost has the right idea by dividing the world between a low-level Unix system (all system software; all involving stable-version interdependency), and "application bundles" that embed their own vendored Frameworks. But they don't go far enough; there's no such thing as a "CLI application bundle" which gets probed on Spotlight discovery to provide a new bin/ directory for your $PATH, the way GUI application bundles get probed to provide new file-type associations. If there was, they could move a lot of their own "leaf nodes" out into CLI bundles. (The "Command Line Tools" package containing gcc et al. is a prime candidate for a CLI bundle. It's not system software; it's its own vendored build environment!) Despite what I said above, it might be perfectly sensible to run something like gem/npm on a production system—but only for requesting the "leaf packages" you'll actually be using on that machine—both applications and utilities. Such an "application package manager" could theoretically even do dependency-resolution—but only insofar as it would be doing it to optimize download time by breaking out things like "foo-1.0/vendor/[someframework]-1.1.13-[exactsha]" into its own slug so different apps' downloads could reuse it from cache. The result of running "app install [foo]" would still have to be a copy of foo-1.0 with its [someframework] vendored inside it, not a symlink or other such reference from foo to someframework. (Basically, if you get the idea, I'm talking about a cross between Homebrew Casks and Nix.) If you twist your mind just right, container solutions like Docker are the "application package manager" I'm talking about. But they actually provide too much isolation to run a lot of important application software (games, for example), because they're built under the assumption that you're running someone else's untrusted application software, and also on the assumption that the environment above the container isn't itself a secure/ephemeral sandbox like a VM. It'd be pretty easy, though, to take the ideas from container management, and use them to create a one-true-tool for creating, deploying, and managing the lifecycle of "releases" of application software in an OS-neutral way. While rvm(1) shouldn't be installed in production, appvm(1) could very well be.

brazzledazzle 11y ago

I think the point was less about a rootkit and more about maintainability. That said, a secondary repo hosted on SSL with trusted keys is a lot more secure than piping a curl'd script into bash. If you trust the application developer but not the infrastructure in-between anyway.