5 ms·

> Two different communities create two very different kinds of software, which run on the same systems, but are created and distributed in very different ways.

by jauer 6y ago

> Two different communities create two very different kinds of software, which run on the same systems, but are created and distributed in very different ways. This is where the disconnect is coming from. The distro maintainers are coming from a world of multi-user systems where backwards compatibility and updating deps without disturbing a user's workload / forcing them to recompile is paramount. Go (and a fair amount of rust/python work) come from the land of CI/CD and, to a lesser extent, monorepos. When you are rebuilding the world above a bare minimum of the OS literally on every commit (or at least several times per day), it's easier to reason about code that is running if you can look at the commit that a bin was built from and know exactly what's inside (including all deps).

bscphil 6y ago
I agree. I think the difference has been that until recently "the land of CI/CD" and so on has been certain segments of the corporate world, and not how typical open source developers did things*. So when the former developed new technologies and new languages, they created build tools for them that anticipated being used in the ways that they usually produce software. The "problem", in the sense that it's a problem, is that these languages and related technologies are all pretty good! And so it's understandable that many developers who would traditionally be in the open source ecosystem want to use them. As a result they end up creating software that can't easily be shipped in traditional distributions. Ecosystem fragmentation is the unavoidable result. * By typical open source developers, I mean the sort of developers (and their development practices) that produced most of the software on my computer. I don't mean Firefox: Mozilla and Google have much more standard corporate development practices despite both producing quite a bit of open source software.
- tialaramex 6y ago
  Although continuous integration starts in proprietary software, it's been present in Free Software for at least decades. Netscape may well be the second or third medium-large software outfit to do continuous integration the way it's done today (we know Microsoft had a team doing this by hand every single day for Windows NT but that's completely insane) because some of its team had experienced this approach elsewhere and knew they needed it if they wanted to ship software that actually works. When Mozilla was created, Tinderbox (that system) along with the Mozilla browser (and so today Firefox) and Bugzilla (a bug tracker) were freed. I know it probably seems like last week, but that was more than twenty years ago.
- Ericson2314 6y ago
  I like Nixpkgs and NixOS for understanding both worlds. The Nix ecosystem's best path to mainstream success is being that go-between for everyone.
- josephg 6y ago
  > The "problem", in the sense that it's a problem, is that these languages and related technologies are all pretty good! Yes; and frankly the development ecosystem for making software for linux & friends on top of apt/etc is terrible - at least from the perspective of a modern professional software engineer. The assumption is C, and C programs have no package manager - so of course dependencies get bundled / vendored sometimes when the alternative is linking to potentially out of date dependencies in apt. Autoconf/automake is awful to learn and understand. CMake is better - but its horrendously complicated because it tries to solve the impossible job of paving over all the junky custom compilation scripts that came before. (And it still has no cargo equivalent for actually fetching your deps.) And then to work around all of that, each distribution will make weird, custom, maybe buggy patches to your software before adding it to their package managers. (Which has caused some high profile bugs and security issues a number of times.) Now when there's a bug, nobody knows who's fault it is! This worked in a world when there wasn't much software, when releases were rare and when most programs only had one or two dependencies. None of these properties are true any more. Rust, go, python and nodejs don't fit well with linux's package managers. The obvious alternative would be putting every crate, gem, pip package and npm package into apt, rpm and all the rest. And keeping them up to date with every version. But lets be real - that would be horrible. Apt et al aren't (currently) up to the task. (Can you imagine every npm package needing a maintainer in apt alone? I can just imagine the github issues: "I'm in debian stable and this transitive dep you're using only has version 0.1 available, from 6 years ago. What do I do?". Yikes.) I'm sympathetic to the argument that modern million dependency software development has its own problems; but right now it (sadly) has no competition in terms of ergonomics and build reliability.
  - zrm 6y ago
    > This worked in a world when there wasn't much software, when releases were rare and when most programs only had one or two dependencies. None of these properties are true any more. I don't think this is even the problem. It's that upstream maintainers have stopped worrying about compatibility. Once upon a time you would have regular minor releases of some package. 3.0.2, 3.0.3, 3.0.4, but they were all backwards compatible. If you had version 3.0.4 and some software that was built against 3.0.2, it still worked against 3.0.4 because the only difference was that things were added or compatibly improved, not removed or incompatibly changed. Version 3.0.x wasn't compatible with version 2.9.x, but then the package maintainer for the distribution only has to package versions 3.0.4 and 2.9.16, i.e. suitably recent minor versions of each compatibility revision. Compatibility revisions so old that nobody relevant uses them anymore can be ignored, so they only had to package two or three incompatible versions which together are compatible with everything in active use. The problem today is that everything is a compatibility-breaking change, so there are dozens of releases from this year alone that are all mutually incompatible and would have to be packaged separately. And that doesn't scale.
mwcampbell 6y ago
This makes me wonder if there's a Linux base system suitable for servers that embraces the newer approach. That is, a minimal base system that's built for immutable container images, providing just what's needed to bootstrap the current generation of language-specific build and package systems. The Alpine Linux Docker images might be a good choice for now, but IIUC, Alpine Linux itself still embraces the older distro approach.
- MetaDark 6y ago
  Maybe the Nix package manager / NixOS is what you're looking for? I think it takes the best features from both worlds. Every package installed with Nix is isolated into content-addressable* directories, so for example, my install of Firefox is located at /nix/store/c7pmng2x05dkigpbhnjs8fdzd8kk31np-firefox-85.0.2/bin/firefox. This is pretty inconvenient to use directly, so Nix generates a profile that symlinks all your packages into one place (eg. /run/current/system/sw, ~/.nix-profile), and then environment variables like PATH can just include <PROFILE_DIR>/bin. With this approach, I can have multiple versions of the same package installed simultaneously, without them conflicting with each other. Like in a traditional distro, any dependencies that are shared between packages aren't duplicated, but if a package needs to explicitly depend on a different version, it can. Also, because Nix is designed as a functional package manager for building packages from source (even though it has a binary cache), you can trace back exactly what sources were used to build your package and its dependencies, all the way back to the bootstrap binaries used to build any self-hosting compilers (gcc, rust, openjdk, ...) * Most packages use a hash that's generated from the inputs used to build it, rather than the output that's generated.
  - vnlp 6y ago
    Sounds like another idea of djb took off (the slashpackage hierarchy).
    - MetaDark 6y ago
      I'm not sure what the slashpackage hierarchy is, but Nix & NixOS started in 2003.
baybal2 6y ago
> it's easier to reason about code that is running if you can look at the commit that a bin was built from and know exactly what's inside (including all deps). Believe me, it's usually the opposite. Lack of proper releases, testing, and versioning results in unending checkout-fu to figure out what commits for each of 20 libraries will work for each other. The idea is plainly stupid, without any redeeming qualities. The entirety of this cargo cult hinges on the point of "If people calling it a genius invention for last 8-10 years admit it not being it, some major reputation, and cred loss will be incurred"
- jauer 6y ago
  > Lack of proper releases, testing, and versioning results in unending checkout-fu to figure out what commits for each of 20 libraries will work for each other. Admittedly I'm basing this on my experience in a very large monorepo environment, but there's no figuring out which commits will work with each other. Every commit with every library will work, otherwise it doesn't get committed. Yes, this involves massive CI infra and tooling to aid in refactoring. You want to make a breaking change to a lib? Great, it's on you to update every piece of code that calls it. It's great when you can control every piece of your infra, but I totally get how it's unfeasible (and maintainer hell) for the distro community.
  - loopz 6y ago
    Distros are great for off-the-shelf software especially if you don't care which version you get too much. When versions matter, you quickly get into dependency-hell. So long-lived software tend to stabilize, and then remain unchanged. K8s, go and even ruby, tend to change and evolve. It's usually a bad idea to pull such software from distros then, even if available. The means to get software is simply too different, and it's a non-problem for everyone but completist distro maintainers.
- rictic 6y ago
  Are you actually talking about static linking here, or just venting your spleen about sloppiness in software engineering practice more generally? Because tracking dependencies in source control (specifically, checking in lock files) is tremendous for reproducibility. It means that when bisecting through the commit history to find when a problem began is not just bisecting through the local source code, but also the specific versions of every dependency. So regardless of whether the issue you're investigating is in the package, a dependency, or an unexpected interaction between the two, you're able to find the first commit that introduced the issue in O(log(commits)) time, rather than needing O(commits * (num dependencies * dependency versions)) time.
pjmlp 6y ago
Additionally, when you have languages with rich library ecosystems, the OS kind of becomes irrelevant, the platform is the language ecosystem. Just to pick Go as an example (not to be lost discussing VMs and such), it doesn't matter if I am targeting bare metal, Linux, Windows, IBM z/OS, AWS special cloud runtime, whatever. As long as the Go code is the same, and someone has done the low level runtime support, it is a compile away and done. Finally by pushing containers no matter what, the Linux community has made this even easier.
- rini17 6y ago
  What if there's another software in another language you want to interoperate with? What if you want to avoid containers with their complexity and dubious security record?
  - pjmlp 6y ago
    Some form of IPC, usually OS agnostic ones. That is what I have been doing the last 20 years, in the context of C++, Java and .NET.

by jauer 6y ago

bscphil 6y ago

I agree. I think the difference has been that until recently "the land of CI/CD" and so on has been certain segments of the corporate world, and not how typical open source developers did things*. So when the former developed new technologies and new languages, they created build tools for them that anticipated being used in the ways that they usually produce software. The "problem", in the sense that it's a problem, is that these languages and related technologies are all pretty good! And so it's understandable that many developers who would traditionally be in the open source ecosystem want to use them. As a result they end up creating software that can't easily be shipped in traditional distributions. Ecosystem fragmentation is the unavoidable result. * By typical open source developers, I mean the sort of developers (and their development practices) that produced most of the software on my computer. I don't mean Firefox: Mozilla and Google have much more standard corporate development practices despite both producing quite a bit of open source software.

tialaramex 6y ago
Although continuous integration starts in proprietary software, it's been present in Free Software for at least decades. Netscape may well be the second or third medium-large software outfit to do continuous integration the way it's done today (we know Microsoft had a team doing this by hand every single day for Windows NT but that's completely insane) because some of its team had experienced this approach elsewhere and knew they needed it if they wanted to ship software that actually works. When Mozilla was created, Tinderbox (that system) along with the Mozilla browser (and so today Firefox) and Bugzilla (a bug tracker) were freed. I know it probably seems like last week, but that was more than twenty years ago.
Ericson2314 6y ago
I like Nixpkgs and NixOS for understanding both worlds. The Nix ecosystem's best path to mainstream success is being that go-between for everyone.
josephg 6y ago
> The "problem", in the sense that it's a problem, is that these languages and related technologies are all pretty good! Yes; and frankly the development ecosystem for making software for linux & friends on top of apt/etc is terrible - at least from the perspective of a modern professional software engineer. The assumption is C, and C programs have no package manager - so of course dependencies get bundled / vendored sometimes when the alternative is linking to potentially out of date dependencies in apt. Autoconf/automake is awful to learn and understand. CMake is better - but its horrendously complicated because it tries to solve the impossible job of paving over all the junky custom compilation scripts that came before. (And it still has no cargo equivalent for actually fetching your deps.) And then to work around all of that, each distribution will make weird, custom, maybe buggy patches to your software before adding it to their package managers. (Which has caused some high profile bugs and security issues a number of times.) Now when there's a bug, nobody knows who's fault it is! This worked in a world when there wasn't much software, when releases were rare and when most programs only had one or two dependencies. None of these properties are true any more. Rust, go, python and nodejs don't fit well with linux's package managers. The obvious alternative would be putting every crate, gem, pip package and npm package into apt, rpm and all the rest. And keeping them up to date with every version. But lets be real - that would be horrible. Apt et al aren't (currently) up to the task. (Can you imagine every npm package needing a maintainer in apt alone? I can just imagine the github issues: "I'm in debian stable and this transitive dep you're using only has version 0.1 available, from 6 years ago. What do I do?". Yikes.) I'm sympathetic to the argument that modern million dependency software development has its own problems; but right now it (sadly) has no competition in terms of ergonomics and build reliability.
- zrm 6y ago
  > This worked in a world when there wasn't much software, when releases were rare and when most programs only had one or two dependencies. None of these properties are true any more. I don't think this is even the problem. It's that upstream maintainers have stopped worrying about compatibility. Once upon a time you would have regular minor releases of some package. 3.0.2, 3.0.3, 3.0.4, but they were all backwards compatible. If you had version 3.0.4 and some software that was built against 3.0.2, it still worked against 3.0.4 because the only difference was that things were added or compatibly improved, not removed or incompatibly changed. Version 3.0.x wasn't compatible with version 2.9.x, but then the package maintainer for the distribution only has to package versions 3.0.4 and 2.9.16, i.e. suitably recent minor versions of each compatibility revision. Compatibility revisions so old that nobody relevant uses them anymore can be ignored, so they only had to package two or three incompatible versions which together are compatible with everything in active use. The problem today is that everything is a compatibility-breaking change, so there are dozens of releases from this year alone that are all mutually incompatible and would have to be packaged separately. And that doesn't scale.

mwcampbell 6y ago

This makes me wonder if there's a Linux base system suitable for servers that embraces the newer approach. That is, a minimal base system that's built for immutable container images, providing just what's needed to bootstrap the current generation of language-specific build and package systems. The Alpine Linux Docker images might be a good choice for now, but IIUC, Alpine Linux itself still embraces the older distro approach.

MetaDark 6y ago
Maybe the Nix package manager / NixOS is what you're looking for? I think it takes the best features from both worlds. Every package installed with Nix is isolated into content-addressable* directories, so for example, my install of Firefox is located at /nix/store/c7pmng2x05dkigpbhnjs8fdzd8kk31np-firefox-85.0.2/bin/firefox. This is pretty inconvenient to use directly, so Nix generates a profile that symlinks all your packages into one place (eg. /run/current/system/sw, ~/.nix-profile), and then environment variables like PATH can just include <PROFILE_DIR>/bin. With this approach, I can have multiple versions of the same package installed simultaneously, without them conflicting with each other. Like in a traditional distro, any dependencies that are shared between packages aren't duplicated, but if a package needs to explicitly depend on a different version, it can. Also, because Nix is designed as a functional package manager for building packages from source (even though it has a binary cache), you can trace back exactly what sources were used to build your package and its dependencies, all the way back to the bootstrap binaries used to build any self-hosting compilers (gcc, rust, openjdk, ...) * Most packages use a hash that's generated from the inputs used to build it, rather than the output that's generated.
- vnlp 6y ago
  Sounds like another idea of djb took off (the slashpackage hierarchy).
  - MetaDark 6y ago
    I'm not sure what the slashpackage hierarchy is, but Nix & NixOS started in 2003.

baybal2 6y ago

> it's easier to reason about code that is running if you can look at the commit that a bin was built from and know exactly what's inside (including all deps). Believe me, it's usually the opposite. Lack of proper releases, testing, and versioning results in unending checkout-fu to figure out what commits for each of 20 libraries will work for each other. The idea is plainly stupid, without any redeeming qualities. The entirety of this cargo cult hinges on the point of "If people calling it a genius invention for last 8-10 years admit it not being it, some major reputation, and cred loss will be incurred"

jauer 6y ago
> Lack of proper releases, testing, and versioning results in unending checkout-fu to figure out what commits for each of 20 libraries will work for each other. Admittedly I'm basing this on my experience in a very large monorepo environment, but there's no figuring out which commits will work with each other. Every commit with every library will work, otherwise it doesn't get committed. Yes, this involves massive CI infra and tooling to aid in refactoring. You want to make a breaking change to a lib? Great, it's on you to update every piece of code that calls it. It's great when you can control every piece of your infra, but I totally get how it's unfeasible (and maintainer hell) for the distro community.
- loopz 6y ago
  Distros are great for off-the-shelf software especially if you don't care which version you get too much. When versions matter, you quickly get into dependency-hell. So long-lived software tend to stabilize, and then remain unchanged. K8s, go and even ruby, tend to change and evolve. It's usually a bad idea to pull such software from distros then, even if available. The means to get software is simply too different, and it's a non-problem for everyone but completist distro maintainers.
rictic 6y ago
Are you actually talking about static linking here, or just venting your spleen about sloppiness in software engineering practice more generally? Because tracking dependencies in source control (specifically, checking in lock files) is tremendous for reproducibility. It means that when bisecting through the commit history to find when a problem began is not just bisecting through the local source code, but also the specific versions of every dependency. So regardless of whether the issue you're investigating is in the package, a dependency, or an unexpected interaction between the two, you're able to find the first commit that introduced the issue in O(log(commits)) time, rather than needing O(commits * (num dependencies * dependency versions)) time.

pjmlp 6y ago

Additionally, when you have languages with rich library ecosystems, the OS kind of becomes irrelevant, the platform is the language ecosystem. Just to pick Go as an example (not to be lost discussing VMs and such), it doesn't matter if I am targeting bare metal, Linux, Windows, IBM z/OS, AWS special cloud runtime, whatever. As long as the Go code is the same, and someone has done the low level runtime support, it is a compile away and done. Finally by pushing containers no matter what, the Linux community has made this even easier.

rini17 6y ago
What if there's another software in another language you want to interoperate with? What if you want to avoid containers with their complexity and dubious security record?
- pjmlp 6y ago
  Some form of IPC, usually OS agnostic ones. That is what I have been doing the last 20 years, in the context of C++, Java and .NET.