2 ms·

I agree. I think the difference has been that until recently "the land of CI/CD" and so on has been certain segments of the corporate world, and not how typical

I agree. I think the difference has been that until recently "the land of CI/CD" and so on has been certain segments of the corporate world, and not how typical open source developers did things*. So when the former developed new technologies and new languages, they created build tools for them that anticipated being used in the ways that they usually produce software. The "problem", in the sense that it's a problem, is that these languages and related technologies are all pretty good! And so it's understandable that many developers who would traditionally be in the open source ecosystem want to use them. As a result they end up creating software that can't easily be shipped in traditional distributions. Ecosystem fragmentation is the unavoidable result. * By typical open source developers, I mean the sort of developers (and their development practices) that produced most of the software on my computer. I don't mean Firefox: Mozilla and Google have much more standard corporate development practices despite both producing quite a bit of open source software.

tialaramex 6y ago
Although continuous integration starts in proprietary software, it's been present in Free Software for at least decades. Netscape may well be the second or third medium-large software outfit to do continuous integration the way it's done today (we know Microsoft had a team doing this by hand every single day for Windows NT but that's completely insane) because some of its team had experienced this approach elsewhere and knew they needed it if they wanted to ship software that actually works. When Mozilla was created, Tinderbox (that system) along with the Mozilla browser (and so today Firefox) and Bugzilla (a bug tracker) were freed. I know it probably seems like last week, but that was more than twenty years ago.
Ericson2314 6y ago
I like Nixpkgs and NixOS for understanding both worlds. The Nix ecosystem's best path to mainstream success is being that go-between for everyone.
josephg 6y ago
> The "problem", in the sense that it's a problem, is that these languages and related technologies are all pretty good! Yes; and frankly the development ecosystem for making software for linux & friends on top of apt/etc is terrible - at least from the perspective of a modern professional software engineer. The assumption is C, and C programs have no package manager - so of course dependencies get bundled / vendored sometimes when the alternative is linking to potentially out of date dependencies in apt. Autoconf/automake is awful to learn and understand. CMake is better - but its horrendously complicated because it tries to solve the impossible job of paving over all the junky custom compilation scripts that came before. (And it still has no cargo equivalent for actually fetching your deps.) And then to work around all of that, each distribution will make weird, custom, maybe buggy patches to your software before adding it to their package managers. (Which has caused some high profile bugs and security issues a number of times.) Now when there's a bug, nobody knows who's fault it is! This worked in a world when there wasn't much software, when releases were rare and when most programs only had one or two dependencies. None of these properties are true any more. Rust, go, python and nodejs don't fit well with linux's package managers. The obvious alternative would be putting every crate, gem, pip package and npm package into apt, rpm and all the rest. And keeping them up to date with every version. But lets be real - that would be horrible. Apt et al aren't (currently) up to the task. (Can you imagine every npm package needing a maintainer in apt alone? I can just imagine the github issues: "I'm in debian stable and this transitive dep you're using only has version 0.1 available, from 6 years ago. What do I do?". Yikes.) I'm sympathetic to the argument that modern million dependency software development has its own problems; but right now it (sadly) has no competition in terms of ergonomics and build reliability.
- zrm 6y ago
  > This worked in a world when there wasn't much software, when releases were rare and when most programs only had one or two dependencies. None of these properties are true any more. I don't think this is even the problem. It's that upstream maintainers have stopped worrying about compatibility. Once upon a time you would have regular minor releases of some package. 3.0.2, 3.0.3, 3.0.4, but they were all backwards compatible. If you had version 3.0.4 and some software that was built against 3.0.2, it still worked against 3.0.4 because the only difference was that things were added or compatibly improved, not removed or incompatibly changed. Version 3.0.x wasn't compatible with version 2.9.x, but then the package maintainer for the distribution only has to package versions 3.0.4 and 2.9.16, i.e. suitably recent minor versions of each compatibility revision. Compatibility revisions so old that nobody relevant uses them anymore can be ignored, so they only had to package two or three incompatible versions which together are compatible with everything in active use. The problem today is that everything is a compatibility-breaking change, so there are dozens of releases from this year alone that are all mutually incompatible and would have to be packaged separately. And that doesn't scale.
  - josephg 6y ago
    Interesting - I didn’t know that. Can you give some examples?
    - zrm 6y ago
      https://wiki.openssl.org/index.php/Versioning https://wiki.openssl.org/index.php/Versioning https://developer.gnome.org/gtk3/stable/gtk3-Feature-Test-Macros.html https://developer.gnome.org/gtk3/stable/gtk3-Feature-Test-Ma... https://everything.curl.dev/libcurl/api https://everything.curl.dev/libcurl/api
- drran 6y ago
  So, you deliberately chose distro intended for users, with low version churn, instead of distro intended for developers, e.g. Fedora, which ships even pre-release versions sometimes, and now you blame ... apt? Just curious, what you are using for coding? MS Word or Excel? For example, Linus uses Fedora/MATE/Emacs. It's relatively easy to convert between packages between different packagers/distros. Automatic converters are exists for deb/rpm/pip/cpan/ctan/cargo, so it's easy to convert all existing packages into one packaging system and drop all of them into huge mono-repo. Yes, it's much easier to throw a new version to repo in many non-linux repositories. It's the equivalent of rolling distros, such as Arch, or development version of distro, such as Rawhide in Fedora or Sid in Debian. However, it also much easier to: break the world and make into news, distribute keyloggers and steal passwords and keys, forget to backport security fixes to users of older versions, pivot into completely orthogonal thing, etc. No code reviews means no responsibility.

tialaramex 6y ago

Although continuous integration starts in proprietary software, it's been present in Free Software for at least decades. Netscape may well be the second or third medium-large software outfit to do continuous integration the way it's done today (we know Microsoft had a team doing this by hand every single day for Windows NT but that's completely insane) because some of its team had experienced this approach elsewhere and knew they needed it if they wanted to ship software that actually works. When Mozilla was created, Tinderbox (that system) along with the Mozilla browser (and so today Firefox) and Bugzilla (a bug tracker) were freed. I know it probably seems like last week, but that was more than twenty years ago.

Ericson2314 6y ago

I like Nixpkgs and NixOS for understanding both worlds. The Nix ecosystem's best path to mainstream success is being that go-between for everyone.

josephg 6y ago

> The "problem", in the sense that it's a problem, is that these languages and related technologies are all pretty good! Yes; and frankly the development ecosystem for making software for linux & friends on top of apt/etc is terrible - at least from the perspective of a modern professional software engineer. The assumption is C, and C programs have no package manager - so of course dependencies get bundled / vendored sometimes when the alternative is linking to potentially out of date dependencies in apt. Autoconf/automake is awful to learn and understand. CMake is better - but its horrendously complicated because it tries to solve the impossible job of paving over all the junky custom compilation scripts that came before. (And it still has no cargo equivalent for actually fetching your deps.) And then to work around all of that, each distribution will make weird, custom, maybe buggy patches to your software before adding it to their package managers. (Which has caused some high profile bugs and security issues a number of times.) Now when there's a bug, nobody knows who's fault it is! This worked in a world when there wasn't much software, when releases were rare and when most programs only had one or two dependencies. None of these properties are true any more. Rust, go, python and nodejs don't fit well with linux's package managers. The obvious alternative would be putting every crate, gem, pip package and npm package into apt, rpm and all the rest. And keeping them up to date with every version. But lets be real - that would be horrible. Apt et al aren't (currently) up to the task. (Can you imagine every npm package needing a maintainer in apt alone? I can just imagine the github issues: "I'm in debian stable and this transitive dep you're using only has version 0.1 available, from 6 years ago. What do I do?". Yikes.) I'm sympathetic to the argument that modern million dependency software development has its own problems; but right now it (sadly) has no competition in terms of ergonomics and build reliability.

zrm 6y ago
> This worked in a world when there wasn't much software, when releases were rare and when most programs only had one or two dependencies. None of these properties are true any more. I don't think this is even the problem. It's that upstream maintainers have stopped worrying about compatibility. Once upon a time you would have regular minor releases of some package. 3.0.2, 3.0.3, 3.0.4, but they were all backwards compatible. If you had version 3.0.4 and some software that was built against 3.0.2, it still worked against 3.0.4 because the only difference was that things were added or compatibly improved, not removed or incompatibly changed. Version 3.0.x wasn't compatible with version 2.9.x, but then the package maintainer for the distribution only has to package versions 3.0.4 and 2.9.16, i.e. suitably recent minor versions of each compatibility revision. Compatibility revisions so old that nobody relevant uses them anymore can be ignored, so they only had to package two or three incompatible versions which together are compatible with everything in active use. The problem today is that everything is a compatibility-breaking change, so there are dozens of releases from this year alone that are all mutually incompatible and would have to be packaged separately. And that doesn't scale.
- josephg 6y ago
  Interesting - I didn’t know that. Can you give some examples?
  - zrm 6y ago
    https://wiki.openssl.org/index.php/Versioning https://wiki.openssl.org/index.php/Versioning https://developer.gnome.org/gtk3/stable/gtk3-Feature-Test-Macros.html https://developer.gnome.org/gtk3/stable/gtk3-Feature-Test-Ma... https://everything.curl.dev/libcurl/api https://everything.curl.dev/libcurl/api
drran 6y ago
So, you deliberately chose distro intended for users, with low version churn, instead of distro intended for developers, e.g. Fedora, which ships even pre-release versions sometimes, and now you blame ... apt? Just curious, what you are using for coding? MS Word or Excel? For example, Linus uses Fedora/MATE/Emacs. It's relatively easy to convert between packages between different packagers/distros. Automatic converters are exists for deb/rpm/pip/cpan/ctan/cargo, so it's easy to convert all existing packages into one packaging system and drop all of them into huge mono-repo. Yes, it's much easier to throw a new version to repo in many non-linux repositories. It's the equivalent of rolling distros, such as Arch, or development version of distro, such as Rawhide in Fedora or Sid in Debian. However, it also much easier to: break the world and make into news, distribute keyloggers and steal passwords and keys, forget to backport security fixes to users of older versions, pivot into completely orthogonal thing, etc. No code reviews means no responsibility.