5 ms·

There is a deep trend happening in software development. As the number of dependencies for building an application grows, it becomes exponentially harder to sh

There is a deep trend happening in software development. As the number of dependencies for building an application grows, it becomes exponentially harder to shake the tree. This used to be the role of Linux distributions, they were acting as a push-back force, asking projects to support multiple versions of C libraries. This was acceptable because there is not C package manager. Now that each language has their own package manager, the role of distributions have faded. They are even viewed as a form of nuisance by some developers. It's easier to support one fixed set of dependencies and not have to worry about backward-compatibility. This is something all distributions have been struggling with for a while now, especially with NodeJS. This trend is happening on all platforms, but is more pronounced in Linux because of the diversity of the system libraries ecosystem. On Windows and macOS, there are some SDKs the application can lean on. On Linux, the only stable "SDK" is the Linux kernel API.

vetinari 5y ago
> On Windows and macOS, there are some SDKs the application can lean on. On Linux, the only stable "SDK" is the Linux kernel API. Now, with Flatpak, each runtime is an SDK on its own. However, unlike Windows and macOS, specific runtime is not being bound to a specific OS release, but to the app requirement.
- zimbatm 5y ago
  That's right, it makes the problem worse. The first time I noticed that trend was on macOS, where applications bundle most of their libraries, and binaries compiled for multiple architectures. Then we had Electron apps that ship with their own full copy of a browser and dependent libraries. When NPM was designed, they observed that resolving colliding versions of the same dependency was sometimes difficult. Their answer was to remove that restriction and allow multiple versions of the same library in a project. Nowadays, it's not uncommon to have 1000+ dependencies in a simple hello world npm project. Our industry is moving away from that feedback force that was forcing developers to agree on interfaces and release stable APIs.
  - mschuster91 5y ago
    > Our industry is moving away from that feedback force that was forcing developers to agree on interfaces and release stable APIs. And parts of the industry are beginning to move back and favor stability, as a result of - numerous NPM packages being taken over by coin miners and other malware - at the scale of some of them, even a ten minute takeover window were millions of installs, and source chain audits are almost impossible - framework churn being an actual liability for large enterprises - many were burned when AngularJS fell out of love and now they were stuck with a tech stack widely considered "out of date". Most new projects these days seem to be ReactJS where Facebook's heavy involvement promises at least some long term support - developer / tooling churn - same as above, the constant amount of training to keep up with breaking changes may be tolerable at startup, money-burn phase, but once your company achieves a certain size it becomes untenable
southerntofu 5y ago
> Now that each language has their own package manager, the role of distributions have faded. Yet most language-specific package managers are deeply flawed compared to distro package managers. But i agree with your sentiment that beyond LTS for enterprise, the deb/rpm packaging paradigm is becoming obsolete. I believe nix/guix form an interesting new paradigm for packaging where there is still a trusted third party (the nix/guix community and repo) but where packaging doesn't get in the way of app developers. I'm especially interested in how these declarative package managers could "export" to more widely-used packaging schemes. guix can already export to .deb, but there's no reason it couldn't produce an AppImage or a flatpak.
simiones 5y ago
> On Windows and macOS, there are some SDKs the application can lean on. On Linux, the only stable "SDK" is the Linux kernel API. This is a very good observation I think. Basically, what Linux distributions have failed to do is to create a higher level SDK for their platforms - instead, they rely on a soup of libraries and on the package maintainer model to know which of these is which. Basically, they are failing to provide a true OS: there is only a Kernel + various user space apps, from systemd to KDevelop, with no explicit line in the sand between what is the system SDK and what is a simple app bundled with the OS.
- WastingMyTime89 5y ago
  > This is a very good observation I think. Basically, what Linux distributions have failed to do is to create a higher level SDK for their platforms - instead, they rely on a soup of libraries and on the package maintainer model to know which of these is which. Not really no. It's both simpler and more complicated. At the very low level, everyone expects to get a POSIX system. At the user level, most users use either Gnome or KDE (well mostly Gnome to be honest but let's pretend) which provides what could be called a highlevel SDK for applications. That leaves the layer inbetween which used to be somewhat distribution specific but is seeing more and more consolidation with the advent of systemd and flatpak.
  - simiones 5y ago
    Even Gnome and KDE are not really "high level SDKs". They mostly have GUI support of various kinds, but not a full SDK for interacting with the OS (e.g. interacting with networking, with other apps, with system services etc.).
- nonameiguess 5y ago
  I don't think this is really true. There is a fairly stable core set of C libraries present on virtually any Linux distro with a desktop environment that does most of what any application needs in terms of system services. The basic problem is more what the parent was discussing: these are C libraries, and rather than using the FFI for their app language of choice and using these pre-installed libraries, developers would rather rewrite all functionality in the same language as their applications, and the Rust crate, Go mod, or Node package you find when Googling whether something exists in your language of choice is likely to be much less stable than something like libcurl. Mac and Windows solve this by creating their own languages for app development and then porting the system services to be provided in Swift and C# or whatever in addition to the original C implementation. There is no corporation behind most Linux distros willing to do that other than Canonical and Redhat, but they're perfectly happy to just stick with C, as is Gnome (KDE being perfectly happy to stick with C++). Most app developers are not, however. For what it's worth, Redhat did kind of solve this problem for the Linux container ecosystem with libcontainer, rewriting all of the core functionality in Go, in recognition of the fact that this is the direction the ecosystem was moving in thanks to Docker and Kubernetes using Go, and now app developers for the CNCF ecosystem can just use that single set of core dependencies that stays pretty stable.
jcelerier 5y ago
> On Windows and macOS, there are some SDKs the application can lean on. On Linux, the only stable "SDK" is the Linux kernel API. In practice I don't really see a difference, I link against Qt on all platforms, win, mac, linux, wasm, android... and that's it.
maccolgan 5y ago
Dynamic linking is a creation of the devil.
jiggawatts 5y ago
I watched a great rant by Linus T himself on how distributing code for Linux is a "pain in the arse", but is comparatively easier on Windows and MacOS: https://www.youtube.com/watch?v=Pzl1B7nB9Kc https://www.youtube.com/watch?v=Pzl1B7nB9Kc Something I took away from that is that it is Linus himself, personally, that is responsible (in a way) for Docker existing. Docker depends on the stable kernel ABI in order to allow container images to be portable and "just work" stably and reliably. This is a "guarantee" that Linus gets shouty about if the kernel devs break it. Docker fills a need on Linux because the user-mode libraries across all distributions are a total mess. Microsoft is trying to be the "hip kid" by copying Docker into Windows, where everything is backwards to the Linux situations: The NT kernel ABI is treated as unstable, and changes as fast as every few months. No one ever codes directly against the kernel on Windows (for some values of "no one" and "ever".) The user-mode libraries like Win32 and .NET are very stable, papering over the inconsistency of the kernel library. You can run applications compiled in the year 2000 today, unmodified, and more often than not they'll "just work". There just isn't a "burning need" for Docker on Windows. People that try to reproduce the cool new Linux workflow however are in for a world of hurt, because they'll rapidly discover that the images they built just weeks ago might not run any more because the latest Windows Update bumped the kernel version. I read all the way through this and wept: https://docs.microsoft.com/en-us/virtualization/windowscontainers/deploy-containers/version-compatibility https://docs.microsoft.com/en-us/virtualization/windowsconta...

vetinari 5y ago

> On Windows and macOS, there are some SDKs the application can lean on. On Linux, the only stable "SDK" is the Linux kernel API. Now, with Flatpak, each runtime is an SDK on its own. However, unlike Windows and macOS, specific runtime is not being bound to a specific OS release, but to the app requirement.

zimbatm 5y ago
That's right, it makes the problem worse. The first time I noticed that trend was on macOS, where applications bundle most of their libraries, and binaries compiled for multiple architectures. Then we had Electron apps that ship with their own full copy of a browser and dependent libraries. When NPM was designed, they observed that resolving colliding versions of the same dependency was sometimes difficult. Their answer was to remove that restriction and allow multiple versions of the same library in a project. Nowadays, it's not uncommon to have 1000+ dependencies in a simple hello world npm project. Our industry is moving away from that feedback force that was forcing developers to agree on interfaces and release stable APIs.
- mschuster91 5y ago
  > Our industry is moving away from that feedback force that was forcing developers to agree on interfaces and release stable APIs. And parts of the industry are beginning to move back and favor stability, as a result of - numerous NPM packages being taken over by coin miners and other malware - at the scale of some of them, even a ten minute takeover window were millions of installs, and source chain audits are almost impossible - framework churn being an actual liability for large enterprises - many were burned when AngularJS fell out of love and now they were stuck with a tech stack widely considered "out of date". Most new projects these days seem to be ReactJS where Facebook's heavy involvement promises at least some long term support - developer / tooling churn - same as above, the constant amount of training to keep up with breaking changes may be tolerable at startup, money-burn phase, but once your company achieves a certain size it becomes untenable

southerntofu 5y ago

> Now that each language has their own package manager, the role of distributions have faded. Yet most language-specific package managers are deeply flawed compared to distro package managers. But i agree with your sentiment that beyond LTS for enterprise, the deb/rpm packaging paradigm is becoming obsolete. I believe nix/guix form an interesting new paradigm for packaging where there is still a trusted third party (the nix/guix community and repo) but where packaging doesn't get in the way of app developers. I'm especially interested in how these declarative package managers could "export" to more widely-used packaging schemes. guix can already export to .deb, but there's no reason it couldn't produce an AppImage or a flatpak.

simiones 5y ago

> On Windows and macOS, there are some SDKs the application can lean on. On Linux, the only stable "SDK" is the Linux kernel API. This is a very good observation I think. Basically, what Linux distributions have failed to do is to create a higher level SDK for their platforms - instead, they rely on a soup of libraries and on the package maintainer model to know which of these is which. Basically, they are failing to provide a true OS: there is only a Kernel + various user space apps, from systemd to KDevelop, with no explicit line in the sand between what is the system SDK and what is a simple app bundled with the OS.

WastingMyTime89 5y ago
> This is a very good observation I think. Basically, what Linux distributions have failed to do is to create a higher level SDK for their platforms - instead, they rely on a soup of libraries and on the package maintainer model to know which of these is which. Not really no. It's both simpler and more complicated. At the very low level, everyone expects to get a POSIX system. At the user level, most users use either Gnome or KDE (well mostly Gnome to be honest but let's pretend) which provides what could be called a highlevel SDK for applications. That leaves the layer inbetween which used to be somewhat distribution specific but is seeing more and more consolidation with the advent of systemd and flatpak.
- simiones 5y ago
  Even Gnome and KDE are not really "high level SDKs". They mostly have GUI support of various kinds, but not a full SDK for interacting with the OS (e.g. interacting with networking, with other apps, with system services etc.).
nonameiguess 5y ago
I don't think this is really true. There is a fairly stable core set of C libraries present on virtually any Linux distro with a desktop environment that does most of what any application needs in terms of system services. The basic problem is more what the parent was discussing: these are C libraries, and rather than using the FFI for their app language of choice and using these pre-installed libraries, developers would rather rewrite all functionality in the same language as their applications, and the Rust crate, Go mod, or Node package you find when Googling whether something exists in your language of choice is likely to be much less stable than something like libcurl. Mac and Windows solve this by creating their own languages for app development and then porting the system services to be provided in Swift and C# or whatever in addition to the original C implementation. There is no corporation behind most Linux distros willing to do that other than Canonical and Redhat, but they're perfectly happy to just stick with C, as is Gnome (KDE being perfectly happy to stick with C++). Most app developers are not, however. For what it's worth, Redhat did kind of solve this problem for the Linux container ecosystem with libcontainer, rewriting all of the core functionality in Go, in recognition of the fact that this is the direction the ecosystem was moving in thanks to Docker and Kubernetes using Go, and now app developers for the CNCF ecosystem can just use that single set of core dependencies that stays pretty stable.

jcelerier 5y ago

> On Windows and macOS, there are some SDKs the application can lean on. On Linux, the only stable "SDK" is the Linux kernel API. In practice I don't really see a difference, I link against Qt on all platforms, win, mac, linux, wasm, android... and that's it.

maccolgan 5y ago

Dynamic linking is a creation of the devil.

jiggawatts 5y ago

I watched a great rant by Linus T himself on how distributing code for Linux is a "pain in the arse", but is comparatively easier on Windows and MacOS: https://www.youtube.com/watch?v=Pzl1B7nB9Kc https://www.youtube.com/watch?v=Pzl1B7nB9Kc Something I took away from that is that it is Linus himself, personally, that is responsible (in a way) for Docker existing. Docker depends on the stable kernel ABI in order to allow container images to be portable and "just work" stably and reliably. This is a "guarantee" that Linus gets shouty about if the kernel devs break it. Docker fills a need on Linux because the user-mode libraries across all distributions are a total mess. Microsoft is trying to be the "hip kid" by copying Docker into Windows, where everything is backwards to the Linux situations: The NT kernel ABI is treated as unstable, and changes as fast as every few months. No one ever codes directly against the kernel on Windows (for some values of "no one" and "ever".) The user-mode libraries like Win32 and .NET are very stable, papering over the inconsistency of the kernel library. You can run applications compiled in the year 2000 today, unmodified, and more often than not they'll "just work". There just isn't a "burning need" for Docker on Windows. People that try to reproduce the cool new Linux workflow however are in for a world of hurt, because they'll rapidly discover that the images they built just weeks ago might not run any more because the latest Windows Update bumped the kernel version. I read all the way through this and wept: https://docs.microsoft.com/en-us/virtualization/windowscontainers/deploy-containers/version-compatibility https://docs.microsoft.com/en-us/virtualization/windowsconta...