3 ms·

In fact, I am tired of the same issue e.g. when evaluating machine learning tools with Python. It's as if everyone just says "dependency = ${exact_version_I_hav

In fact, I am tired of the same issue e.g. when evaluating machine learning tools with Python. It's as if everyone just says "dependency = ${exact_version_I_have}" and never considers anything else. Not helped by the fact that most of these projects break ABI every other day which I still think it is just plain evil. Please don't say "just use containers/virtualenvs/vendoring/whatever" because at some point you obviously want these "containers" to use your real hardware. And your GPU driver is only going to be at one version, no matter how many statically linked executables you have.

aragilar 2y ago
It's 100% culture: this happens across all languages.
rfoo 2y ago
> And your GPU driver is only going to be at one version, no matter how many statically linked executables you have. This is a non-issue. libcuda.so (which is part of the NVIDIA driver, their userland part) can not be statically linked into anything, since it's closed-source and no .a shipped at all, so no possibility of misconfigure it to be statically linked. And NVIDIA drivers, as a whole (including a pair of libcuda.so and corresponding .ko), is infinitely backward-compatible, you can always use new driver with old libraries. So the problem you mentioned does not exist at all. Statically linking to cuBLAS / cuDNN etc never add more pain and can only reduce pain. Unless you hold it wrong.
- AshamedCaptain 2y ago
  That is hardly true for NVIDIA over the years, as I have experienced personally, and it's definitely not true for about every other GPU driver in existence -- particularly AMD's RoCM, which even recommends specific Python versions to go with each ROCM version. In addition, replace "GPU" with any other piece of hardware, such as NPU. Or even software, such as your desktop environment. Do you think a statically linked binary will run with Wayland+1 ? It's already a chore to run games that link with older SDL versions, and the only reason you can at all is that SDL was dynamically linked... Even the kernel itself does a pretty poor job at maintaining backwards compatibility (e.g. contents of /sys changing every other year). Statically linking just hardcodes one of these things, and then forgets about the rest, such as IPC APIs, desktop APIs, filesystem structure, etc. which due to the poor culture out there are moving as much as the libraries themselves.
  - mananaysiempre 2y ago
    > Do you think a statically linked binary will run with Wayland+1 ? Yes, yes it will. The baseline stuff (the base protocol and xdg_shell) has been stable for years. Of course, one might argue that the caveman byte poking provided there turns a modern machine into a slightly faster 15-year-old one, and I agree it does. (It’s still fun to do as an experiment.) But the moment you try to actually leverage the GPU the modern Linux desktop forces you to abandon static linking in order to load pieces of the graphics driver into your address space, and unfortunately those pieces will in turn force you to dynamically link the distro’s preferred libc (likely Glibc) and libwayland (as opposed to any other protocol implementation). So either you poke pixels into a byte buffer by hand, and your question gets an answer in the affirmative simply because that foundational layer is fairly small, or you take advantage of the hardware, and your question becomes moot because you can no longer statically link.
epage 2y ago
In Rust, it is frowned upon to pin to an exact version. We do encourage people to specify what semver version range will work. We don't have great support for depending on and verifying multiple-major version ranges when a library broke compatibility but not in a way that affects you.
- aragilar 2y ago
  Is this being tracked somewhere?
ris 2y ago
> It's as if everyone just says "dependency = ${exact_version_I_have}" and never considers anything else. This is what I call "version soup". The idea that every project can choose an near-arbitrary set of very exact version numbers of dependencies and expect it to work. And that every project on earth has the burden of continually stirring their soup, often through a tool like dependabot, hoping (cross fingers) no compatibility problems come about. Of course, there's no guarantee that those versions will work together. The extremely limited abilities packaging tools have for expressing dependency restrictions (>= this or that version) generally lack the ability to even handle the concept of stable branches with backports. i.e. "No, it must be >= 1.2.3.. what do you mean they backported the relevant fix to 1.1.4? 1.1.4 < 1.2.3 so it's unacceptable". The way these specifications are then used by authors brings an extra layer of noise to the compatibility situation - very few (understandably) will actually go and check the limits of version compatibility. In nixpkgs we attempt to address this situation for the python ecosystem by providing one version of each package (with few exceptions) per release. But in return, we put in work to make sure those versions actually work with each other - generally by getting the projects' test suites integrated into the build system. The idea is that an app built to depend on nixpkgs packages should be able to expect to do dependency upgrades as a "jump" every 6 months when there's a new nixpkgs release, but otherwise be able to depend on a stable suite of packages that still receives security updates & backports.
- AshamedCaptain 2y ago
  > In nixpkgs we attempt to address this situation for the python ecosystem by providing one version of each package (with few exceptions) per release. But in return, we put in work to make sure those versions actually work with each other - generally by getting the projects' test suites integrated into the build system. The idea is that an app built to depend on nixpkgs packages should be able to expect to do dependency upgrades as a "jump" every 6 months when there's a new nixpkgs release, but otherwise be able to depend on a stable suite of packages that still receives security updates & backports. Is this different from any other Linux distribution?
  - ris 2y ago

aragilar 2y ago

It's 100% culture: this happens across all languages.

rfoo 2y ago

> And your GPU driver is only going to be at one version, no matter how many statically linked executables you have. This is a non-issue. libcuda.so (which is part of the NVIDIA driver, their userland part) can not be statically linked into anything, since it's closed-source and no .a shipped at all, so no possibility of misconfigure it to be statically linked. And NVIDIA drivers, as a whole (including a pair of libcuda.so and corresponding .ko), is infinitely backward-compatible, you can always use new driver with old libraries. So the problem you mentioned does not exist at all. Statically linking to cuBLAS / cuDNN etc never add more pain and can only reduce pain. Unless you hold it wrong.

AshamedCaptain 2y ago
That is hardly true for NVIDIA over the years, as I have experienced personally, and it's definitely not true for about every other GPU driver in existence -- particularly AMD's RoCM, which even recommends specific Python versions to go with each ROCM version. In addition, replace "GPU" with any other piece of hardware, such as NPU. Or even software, such as your desktop environment. Do you think a statically linked binary will run with Wayland+1 ? It's already a chore to run games that link with older SDL versions, and the only reason you can at all is that SDL was dynamically linked... Even the kernel itself does a pretty poor job at maintaining backwards compatibility (e.g. contents of /sys changing every other year). Statically linking just hardcodes one of these things, and then forgets about the rest, such as IPC APIs, desktop APIs, filesystem structure, etc. which due to the poor culture out there are moving as much as the libraries themselves.
- mananaysiempre 2y ago
  > Do you think a statically linked binary will run with Wayland+1 ? Yes, yes it will. The baseline stuff (the base protocol and xdg_shell) has been stable for years. Of course, one might argue that the caveman byte poking provided there turns a modern machine into a slightly faster 15-year-old one, and I agree it does. (It’s still fun to do as an experiment.) But the moment you try to actually leverage the GPU the modern Linux desktop forces you to abandon static linking in order to load pieces of the graphics driver into your address space, and unfortunately those pieces will in turn force you to dynamically link the distro’s preferred libc (likely Glibc) and libwayland (as opposed to any other protocol implementation). So either you poke pixels into a byte buffer by hand, and your question gets an answer in the affirmative simply because that foundational layer is fairly small, or you take advantage of the hardware, and your question becomes moot because you can no longer statically link.

epage 2y ago

In Rust, it is frowned upon to pin to an exact version. We do encourage people to specify what semver version range will work. We don't have great support for depending on and verifying multiple-major version ranges when a library broke compatibility but not in a way that affects you.

aragilar 2y ago
Is this being tracked somewhere?

ris 2y ago

> It's as if everyone just says "dependency = ${exact_version_I_have}" and never considers anything else. This is what I call "version soup". The idea that every project can choose an near-arbitrary set of very exact version numbers of dependencies and expect it to work. And that every project on earth has the burden of continually stirring their soup, often through a tool like dependabot, hoping (cross fingers) no compatibility problems come about. Of course, there's no guarantee that those versions will work together. The extremely limited abilities packaging tools have for expressing dependency restrictions (>= this or that version) generally lack the ability to even handle the concept of stable branches with backports. i.e. "No, it must be >= 1.2.3.. what do you mean they backported the relevant fix to 1.1.4? 1.1.4 < 1.2.3 so it's unacceptable". The way these specifications are then used by authors brings an extra layer of noise to the compatibility situation - very few (understandably) will actually go and check the limits of version compatibility. In nixpkgs we attempt to address this situation for the python ecosystem by providing one version of each package (with few exceptions) per release. But in return, we put in work to make sure those versions actually work with each other - generally by getting the projects' test suites integrated into the build system. The idea is that an app built to depend on nixpkgs packages should be able to expect to do dependency upgrades as a "jump" every 6 months when there's a new nixpkgs release, but otherwise be able to depend on a stable suite of packages that still receives security updates & backports.

AshamedCaptain 2y ago
> In nixpkgs we attempt to address this situation for the python ecosystem by providing one version of each package (with few exceptions) per release. But in return, we put in work to make sure those versions actually work with each other - generally by getting the projects' test suites integrated into the build system. The idea is that an app built to depend on nixpkgs packages should be able to expect to do dependency upgrades as a "jump" every 6 months when there's a new nixpkgs release, but otherwise be able to depend on a stable suite of packages that still receives security updates & backports. Is this different from any other Linux distribution?
- ris 2y ago