4 ms·

This isn't Rust specific. The same issue exists in all languages where the versions can be restricted at project level. It's an issue in Debian because they can

This isn't Rust specific. The same issue exists in all languages where the versions can be restricted at project level. It's an issue in Debian because they can't handle multiple concurrent versions (beyond renaming the package) and want every package built without internet access. But that same issue affects Ruby, Python, etc. if the project specifies strict versions. And if you deal with filesystems, you really want strict versions - I understand why the author wants to keep it that way. It's more of a self-inflicted Debian policy pain rather than anything to do with Rust. The author could be nice and vendor, but also he's completely right to refuse. The request is basically "we put handcuffs on ourselves, so please you do the work instead".

agwa 2y ago
The author would be fine with vendoring. The problem is that Debian doesn't allow it.
- AshamedCaptain 2y ago
  And that's because the users of Debian are not fine with vendoring.
- cesarb 2y ago
  > The problem is that Debian doesn't allow it. Vendoring used to be more common, even in Debian. But then there was an important zlib vulnerability (which IIRC could be exploited through manipulated compressed data), and they had to chase and fix all the many copies of zlib embedded all over the place. To make things worse, some of them were not only old versions of zlib, but also had modified that zlib code, so each had to be reviewed before applying the fix. Debian (and other traditional distributions) learned from that incident, and started a strong push to not only remove vendored copies of libraries, but also when possible use a single version of these libraries. Which means that, whenever an important security issue is once again found in a widely used library, only a single copy has to be fixed and updated.
  - chandlerswift 2y ago
    I'm interested in hearing more about the history behind Debian and zlib! I did some searching and the closest thing I could find was a nod to the same incident in the Upstream Guide[0]. Do you know of a place where I could read more about it? [0]: https://wiki.debian.org/UpstreamGuide#No_inclusion_of_third_party_code https://wiki.debian.org/UpstreamGuide#No_inclusion_of_third_...
    - cesarb 2y ago
      It's been so long ago, that it's hard to find all the discussions I had seen back then. I recall that it was after a long time without any zlib release, so looking at the zlib history, I think it was this one fixed in zlib 1.1.4 from 11 March 2002: http://www.zlib.org/advisory-2002-03-11.txt http://www.zlib.org/advisory-2002-03-11.txt Looking at the debian-devel archives around that date, I found a Debian developer complaining about the vendored zlib copies (https://lists.debian.org/debian-devel/2002/03/msg00716.html https://lists.debian.org/debian-devel/2002/03/msg00716.html), but not the full discussion about getting rid of vendored libraries, so it must have happened elsewhere.
      - chandlerswift 2y ago
        Thanks for taking the time these references up for me!
JackSlateur 2y ago
if you deal with filesystems, you really want strict versions How are doing everybody else ? What is specific with filesystems, here ? This is a bad dependency management, nothing more. Vendoring is a PITA.
- viraptor 2y ago
  Level of potentially messing up. If you have a chat app, relax the deps and it crashes - oh well. But if you have a tool which can cause data loss, you really want to be sure everyone's running what you tested. It's similar to why restic (backup software) has official, static, reproducible binaries in their releases.
  - JackSlateur 2y ago
    And if my kernel crashes ? And if whatever library bugged, oh well, random people can access my systems ? And if my bgpd bugged, oh well, we just inject random routes et forward data randomly ? Makes no sense.
    - viraptor 2y ago
      Your kernel has no external library dependencies. Relaxing library requirements can introduce security issues when the dependency behaviour changes, so I'm not sure what the argument is. If your bgp is bugged, yeah it sucks, but it's recoverable. We're using encrypted traffic these days to ensure that even captured traffic is safe. The difference here is that data corruption is not recoverable.
      - deleted 2y ago
        [deleted]
      - JackSlateur 2y ago
        Linux uses at least libc
        2y ago
1oooqooq 2y ago
it is rust specific because rust is the first attemp to replace proper system engineering languages with one that while nicer on memory management and overall ergonomics, is worse in tooking. cargo brings many malpractices from java/JavaScript (maven, npm, etc) that were always shunned in systems engineering and, mark my words, will be a security nightmare for linux in the near future.
- viraptor 2y ago
  Go, D, Ocaml, Nim have the same issue, most are older than Rust and are arguably systems languages. C/C++ built with Bazel or Conan will have the same issue too.
  - 1oooqooq 2y ago
    those are userland system applications at most... which i grant is also what the original article talked about, but still.
  - rc00 2y ago
    Go would only be comparable if you had hundreds of vendored dependencies but Go's community and culture have encouraged use of the standard library over external dependencies where possible. A core problem with Rust is the lack of an adequate standard library. The problem with Cargo is when you have an application with hundreds of dependencies. That does not exist in any mainstream Go application. 1. Why Rust Libraries May Never Exist - https://www.youtube.com/watch?v=769VqNup21Q https://www.youtube.com/watch?v=769VqNup21Q 2. The Rust Cargo ecosystem is broken - https://www.youtube.com/watch?v=OdMQwDtumJ8 https://www.youtube.com/watch?v=OdMQwDtumJ8
    - viraptor 2y ago
      Having a big standard library doesn't help that much since it needs separate updates. Every other golang app I use requires updated crypto, net, or sync. In theory they come in the standard library, but the moment you need updates, they're exactly the same as other dependencies for packaging purposes. (it's very common, yes I do packaging)
AshamedCaptain 2y ago
In fact, I am tired of the same issue e.g. when evaluating machine learning tools with Python. It's as if everyone just says "dependency = ${exact_version_I_have}" and never considers anything else. Not helped by the fact that most of these projects break ABI every other day which I still think it is just plain evil. Please don't say "just use containers/virtualenvs/vendoring/whatever" because at some point you obviously want these "containers" to use your real hardware. And your GPU driver is only going to be at one version, no matter how many statically linked executables you have.
- aragilar 2y ago
  It's 100% culture: this happens across all languages.
- rfoo 2y ago
  > And your GPU driver is only going to be at one version, no matter how many statically linked executables you have. This is a non-issue. libcuda.so (which is part of the NVIDIA driver, their userland part) can not be statically linked into anything, since it's closed-source and no .a shipped at all, so no possibility of misconfigure it to be statically linked. And NVIDIA drivers, as a whole (including a pair of libcuda.so and corresponding .ko), is infinitely backward-compatible, you can always use new driver with old libraries. So the problem you mentioned does not exist at all. Statically linking to cuBLAS / cuDNN etc never add more pain and can only reduce pain. Unless you hold it wrong.
  - AshamedCaptain 2y ago
    That is hardly true for NVIDIA over the years, as I have experienced personally, and it's definitely not true for about every other GPU driver in existence -- particularly AMD's RoCM, which even recommends specific Python versions to go with each ROCM version. In addition, replace "GPU" with any other piece of hardware, such as NPU. Or even software, such as your desktop environment. Do you think a statically linked binary will run with Wayland+1 ? It's already a chore to run games that link with older SDL versions, and the only reason you can at all is that SDL was dynamically linked... Even the kernel itself does a pretty poor job at maintaining backwards compatibility (e.g. contents of /sys changing every other year). Statically linking just hardcodes one of these things, and then forgets about the rest, such as IPC APIs, desktop APIs, filesystem structure, etc. which due to the poor culture out there are moving as much as the libraries themselves.
cogman10 2y ago
It should be noted that, particularly for rust, it's a silly policy as rust does not support dynamic linking of rust libraries. All rust applications are statically linked. So trying to force all projects using foo onto the same version of foo is just a huge headache with no real benefit to anything.
- haileys 2y ago
  It remains a problem for source packages, which are an important part of Debian necessary for offline and reproducible builds.
- bonzini 2y ago
  Almost—even if you need to rebuild dependencies, you can still patch security issues just once instead of doing so in every vendored copy. That's the first point in the parent comment. The same applies to Go.
- rlpb 2y ago
  > So trying to force all projects using foo onto the same version of foo is just a huge headache with no real benefit to anything. It creates a security nightmare. Distributions expect to be able to security patch without waiting for laggard upstreams to bump their pinned dependency versions. If the ecosystem insists that only upstream-sanctioned versions are acceptable, then that runs contrary to this expectation, because it blocks distributions from patching or bumping dependency versions. If this case should be an exception, then you end up with a security team having to patch myriad different versions. That's the security nightmare.
  - deleted 2y ago
    [deleted]
  - cogman10 2y ago
    Fair point. I do think we need to reevaluate how security is managed in terms of software. It's often the case that we say "this library version has a CVE, we need to update it!" However, it's not often the case that the CVE in the library is even exploitable from the parent application. I'm thinking of a recent 9 CVE against zlib... But actually it was a rarely used extension API for working with compressed files. Something like path tracing to see if a vulnerable API is even accessible from the parent application/library before instructing the universe to adopt a new version.
nubinetwork 2y ago
> This isn't Rust specific. The same issue exists in all languages where the versions can be restricted at project level Can confirm, I occasionally use an ai program that has to hard code all of their python dependencies, because it's the only way to get it to compile, let alone run properly... and then they go and change the underlying package manager, and figure out they have to hard code even more... it's a bloody mess, but thankfully you can just run it as a docker container once it's all working...
- ptero 2y ago
  Not disagreeing, but IME much more often I have to run something that locks specific versions of their python dependencies because ... well ... why not -- kool kids run everything in a virtual environment anyway. Version locking is a powerful tool that should be used very sparingly. My 2c.
- greatgib 2y ago
  I think that 2 different things are mixed up. For Rust it is language specific, because the thing being statically built, doing differently is very hard. For other languages like Python, there is not a need and you should not do that. The dynamic part of it make it so that developer should be resilient to different versions. And python also provide a lot of conveniences for that (just think about six). But in your case, you faced the current situation of lot of "data scientist" jumping the band wagon of AI are creating a lot of things like apprentice sorcerers without a real experience of software engineering.
  - nubinetwork 2y ago
    > For other languages like Python, there is not a need and you should not do that. Normally I would agree, but when it comes to cuda and rocm, and the one that intel uses... you find that you have to pin torch/audio/video, and then you have to start pinning its dependencies (and their dependencies etc etc) or else you start getting random build or runtime failures...
  - rcxdude 2y ago
    No, in both cases the issue is the same: whether the actual interface of the library is staying the same or not, and over what time period. Python packages have plenty of the same kind of breakages. usually it's unintentional, a bug or similar, sometimes it's a deliberate breaking change in a point release. Either way the effect is the same. Version X works in project A but newer version Y doesn't, and the reverse can be true for project B. It's just that in python you're most likely to get the problem when you run the code, wheras compiled languages can also fail at the build step as well as at runtime. It can be pretty bad with the bigger ML/scientific computing packages, hence all the version pinning that happens, though I will concur that generally speaking AI researchers don't understand package management in the slightest and you get all kinds of horrors as a result (e.g. mixing anaconda and pip in the same project, bonus points if it's blasted over the distro's python packaging for even more chaos. works on one very broken machine if you are lucky and don't touch anything). Debian's approach is similarly frustrating: they will try to make your code work with an older version of a library than the one you developed with, which doesn't even work within semver (semver only defines backwards compatibility, not forwards). That's what caused the problem in this case: rolling forward dependencies was not recommended but didn't break anything, rolling a dependency backwards meant it stopped working, which is something absolutely no-one should be surprised or upset by (at least it meant the build broke as opposed to debian shipping a broken package like would have likely happened with python. It just meant that debian was shipping a version with bugs fixed in upstream). If you're going to do that you've got to be prepared to take responsibility for fixing the problem that you've introduced.

agwa 2y ago

The author would be fine with vendoring. The problem is that Debian doesn't allow it.

AshamedCaptain 2y ago
And that's because the users of Debian are not fine with vendoring.
cesarb 2y ago
> The problem is that Debian doesn't allow it. Vendoring used to be more common, even in Debian. But then there was an important zlib vulnerability (which IIRC could be exploited through manipulated compressed data), and they had to chase and fix all the many copies of zlib embedded all over the place. To make things worse, some of them were not only old versions of zlib, but also had modified that zlib code, so each had to be reviewed before applying the fix. Debian (and other traditional distributions) learned from that incident, and started a strong push to not only remove vendored copies of libraries, but also when possible use a single version of these libraries. Which means that, whenever an important security issue is once again found in a widely used library, only a single copy has to be fixed and updated.
- chandlerswift 2y ago
  I'm interested in hearing more about the history behind Debian and zlib! I did some searching and the closest thing I could find was a nod to the same incident in the Upstream Guide[0]. Do you know of a place where I could read more about it? [0]: https://wiki.debian.org/UpstreamGuide#No_inclusion_of_third_party_code https://wiki.debian.org/UpstreamGuide#No_inclusion_of_third_...
  - cesarb 2y ago
    It's been so long ago, that it's hard to find all the discussions I had seen back then. I recall that it was after a long time without any zlib release, so looking at the zlib history, I think it was this one fixed in zlib 1.1.4 from 11 March 2002: http://www.zlib.org/advisory-2002-03-11.txt http://www.zlib.org/advisory-2002-03-11.txt Looking at the debian-devel archives around that date, I found a Debian developer complaining about the vendored zlib copies (https://lists.debian.org/debian-devel/2002/03/msg00716.html https://lists.debian.org/debian-devel/2002/03/msg00716.html), but not the full discussion about getting rid of vendored libraries, so it must have happened elsewhere.
    - chandlerswift 2y ago
      Thanks for taking the time these references up for me!

JackSlateur 2y ago

if you deal with filesystems, you really want strict versions How are doing everybody else ? What is specific with filesystems, here ? This is a bad dependency management, nothing more. Vendoring is a PITA.

viraptor 2y ago
Level of potentially messing up. If you have a chat app, relax the deps and it crashes - oh well. But if you have a tool which can cause data loss, you really want to be sure everyone's running what you tested. It's similar to why restic (backup software) has official, static, reproducible binaries in their releases.
- JackSlateur 2y ago
  And if my kernel crashes ? And if whatever library bugged, oh well, random people can access my systems ? And if my bgpd bugged, oh well, we just inject random routes et forward data randomly ? Makes no sense.
  - viraptor 2y ago
    Your kernel has no external library dependencies. Relaxing library requirements can introduce security issues when the dependency behaviour changes, so I'm not sure what the argument is. If your bgp is bugged, yeah it sucks, but it's recoverable. We're using encrypted traffic these days to ensure that even captured traffic is safe. The difference here is that data corruption is not recoverable.
    - deleted 2y ago
      [deleted]
    - JackSlateur 2y ago
      Linux uses at least libc
      - 2y ago

1oooqooq 2y ago

it is rust specific because rust is the first attemp to replace proper system engineering languages with one that while nicer on memory management and overall ergonomics, is worse in tooking. cargo brings many malpractices from java/JavaScript (maven, npm, etc) that were always shunned in systems engineering and, mark my words, will be a security nightmare for linux in the near future.

viraptor 2y ago
Go, D, Ocaml, Nim have the same issue, most are older than Rust and are arguably systems languages. C/C++ built with Bazel or Conan will have the same issue too.
- 1oooqooq 2y ago
  those are userland system applications at most... which i grant is also what the original article talked about, but still.
- rc00 2y ago
  Go would only be comparable if you had hundreds of vendored dependencies but Go's community and culture have encouraged use of the standard library over external dependencies where possible. A core problem with Rust is the lack of an adequate standard library. The problem with Cargo is when you have an application with hundreds of dependencies. That does not exist in any mainstream Go application. 1. Why Rust Libraries May Never Exist - https://www.youtube.com/watch?v=769VqNup21Q https://www.youtube.com/watch?v=769VqNup21Q 2. The Rust Cargo ecosystem is broken - https://www.youtube.com/watch?v=OdMQwDtumJ8 https://www.youtube.com/watch?v=OdMQwDtumJ8
  - viraptor 2y ago
    Having a big standard library doesn't help that much since it needs separate updates. Every other golang app I use requires updated crypto, net, or sync. In theory they come in the standard library, but the moment you need updates, they're exactly the same as other dependencies for packaging purposes. (it's very common, yes I do packaging)

AshamedCaptain 2y ago

In fact, I am tired of the same issue e.g. when evaluating machine learning tools with Python. It's as if everyone just says "dependency = ${exact_version_I_have}" and never considers anything else. Not helped by the fact that most of these projects break ABI every other day which I still think it is just plain evil. Please don't say "just use containers/virtualenvs/vendoring/whatever" because at some point you obviously want these "containers" to use your real hardware. And your GPU driver is only going to be at one version, no matter how many statically linked executables you have.

aragilar 2y ago
It's 100% culture: this happens across all languages.
rfoo 2y ago
> And your GPU driver is only going to be at one version, no matter how many statically linked executables you have. This is a non-issue. libcuda.so (which is part of the NVIDIA driver, their userland part) can not be statically linked into anything, since it's closed-source and no .a shipped at all, so no possibility of misconfigure it to be statically linked. And NVIDIA drivers, as a whole (including a pair of libcuda.so and corresponding .ko), is infinitely backward-compatible, you can always use new driver with old libraries. So the problem you mentioned does not exist at all. Statically linking to cuBLAS / cuDNN etc never add more pain and can only reduce pain. Unless you hold it wrong.
- AshamedCaptain 2y ago
  That is hardly true for NVIDIA over the years, as I have experienced personally, and it's definitely not true for about every other GPU driver in existence -- particularly AMD's RoCM, which even recommends specific Python versions to go with each ROCM version. In addition, replace "GPU" with any other piece of hardware, such as NPU. Or even software, such as your desktop environment. Do you think a statically linked binary will run with Wayland+1 ? It's already a chore to run games that link with older SDL versions, and the only reason you can at all is that SDL was dynamically linked... Even the kernel itself does a pretty poor job at maintaining backwards compatibility (e.g. contents of /sys changing every other year). Statically linking just hardcodes one of these things, and then forgets about the rest, such as IPC APIs, desktop APIs, filesystem structure, etc. which due to the poor culture out there are moving as much as the libraries themselves.

cogman10 2y ago

It should be noted that, particularly for rust, it's a silly policy as rust does not support dynamic linking of rust libraries. All rust applications are statically linked. So trying to force all projects using foo onto the same version of foo is just a huge headache with no real benefit to anything.

haileys 2y ago
It remains a problem for source packages, which are an important part of Debian necessary for offline and reproducible builds.
bonzini 2y ago
Almost—even if you need to rebuild dependencies, you can still patch security issues just once instead of doing so in every vendored copy. That's the first point in the parent comment. The same applies to Go.
rlpb 2y ago
> So trying to force all projects using foo onto the same version of foo is just a huge headache with no real benefit to anything. It creates a security nightmare. Distributions expect to be able to security patch without waiting for laggard upstreams to bump their pinned dependency versions. If the ecosystem insists that only upstream-sanctioned versions are acceptable, then that runs contrary to this expectation, because it blocks distributions from patching or bumping dependency versions. If this case should be an exception, then you end up with a security team having to patch myriad different versions. That's the security nightmare.
- deleted 2y ago
  [deleted]
- cogman10 2y ago
  Fair point. I do think we need to reevaluate how security is managed in terms of software. It's often the case that we say "this library version has a CVE, we need to update it!" However, it's not often the case that the CVE in the library is even exploitable from the parent application. I'm thinking of a recent 9 CVE against zlib... But actually it was a rarely used extension API for working with compressed files. Something like path tracing to see if a vulnerable API is even accessible from the parent application/library before instructing the universe to adopt a new version.

nubinetwork 2y ago

> This isn't Rust specific. The same issue exists in all languages where the versions can be restricted at project level Can confirm, I occasionally use an ai program that has to hard code all of their python dependencies, because it's the only way to get it to compile, let alone run properly... and then they go and change the underlying package manager, and figure out they have to hard code even more... it's a bloody mess, but thankfully you can just run it as a docker container once it's all working...

ptero 2y ago
Not disagreeing, but IME much more often I have to run something that locks specific versions of their python dependencies because ... well ... why not -- kool kids run everything in a virtual environment anyway. Version locking is a powerful tool that should be used very sparingly. My 2c.
greatgib 2y ago
I think that 2 different things are mixed up. For Rust it is language specific, because the thing being statically built, doing differently is very hard. For other languages like Python, there is not a need and you should not do that. The dynamic part of it make it so that developer should be resilient to different versions. And python also provide a lot of conveniences for that (just think about six). But in your case, you faced the current situation of lot of "data scientist" jumping the band wagon of AI are creating a lot of things like apprentice sorcerers without a real experience of software engineering.
- nubinetwork 2y ago
  > For other languages like Python, there is not a need and you should not do that. Normally I would agree, but when it comes to cuda and rocm, and the one that intel uses... you find that you have to pin torch/audio/video, and then you have to start pinning its dependencies (and their dependencies etc etc) or else you start getting random build or runtime failures...
- rcxdude 2y ago
  No, in both cases the issue is the same: whether the actual interface of the library is staying the same or not, and over what time period. Python packages have plenty of the same kind of breakages. usually it's unintentional, a bug or similar, sometimes it's a deliberate breaking change in a point release. Either way the effect is the same. Version X works in project A but newer version Y doesn't, and the reverse can be true for project B. It's just that in python you're most likely to get the problem when you run the code, wheras compiled languages can also fail at the build step as well as at runtime. It can be pretty bad with the bigger ML/scientific computing packages, hence all the version pinning that happens, though I will concur that generally speaking AI researchers don't understand package management in the slightest and you get all kinds of horrors as a result (e.g. mixing anaconda and pip in the same project, bonus points if it's blasted over the distro's python packaging for even more chaos. works on one very broken machine if you are lucky and don't touch anything). Debian's approach is similarly frustrating: they will try to make your code work with an older version of a library than the one you developed with, which doesn't even work within semver (semver only defines backwards compatibility, not forwards). That's what caused the problem in this case: rolling forward dependencies was not recommended but didn't break anything, rolling a dependency backwards meant it stopped working, which is something absolutely no-one should be surprised or upset by (at least it meant the build broke as opposed to debian shipping a broken package like would have likely happened with python. It just meant that debian was shipping a version with bugs fixed in upstream). If you're going to do that you've got to be prepared to take responsibility for fixing the problem that you've introduced.