4 ms·

> In practice this doesn't work because each application has to be tested against the new version of the library. In practice it works: see autopkgtest and htt

> In practice this doesn't work because each application has to be tested against the new version of the library. In practice it works: see autopkgtest and https://ci.debian.net https://ci.debian.net, which reruns the tests for each reverse dependency of a package, every time a library gets updated. I know for a fact that other commercial, corporate-backed distributions are far away from that sophistication, though.

Aissen 5y ago
If you can retest everything, you can rebuild everything. But that doesn't help the network bandwidth issue. Debian doesn't have incremental binary diff packages anyway (delta debs) in the default repos anyway, so there's room for improvement there.
- ex_amazon_sde 5y ago
  > If you can retest everything, you can rebuild everything No, rebuilding is way heavier. Even worse, some languages insist on building against fixed version of their dependencies. It forces distributions to keep multiple version of the same library and this leads to a combinatorics explosion of packages to fix, backport, compile and test. It's simply unsustainable and it's hurting distributions already.
  - Aissen 5y ago
    Distros have been known to patch packages that wanted a fixed version of a dependency. In fact, most have it has their default policy.
    - ex_amazon_sde 5y ago
      Yes, that's my point.
      - Aissen 5y ago
        To rephrase what I meant; those two aren't exclusive: - statically link a library - have all binaries in the system use the same version of the library, and updated when the library needs to be updated.
toast0 5y ago
So if the CVE says that you need to update library L, and program A uses the thing that's broken and it's ok to update, but program B doesn't use the broken thing, but the L upgrade breaks it, CI would let you know that you're screwed before you upgrade... but you're still screwed. It's worse if the program that needs the update also is broken by the update, of course.
- ajsnigrutin 5y ago
  So, what's the alternative? Now it's your choice... you either lose B but protect the rest of the infrastructure from hackers... or you think the CVE doesn't apply to your usecase (internal thing on a testing server), and don't upgrade L to keep B working. You can also install both versions of L. You can also patch the broken part of L out at the old version, if it's not mission critical. There's a lot of things you can do. Having one giant binary file with everything statically compiled in is worse in every way, except for distribution-as-a-single-file (but you can already do this now, by putting the binary and the libraries in a single zip, dump everytinh in /opt/foo, and let user find the vulnerable library manually... which again, sucks.
  - toast0 5y ago
    If it were static libraries, you'd upgrade the package for A (which would need to be recompiled with updated L) and leave B alone. As a low priority followup, fix either B or L so they can work together again (or wait for someone else to fix and release). Installing both versions of L is usually hard. It's one thing if it's OpenSSL[1] 1.1 vs 1.0, but if 1.0.0e is needed for security and 1.0.0d is needed for other applications, how do you make that work in Debian (or any other system that's at least somewhat mainstream)? [1] Not to pick on OpenSSL, but it's kind of the poster child for important to pick up updates that also break things; but at least they provide security updates across branches that are trying to stay compatible.
Conan_Kudo 5y ago
> I know for a fact that other commercial, corporate-backed distributions are far away from that sophistication, though. No, they're not. Both Fedora/RHEL and openSUSE/SLE do similar checks. Every update submission goes through these kinds of checks. Heck, the guy who created autopkgtest works at Red Hat and helped design the testing system used in Fedora and RHEL now. openSUSE has been doing dependency checks and tests for almost a decade now, with the combination of the openSUSE Build Service and OpenQA.
- throwawaySUSE 5y ago
  Ex SUSE employee here. OBS and OpenQA are nowhere close to Debian CI.
  - Conan_Kudo 5y ago
    Having worked with both systems, Debian CI does not focus on system integration. Fedora CI and Debian CI are more similar than different, but Fedora also has an OpenQA instance for doing the system integration testing as openSUSE does. openSUSE's main weakness is that they don't do deeper inspections of RPM artifacts, the dependency graph, etc. They don't feel they need it because OBS auto-rebuilds everything on every submission anyway. The Fedora CI tooling absolutely does this since auto-rebuilds on build aren't a thing there, and it's done on PRs as well as update submissions.

Aissen 5y ago

If you can retest everything, you can rebuild everything. But that doesn't help the network bandwidth issue. Debian doesn't have incremental binary diff packages anyway (delta debs) in the default repos anyway, so there's room for improvement there.

ex_amazon_sde 5y ago
> If you can retest everything, you can rebuild everything No, rebuilding is way heavier. Even worse, some languages insist on building against fixed version of their dependencies. It forces distributions to keep multiple version of the same library and this leads to a combinatorics explosion of packages to fix, backport, compile and test. It's simply unsustainable and it's hurting distributions already.
- Aissen 5y ago
  Distros have been known to patch packages that wanted a fixed version of a dependency. In fact, most have it has their default policy.
  - ex_amazon_sde 5y ago
    Yes, that's my point.
    - Aissen 5y ago
      To rephrase what I meant; those two aren't exclusive: - statically link a library - have all binaries in the system use the same version of the library, and updated when the library needs to be updated.

toast0 5y ago

So if the CVE says that you need to update library L, and program A uses the thing that's broken and it's ok to update, but program B doesn't use the broken thing, but the L upgrade breaks it, CI would let you know that you're screwed before you upgrade... but you're still screwed. It's worse if the program that needs the update also is broken by the update, of course.

ajsnigrutin 5y ago
So, what's the alternative? Now it's your choice... you either lose B but protect the rest of the infrastructure from hackers... or you think the CVE doesn't apply to your usecase (internal thing on a testing server), and don't upgrade L to keep B working. You can also install both versions of L. You can also patch the broken part of L out at the old version, if it's not mission critical. There's a lot of things you can do. Having one giant binary file with everything statically compiled in is worse in every way, except for distribution-as-a-single-file (but you can already do this now, by putting the binary and the libraries in a single zip, dump everytinh in /opt/foo, and let user find the vulnerable library manually... which again, sucks.
- toast0 5y ago
  If it were static libraries, you'd upgrade the package for A (which would need to be recompiled with updated L) and leave B alone. As a low priority followup, fix either B or L so they can work together again (or wait for someone else to fix and release). Installing both versions of L is usually hard. It's one thing if it's OpenSSL[1] 1.1 vs 1.0, but if 1.0.0e is needed for security and 1.0.0d is needed for other applications, how do you make that work in Debian (or any other system that's at least somewhat mainstream)? [1] Not to pick on OpenSSL, but it's kind of the poster child for important to pick up updates that also break things; but at least they provide security updates across branches that are trying to stay compatible.

Conan_Kudo 5y ago

> I know for a fact that other commercial, corporate-backed distributions are far away from that sophistication, though. No, they're not. Both Fedora/RHEL and openSUSE/SLE do similar checks. Every update submission goes through these kinds of checks. Heck, the guy who created autopkgtest works at Red Hat and helped design the testing system used in Fedora and RHEL now. openSUSE has been doing dependency checks and tests for almost a decade now, with the combination of the openSUSE Build Service and OpenQA.

throwawaySUSE 5y ago
Ex SUSE employee here. OBS and OpenQA are nowhere close to Debian CI.
- Conan_Kudo 5y ago
  Having worked with both systems, Debian CI does not focus on system integration. Fedora CI and Debian CI are more similar than different, but Fedora also has an OpenQA instance for doing the system integration testing as openSUSE does. openSUSE's main weakness is that they don't do deeper inspections of RPM artifacts, the dependency graph, etc. They don't feel they need it because OBS auto-rebuilds everything on every submission anyway. The Fedora CI tooling absolutely does this since auto-rebuilds on build aren't a thing there, and it's done on PRs as well as update submissions.