3 ms·

I used to think package managers and dependency resolvers were inseperable, I even did a little bit of work on the APT resolver. Since using GNU Guix though, I

I used to think package managers and dependency resolvers were inseperable, I even did a little bit of work on the APT resolver. Since using GNU Guix though, I'm so glad it doesn't have a dependency resolver as part of building or installing packages! It's so much better for it, no slow or unpredictable resolving, you know what it's going to do. I think this is one reason why I've never used pip for managing Python software, I've only ever used Debian, and then Guix.

yjftsjthsd-h 6y ago
Wait, how does guix avoid a dependency resolver? It still has dependencies for packages... are they just hardcoding exact dependencies (including versions) and relying on the "can install multiple versions of a package" property to make that work? That seems inefficient, though I guess maybe they need that for the "this hash means this exact binary" outcome?
- cbaines 6y ago
  Guix has packages, and packages have inputs (like dependencies), and you're right in that normally package definitions specify the exact dependencies it the code (they're hardcoded). Guix package definitions are truely code though, so if you want to generate packages on the fly by using a dependency resolver, you can totally write some code to make that happen. With respect to inefficiency, what do you mean? It's quite time efficient when building and installing to not have to attempt to resolve dependencies.
  - jrochkind1 6y ago
    If they are specifying exact hard-coded dependency versions... if a X -> Y -> A, and B -> A, and M -> N -> A... and a security patch is released for A, then X, Y, B, M and N all need new releases to specify new exact dependencies, in order to use the new A'? It's disastrous for security patches, only highly inconvenient for things like performance improvement releases. But this is why we have dependency resolving, right? What am I missing?
    - cbaines 6y ago
      No, you just update the package definition for A and release the updated package definitions, people will then be using the updated A, whether directly or through other packages (X, Y, B, M, N, ...). There is an issue here of rebuilding all those dependent packages with the updated A, especially if it's something like glibc. Guix includes a mechanism called grafts that allows for package replacements, which allows avoiding this, and this is often used for releasing security fixes.
      - jrochkind1 6y ago
        I think I must be misunderstanding. When you said: > you're right in that normally package definitions specify the exact dependencies it the code (they're hardcoded). I thought that meant that package X might specify a dependency on A version 2.4.2 exactly. So if you want it to use A 2.4.3 instead, a new release of package X would have to be created, specifying 2.4.3. But I think this is not in fact what you mean? In which case I don't yet understand the system you are describing and what you mean by not doing dependency resolution.
        danieldk 6y ago
        I thought that meant that package X might specify a dependency on A In Nix and Guix, you would indeed not specify that X needs package A version 2.4.2. You could see it as X using the 'build recipe' of A as its dependency. So, if you bump the version of A to 2.4.3, then all packages that use A as a dependency will be rebuilt (or substituted from the a binary cache if the Guix/NixOS build infrastructure has already built the updated packages).
        jrochkind1 6y ago
        Huh, in that case I have the converse question -- when specifying dependencies for X, can you not say A 1.x but not 2.x, because 2.x has or is expected to have backwards breaking changes? Otherwise, when A releases 2.0 with some backwards incompatibilities, and all packages that use A as a dependency are rebuilt, don't they break? These issues (allow updates within limits) are what I understand as the point of dependency resolution, I'm trying to understand how you do without it.
  - yjftsjthsd-h 6y ago
    > With respect to inefficiency, what do you mean? It's quite time efficient when building and installing to not have to attempt to resolve dependencies. It's going to burn disk space like crazy, isn't it? If I install packages foo-1.0 and bar-1.0 and foo-1.0 uses glibc-2.31 and bar-1.0 uses glibc-2.30 then I now have 2 versions of glibc... but that scales to every package I install and every library every one of them uses. Of course, this can be fixed... by automatically building new versions of every package any time any of their dependencies changes, in which case we're probably not wasting much disk because we traded and are now burning CPU time like there's no tomorrow (and network, and disk I/O, and memory, and anything else used in package builds). Basically, this sounds like reinventing static binaries, with all the downsides thereof.
    - cbaines 6y ago
      Normally you'd just be using one glibc version. Packages that you installed a while ago may be using a different one, but it's not like every package has massively divergent dependencies. Because of the immutable store, you can do file level deduplication, so if you have multiple versions of the same packages, you can deduplicate the identical files. I think the worries about disk usage are relevant, but only on systems with small amounts of storage. These are still relevant though, and it's an important area to improve on. As for burning CPU time, I don't think there's a perfect solution to avoid this, but I think Guix is pretty good. Guix provides substitutes, so you don't have to build things locally on every machine (I'm looking at you Rubygems, pip and Python stuff is pretty bad also).
ryukafalz 6y ago
As a Guix user/occasional packager who has run into broken packages a few times (including Python ones)... I do wish it had a dependency resolver that could at least be run outside the build/install process, e.g. when updating libraries. If I'm updating a library that has a bunch of dependent packages, it's hard to know whether or not something will break downstream. Sometimes this is unavoidable and you really do need to just test everything thoroughly, but sometimes the dependent packages have more knowledge of what library versions they need... and Guix doesn't seem to be aware of this.
- FRidh 6y ago
  Exactly. Python maintainer of Nixpkgs here. What we need is to be able to use a resolver such as included by pip or poetry, to build up our package set. In Nixpkgs this is nowadays unfortunately done manually, and I suppose the same goes for Guix. In Nixpkgs the reason is simple: too eager pinning makes it impossible to resolve a package set that works with the entire set. Now that pip has a resolver what is needed is a way to use constraints not to set only lower and upper bounds, but to enforce a version when resolving. That makes it usable for downstream integrators to construct their primary package set. One could then even make the next step and construct "stable" sets that extend the primary set.

yjftsjthsd-h 6y ago

Wait, how does guix avoid a dependency resolver? It still has dependencies for packages... are they just hardcoding exact dependencies (including versions) and relying on the "can install multiple versions of a package" property to make that work? That seems inefficient, though I guess maybe they need that for the "this hash means this exact binary" outcome?

cbaines 6y ago
Guix has packages, and packages have inputs (like dependencies), and you're right in that normally package definitions specify the exact dependencies it the code (they're hardcoded). Guix package definitions are truely code though, so if you want to generate packages on the fly by using a dependency resolver, you can totally write some code to make that happen. With respect to inefficiency, what do you mean? It's quite time efficient when building and installing to not have to attempt to resolve dependencies.
- jrochkind1 6y ago
  If they are specifying exact hard-coded dependency versions... if a X -> Y -> A, and B -> A, and M -> N -> A... and a security patch is released for A, then X, Y, B, M and N all need new releases to specify new exact dependencies, in order to use the new A'? It's disastrous for security patches, only highly inconvenient for things like performance improvement releases. But this is why we have dependency resolving, right? What am I missing?
  - cbaines 6y ago
    No, you just update the package definition for A and release the updated package definitions, people will then be using the updated A, whether directly or through other packages (X, Y, B, M, N, ...). There is an issue here of rebuilding all those dependent packages with the updated A, especially if it's something like glibc. Guix includes a mechanism called grafts that allows for package replacements, which allows avoiding this, and this is often used for releasing security fixes.
    - jrochkind1 6y ago
      I think I must be misunderstanding. When you said: > you're right in that normally package definitions specify the exact dependencies it the code (they're hardcoded). I thought that meant that package X might specify a dependency on A version 2.4.2 exactly. So if you want it to use A 2.4.3 instead, a new release of package X would have to be created, specifying 2.4.3. But I think this is not in fact what you mean? In which case I don't yet understand the system you are describing and what you mean by not doing dependency resolution.
      - danieldk 6y ago
        I thought that meant that package X might specify a dependency on A In Nix and Guix, you would indeed not specify that X needs package A version 2.4.2. You could see it as X using the 'build recipe' of A as its dependency. So, if you bump the version of A to 2.4.3, then all packages that use A as a dependency will be rebuilt (or substituted from the a binary cache if the Guix/NixOS build infrastructure has already built the updated packages).
        jrochkind1 6y ago
        Huh, in that case I have the converse question -- when specifying dependencies for X, can you not say A 1.x but not 2.x, because 2.x has or is expected to have backwards breaking changes? Otherwise, when A releases 2.0 with some backwards incompatibilities, and all packages that use A as a dependency are rebuilt, don't they break? These issues (allow updates within limits) are what I understand as the point of dependency resolution, I'm trying to understand how you do without it.
- yjftsjthsd-h 6y ago
  > With respect to inefficiency, what do you mean? It's quite time efficient when building and installing to not have to attempt to resolve dependencies. It's going to burn disk space like crazy, isn't it? If I install packages foo-1.0 and bar-1.0 and foo-1.0 uses glibc-2.31 and bar-1.0 uses glibc-2.30 then I now have 2 versions of glibc... but that scales to every package I install and every library every one of them uses. Of course, this can be fixed... by automatically building new versions of every package any time any of their dependencies changes, in which case we're probably not wasting much disk because we traded and are now burning CPU time like there's no tomorrow (and network, and disk I/O, and memory, and anything else used in package builds). Basically, this sounds like reinventing static binaries, with all the downsides thereof.
  - cbaines 6y ago
    Normally you'd just be using one glibc version. Packages that you installed a while ago may be using a different one, but it's not like every package has massively divergent dependencies. Because of the immutable store, you can do file level deduplication, so if you have multiple versions of the same packages, you can deduplicate the identical files. I think the worries about disk usage are relevant, but only on systems with small amounts of storage. These are still relevant though, and it's an important area to improve on. As for burning CPU time, I don't think there's a perfect solution to avoid this, but I think Guix is pretty good. Guix provides substitutes, so you don't have to build things locally on every machine (I'm looking at you Rubygems, pip and Python stuff is pretty bad also).

ryukafalz 6y ago

As a Guix user/occasional packager who has run into broken packages a few times (including Python ones)... I do wish it had a dependency resolver that could at least be run outside the build/install process, e.g. when updating libraries. If I'm updating a library that has a bunch of dependent packages, it's hard to know whether or not something will break downstream. Sometimes this is unavoidable and you really do need to just test everything thoroughly, but sometimes the dependent packages have more knowledge of what library versions they need... and Guix doesn't seem to be aware of this.

FRidh 6y ago
Exactly. Python maintainer of Nixpkgs here. What we need is to be able to use a resolver such as included by pip or poetry, to build up our package set. In Nixpkgs this is nowadays unfortunately done manually, and I suppose the same goes for Guix. In Nixpkgs the reason is simple: too eager pinning makes it impossible to resolve a package set that works with the entire set. Now that pip has a resolver what is needed is a way to use constraints not to set only lower and upper bounds, but to enforce a version when resolving. That makes it usable for downstream integrators to construct their primary package set. One could then even make the next step and construct "stable" sets that extend the primary set.