6 ms·

I hope Nix (or something like it) starts eating market share from other package managers, Docker, and the like. Nix solves this sort of thing at the cost of one

by l0b0 6y ago

I hope Nix (or something like it) starts eating market share from other package managers, Docker, and the like. Nix solves this sort of thing at the cost of one of the cheapest things available, disk space. Every discussion about it mentions how complex it is; I remember giving up on creating a .deb after a few days of looking into that fractal of complexity, versus producing a Nix package within the first day of looking at the language.

Foxboron 6y ago
I don't think nix solves this. You are still left with having to deal with security issues, updates and tracking on pr package basis instead of once for the entire ecosystem. Admittedly, this is a hard problem. And the languages that does use vendoring makes it hard to programatically inspect all of this. But what do you do if say the python library requests has a severe HTTP parsing issue which allows ACE? How many packages would you need to patch on nix? How many packages would you need to patch on Debian, Arch, Fedora, OpenSuse?
- l0b0 6y ago
  Do you mean as a package maintainer or as an end user? I expect automation and reproducible builds to make this near trivial as a maintainer. As an end user binary diffs will be helpful (not sure if Nix supports them yet), but modern hardware and network connections can easily upgrade a thousand small packages in less than a minute.
  - Foxboron 6y ago
    Reproducible as in build environments or deterministic binaries? Nix has only reproducible build environments. Package maintainer. For the end-user there is no practical difference between a container and nix, and you see how well the container ecosystem is currently handling security updates on their distributed images. The problem is not distributing the fix, it's getting the fix patched.
- Blikkentrekker 6y ago
  Gentoo probably solves it better. The developers allow multiple versions of the same library when there are problems and they have deemed it necessary, but libraries where it makes no real sense are not multislotted accordingly. When developers realize that two packages need a different version of the same library due to issues that should not be, they multislot the library in response or if it be trivial patch whatever package relies on faulty behavior.
  - Foxboron 6y ago
    Everyone can provide multiple versions though, Arch does this. It works well to work around stuff in the existing ecosystem (e.g openssl1.0, gtk2/3/4, qt and so on). But I don't quite see how this solves anything for modern languages like Rust and Go? Do you have any documentation for how this is dealt with in languages that utilizes vendoring to the extreme?
    - Blikkentrekker 6y ago
      The difference with those systems is that the different versions are coded in the package name and that they only go so far to provide different versions of packages are that are designed to be installed as such under different sonames, because they're binary distributions. On Gentoo, they are permitted to rename these libraries arbitrarily since software is compiled locally, so my Krita can be linked to Qt libraries with a different path than your Krita, so if ever Qt would break a.p.i. or a.b.i. despite it not updating it's soname to reflect that, Gentoo could elect to manually rename them and compile what need against the appropriate paths.
- soraminazuki 6y ago
  I think you might be misunderstanding how packaging is handled in Nix. Nix devs use semi-automatic tools to convert packages from programming language ecosystems to Nix packages, but these tools still have means to properly apply patches where necessary. Whether the vendoring approach is used depends on the actual tools being used, but that is mostly irrelevant. Being able to apply patches to all intended packages is a requirement for any packaging tool because patching is absolutely essential for packaging work. > How many packages would you need to patch on nix? So to answer your question, you only need to change a single file. For the requests library, this one[1]. You might also be interested in how Nix manages patches for NPM packages[2]. The amount of manual fixes required is surprisingly few. [1]: https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/python-modules/requests/default.nix https://github.com/NixOS/nixpkgs/blob/master/pkgs/developmen... [2]: https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/node-packages/default.nix https://github.com/NixOS/nixpkgs/blob/master/pkgs/developmen...
  - Foxboron 6y ago
    > Whether the vendoring approach is used depends on the actual tools being used, but that is mostly irrelevant. I don't think it is though? Because... >So to answer your question, you only need to change a single file. For the requests library, this one[1]. You might also be interested in how Nix manages patches for NPM packages[2]. The amount of manual fixes required is surprisingly few. Right, I assume python is easier in this scenario since there are not many cases where a python project would install N different versions of one package. I don't quite understand how these work if a python project depends on separate versions? For the nodejs part I'm more curious. node_modules sometimes contain multiple versions of the same dependency, sometimes across multiple major versions. The patching in the files seems fairly trivial sed replacements and rpath rewrites. But how would security patches be applied across versions? I also took a quick look at the go stuff, and it seems like there is no such thing there as `deleteVendor` defaults to false thus each Go application is self-contained. How would patching dependencies work here? https://github.com/NixOS/nixpkgs/search?q=deleteVendor https://github.com/NixOS/nixpkgs/search?q=deleteVendor
    - 6y ago
chalst 6y ago
Debian stable offers unattended patches, which is something I value highly in public-facing server deployments. I haven't seen anything like this even proposed for NixOS. https://wiki.debian.org/UnattendedUpgrades https://wiki.debian.org/UnattendedUpgrades
- antientropic 6y ago
  NixOS does have unattended automatic upgrades: https://nixos.org/manual/nixos/stable/#sec-upgrading-automatic https://nixos.org/manual/nixos/stable/#sec-upgrading-automat...
  - chalst 6y ago
    Mea culpa: I should have been aware of that. I'm looking at https://status.nixos.org/ https://status.nixos.org/ Are the 6-12 month old releases more stable than the 0-6 month releases? (i.e., is 20.03 more stable than 20.09?)
mlang23 6y ago
Maybe I was just unlucky... But when I tried Nix, the first thing that happened was that it did not deliver what it promised. The package was not properly isolated, so it ended up depending on something from /usr//bin. This was pretty disappointing for a first try. Also, while this is a minor cosmetic thing, I highly dislike the location of the nix-store. It doesn't belong in / and*, long path names (due to the hashes) are very impractical to use. All in all, I hear a lot of good things about Nix, esp. from the Haskell community, but whenever I try it, I develop a deeply rooted dislike for the implementation of the concept.
- soraminazuki 6y ago
  > the package was not properly isolated, so it ended up depending on something from /usr/bin. I don't know when you last used Nix, but Nix now enforces sandboxed builds by default so it should be better at catching these kind of things during packaging. But note that isolation in Nix is mostly a build time thing, and it does not prevent running programs from accessing filesystem paths in /usr. You could still fire up a bash prompt and enter "ls /usr/bin", there's nothing stopping you from doing so. > I highly dislike the location of the nix-store. It doesn't belong in / I see many people express this sentiment, but I'm not sure what's wrong with /nix/store when you've mostly[1] abandoned /usr. Nix is fundamentally incompatible with traditional Unix directory layouts. > long path names (due to the hashes) are very impractical to use That's why you never need to specify them directly. You could either install packages globally and get it symlinked into /run/current-system/sw/bin or ~/.nix-profile/bin, both of which are included in PATH, or use nix-shell and direnv to automatically add packages to PATH whenever you enter a specific directory. [1]: "Mostly," because /usr/bin/env is kept for compatibility
bfrog 6y ago
Yes. With docker you may as well forget about updating anything ever, you really don't know whats in there. Some insecure openssl/nginx versions? Who knows! Can't just update them. The real issue with nix is amount of building that needs to take place. If you go to update openssl, everything dependent on it also needs to be rebuilt to get the latest. The difference in comparison to say static linking or docker though is that its easy to automate. Hydra is proof. It's also easy to continue to use shared libraries, sharing the storage and memory burden. This is unlike docker or everything using static linking. Having a few threadrippers laying around to do all that building might be useful.
- l0b0 6y ago
  > The real issue with nix is amount of building that needs to take place. Good point, but this is one of the best problems to have: it can be solved with modest amounts of money for build infrastructure, and it's unlikely to ever get orders of magnitude worse than it already is.

by l0b0 6y ago

Foxboron 6y ago

I don't think nix solves this. You are still left with having to deal with security issues, updates and tracking on pr package basis instead of once for the entire ecosystem. Admittedly, this is a hard problem. And the languages that does use vendoring makes it hard to programatically inspect all of this. But what do you do if say the python library requests has a severe HTTP parsing issue which allows ACE? How many packages would you need to patch on nix? How many packages would you need to patch on Debian, Arch, Fedora, OpenSuse?

l0b0 6y ago
Do you mean as a package maintainer or as an end user? I expect automation and reproducible builds to make this near trivial as a maintainer. As an end user binary diffs will be helpful (not sure if Nix supports them yet), but modern hardware and network connections can easily upgrade a thousand small packages in less than a minute.
- Foxboron 6y ago
  Reproducible as in build environments or deterministic binaries? Nix has only reproducible build environments. Package maintainer. For the end-user there is no practical difference between a container and nix, and you see how well the container ecosystem is currently handling security updates on their distributed images. The problem is not distributing the fix, it's getting the fix patched.
Blikkentrekker 6y ago
Gentoo probably solves it better. The developers allow multiple versions of the same library when there are problems and they have deemed it necessary, but libraries where it makes no real sense are not multislotted accordingly. When developers realize that two packages need a different version of the same library due to issues that should not be, they multislot the library in response or if it be trivial patch whatever package relies on faulty behavior.
- Foxboron 6y ago
  Everyone can provide multiple versions though, Arch does this. It works well to work around stuff in the existing ecosystem (e.g openssl1.0, gtk2/3/4, qt and so on). But I don't quite see how this solves anything for modern languages like Rust and Go? Do you have any documentation for how this is dealt with in languages that utilizes vendoring to the extreme?
  - Blikkentrekker 6y ago
    The difference with those systems is that the different versions are coded in the package name and that they only go so far to provide different versions of packages are that are designed to be installed as such under different sonames, because they're binary distributions. On Gentoo, they are permitted to rename these libraries arbitrarily since software is compiled locally, so my Krita can be linked to Qt libraries with a different path than your Krita, so if ever Qt would break a.p.i. or a.b.i. despite it not updating it's soname to reflect that, Gentoo could elect to manually rename them and compile what need against the appropriate paths.
soraminazuki 6y ago
I think you might be misunderstanding how packaging is handled in Nix. Nix devs use semi-automatic tools to convert packages from programming language ecosystems to Nix packages, but these tools still have means to properly apply patches where necessary. Whether the vendoring approach is used depends on the actual tools being used, but that is mostly irrelevant. Being able to apply patches to all intended packages is a requirement for any packaging tool because patching is absolutely essential for packaging work. > How many packages would you need to patch on nix? So to answer your question, you only need to change a single file. For the requests library, this one[1]. You might also be interested in how Nix manages patches for NPM packages[2]. The amount of manual fixes required is surprisingly few. [1]: https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/python-modules/requests/default.nix https://github.com/NixOS/nixpkgs/blob/master/pkgs/developmen... [2]: https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/node-packages/default.nix https://github.com/NixOS/nixpkgs/blob/master/pkgs/developmen...
- Foxboron 6y ago
  > Whether the vendoring approach is used depends on the actual tools being used, but that is mostly irrelevant. I don't think it is though? Because... >So to answer your question, you only need to change a single file. For the requests library, this one[1]. You might also be interested in how Nix manages patches for NPM packages[2]. The amount of manual fixes required is surprisingly few. Right, I assume python is easier in this scenario since there are not many cases where a python project would install N different versions of one package. I don't quite understand how these work if a python project depends on separate versions? For the nodejs part I'm more curious. node_modules sometimes contain multiple versions of the same dependency, sometimes across multiple major versions. The patching in the files seems fairly trivial sed replacements and rpath rewrites. But how would security patches be applied across versions? I also took a quick look at the go stuff, and it seems like there is no such thing there as `deleteVendor` defaults to false thus each Go application is self-contained. How would patching dependencies work here? https://github.com/NixOS/nixpkgs/search?q=deleteVendor https://github.com/NixOS/nixpkgs/search?q=deleteVendor
  - 6y ago

chalst 6y ago

Debian stable offers unattended patches, which is something I value highly in public-facing server deployments. I haven't seen anything like this even proposed for NixOS. https://wiki.debian.org/UnattendedUpgrades https://wiki.debian.org/UnattendedUpgrades

antientropic 6y ago
NixOS does have unattended automatic upgrades: https://nixos.org/manual/nixos/stable/#sec-upgrading-automatic https://nixos.org/manual/nixos/stable/#sec-upgrading-automat...
- chalst 6y ago
  Mea culpa: I should have been aware of that. I'm looking at https://status.nixos.org/ https://status.nixos.org/ Are the 6-12 month old releases more stable than the 0-6 month releases? (i.e., is 20.03 more stable than 20.09?)

mlang23 6y ago

Maybe I was just unlucky... But when I tried Nix, the first thing that happened was that it did not deliver what it promised. The package was not properly isolated, so it ended up depending on something from /usr//bin. This was pretty disappointing for a first try. Also, while this is a minor cosmetic thing, I highly dislike the location of the nix-store. It doesn't belong in / and*, long path names (due to the hashes) are very impractical to use. All in all, I hear a lot of good things about Nix, esp. from the Haskell community, but whenever I try it, I develop a deeply rooted dislike for the implementation of the concept.

soraminazuki 6y ago
> the package was not properly isolated, so it ended up depending on something from /usr/bin. I don't know when you last used Nix, but Nix now enforces sandboxed builds by default so it should be better at catching these kind of things during packaging. But note that isolation in Nix is mostly a build time thing, and it does not prevent running programs from accessing filesystem paths in /usr. You could still fire up a bash prompt and enter "ls /usr/bin", there's nothing stopping you from doing so. > I highly dislike the location of the nix-store. It doesn't belong in / I see many people express this sentiment, but I'm not sure what's wrong with /nix/store when you've mostly[1] abandoned /usr. Nix is fundamentally incompatible with traditional Unix directory layouts. > long path names (due to the hashes) are very impractical to use That's why you never need to specify them directly. You could either install packages globally and get it symlinked into /run/current-system/sw/bin or ~/.nix-profile/bin, both of which are included in PATH, or use nix-shell and direnv to automatically add packages to PATH whenever you enter a specific directory. [1]: "Mostly," because /usr/bin/env is kept for compatibility

bfrog 6y ago

Yes. With docker you may as well forget about updating anything ever, you really don't know whats in there. Some insecure openssl/nginx versions? Who knows! Can't just update them. The real issue with nix is amount of building that needs to take place. If you go to update openssl, everything dependent on it also needs to be rebuilt to get the latest. The difference in comparison to say static linking or docker though is that its easy to automate. Hydra is proof. It's also easy to continue to use shared libraries, sharing the storage and memory burden. This is unlike docker or everything using static linking. Having a few threadrippers laying around to do all that building might be useful.

l0b0 6y ago
> The real issue with nix is amount of building that needs to take place. Good point, but this is one of the best problems to have: it can be solved with modest amounts of money for build infrastructure, and it's unlikely to ever get orders of magnitude worse than it already is.