12 ms·

I don't think nix solves this. You are still left with having to deal with security issues, updates and tracking on pr package basis instead of once for the ent

I don't think nix solves this. You are still left with having to deal with security issues, updates and tracking on pr package basis instead of once for the entire ecosystem. Admittedly, this is a hard problem. And the languages that does use vendoring makes it hard to programatically inspect all of this. But what do you do if say the python library requests has a severe HTTP parsing issue which allows ACE? How many packages would you need to patch on nix? How many packages would you need to patch on Debian, Arch, Fedora, OpenSuse?

l0b0 6y ago
Do you mean as a package maintainer or as an end user? I expect automation and reproducible builds to make this near trivial as a maintainer. As an end user binary diffs will be helpful (not sure if Nix supports them yet), but modern hardware and network connections can easily upgrade a thousand small packages in less than a minute.
- Foxboron 6y ago
  Reproducible as in build environments or deterministic binaries? Nix has only reproducible build environments. Package maintainer. For the end-user there is no practical difference between a container and nix, and you see how well the container ecosystem is currently handling security updates on their distributed images. The problem is not distributing the fix, it's getting the fix patched.
Blikkentrekker 6y ago
Gentoo probably solves it better. The developers allow multiple versions of the same library when there are problems and they have deemed it necessary, but libraries where it makes no real sense are not multislotted accordingly. When developers realize that two packages need a different version of the same library due to issues that should not be, they multislot the library in response or if it be trivial patch whatever package relies on faulty behavior.
- Foxboron 6y ago
  Everyone can provide multiple versions though, Arch does this. It works well to work around stuff in the existing ecosystem (e.g openssl1.0, gtk2/3/4, qt and so on). But I don't quite see how this solves anything for modern languages like Rust and Go? Do you have any documentation for how this is dealt with in languages that utilizes vendoring to the extreme?
  - Blikkentrekker 6y ago
    The difference with those systems is that the different versions are coded in the package name and that they only go so far to provide different versions of packages are that are designed to be installed as such under different sonames, because they're binary distributions. On Gentoo, they are permitted to rename these libraries arbitrarily since software is compiled locally, so my Krita can be linked to Qt libraries with a different path than your Krita, so if ever Qt would break a.p.i. or a.b.i. despite it not updating it's soname to reflect that, Gentoo could elect to manually rename them and compile what need against the appropriate paths.
soraminazuki 6y ago
I think you might be misunderstanding how packaging is handled in Nix. Nix devs use semi-automatic tools to convert packages from programming language ecosystems to Nix packages, but these tools still have means to properly apply patches where necessary. Whether the vendoring approach is used depends on the actual tools being used, but that is mostly irrelevant. Being able to apply patches to all intended packages is a requirement for any packaging tool because patching is absolutely essential for packaging work. > How many packages would you need to patch on nix? So to answer your question, you only need to change a single file. For the requests library, this one[1]. You might also be interested in how Nix manages patches for NPM packages[2]. The amount of manual fixes required is surprisingly few. [1]: https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/python-modules/requests/default.nix https://github.com/NixOS/nixpkgs/blob/master/pkgs/developmen... [2]: https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/node-packages/default.nix https://github.com/NixOS/nixpkgs/blob/master/pkgs/developmen...
- Foxboron 6y ago
  > Whether the vendoring approach is used depends on the actual tools being used, but that is mostly irrelevant. I don't think it is though? Because... >So to answer your question, you only need to change a single file. For the requests library, this one[1]. You might also be interested in how Nix manages patches for NPM packages[2]. The amount of manual fixes required is surprisingly few. Right, I assume python is easier in this scenario since there are not many cases where a python project would install N different versions of one package. I don't quite understand how these work if a python project depends on separate versions? For the nodejs part I'm more curious. node_modules sometimes contain multiple versions of the same dependency, sometimes across multiple major versions. The patching in the files seems fairly trivial sed replacements and rpath rewrites. But how would security patches be applied across versions? I also took a quick look at the go stuff, and it seems like there is no such thing there as `deleteVendor` defaults to false thus each Go application is self-contained. How would patching dependencies work here? https://github.com/NixOS/nixpkgs/search?q=deleteVendor https://github.com/NixOS/nixpkgs/search?q=deleteVendor
  - 6y ago

l0b0 6y ago

Do you mean as a package maintainer or as an end user? I expect automation and reproducible builds to make this near trivial as a maintainer. As an end user binary diffs will be helpful (not sure if Nix supports them yet), but modern hardware and network connections can easily upgrade a thousand small packages in less than a minute.

Foxboron 6y ago
Reproducible as in build environments or deterministic binaries? Nix has only reproducible build environments. Package maintainer. For the end-user there is no practical difference between a container and nix, and you see how well the container ecosystem is currently handling security updates on their distributed images. The problem is not distributing the fix, it's getting the fix patched.

Blikkentrekker 6y ago

Gentoo probably solves it better. The developers allow multiple versions of the same library when there are problems and they have deemed it necessary, but libraries where it makes no real sense are not multislotted accordingly. When developers realize that two packages need a different version of the same library due to issues that should not be, they multislot the library in response or if it be trivial patch whatever package relies on faulty behavior.

Foxboron 6y ago
Everyone can provide multiple versions though, Arch does this. It works well to work around stuff in the existing ecosystem (e.g openssl1.0, gtk2/3/4, qt and so on). But I don't quite see how this solves anything for modern languages like Rust and Go? Do you have any documentation for how this is dealt with in languages that utilizes vendoring to the extreme?
- Blikkentrekker 6y ago
  The difference with those systems is that the different versions are coded in the package name and that they only go so far to provide different versions of packages are that are designed to be installed as such under different sonames, because they're binary distributions. On Gentoo, they are permitted to rename these libraries arbitrarily since software is compiled locally, so my Krita can be linked to Qt libraries with a different path than your Krita, so if ever Qt would break a.p.i. or a.b.i. despite it not updating it's soname to reflect that, Gentoo could elect to manually rename them and compile what need against the appropriate paths.

soraminazuki 6y ago

I think you might be misunderstanding how packaging is handled in Nix. Nix devs use semi-automatic tools to convert packages from programming language ecosystems to Nix packages, but these tools still have means to properly apply patches where necessary. Whether the vendoring approach is used depends on the actual tools being used, but that is mostly irrelevant. Being able to apply patches to all intended packages is a requirement for any packaging tool because patching is absolutely essential for packaging work. > How many packages would you need to patch on nix? So to answer your question, you only need to change a single file. For the requests library, this one[1]. You might also be interested in how Nix manages patches for NPM packages[2]. The amount of manual fixes required is surprisingly few. [1]: https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/python-modules/requests/default.nix https://github.com/NixOS/nixpkgs/blob/master/pkgs/developmen... [2]: https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/node-packages/default.nix https://github.com/NixOS/nixpkgs/blob/master/pkgs/developmen...

Foxboron 6y ago
> Whether the vendoring approach is used depends on the actual tools being used, but that is mostly irrelevant. I don't think it is though? Because... >So to answer your question, you only need to change a single file. For the requests library, this one[1]. You might also be interested in how Nix manages patches for NPM packages[2]. The amount of manual fixes required is surprisingly few. Right, I assume python is easier in this scenario since there are not many cases where a python project would install N different versions of one package. I don't quite understand how these work if a python project depends on separate versions? For the nodejs part I'm more curious. node_modules sometimes contain multiple versions of the same dependency, sometimes across multiple major versions. The patching in the files seems fairly trivial sed replacements and rpath rewrites. But how would security patches be applied across versions? I also took a quick look at the go stuff, and it seems like there is no such thing there as `deleteVendor` defaults to false thus each Go application is self-contained. How would patching dependencies work here? https://github.com/NixOS/nixpkgs/search?q=deleteVendor https://github.com/NixOS/nixpkgs/search?q=deleteVendor
- 6y ago