4 ms·

A lot of this sounds familiar to me— I maintain a hodgepodge build tool/pipeline at my org whose current output is a monolithic mega-deb file which is becoming

A lot of this sounds familiar to me— I maintain a hodgepodge build tool/pipeline at my org whose current output is a monolithic mega-deb file which is becoming unwieldy on several fronts (storage, transfer, compression time). I'm really interested in the nix philosophy of separate paths and versioning by hash rather than number, but unfortunately my needs are quite specific, and not having actual experienced nix experts around is a barrier. The temptation right now is to roll our own solution which generates nix-style hash debs that otherwise cooperate with the larger Debian ecosystem we're already very familiar with, but I could probably be sold on giving nix itself a closer look if I could hear from folks (or get pointers to documentation) on a few key points: - What is the story in nix for packaging untagged branches of software, or reasoning about "snapshots" where pools of unreleased repos/packages are able to be treated as a single versionable unit? (think not just a nightly build, but builds corresponding to unmerged feature branches, etc) - Does the nix hash account for dependencies only changing the hash for ABI-impacting changes such as when a header file changes? Or does it change dependent hashes always? Or never? - I have an existing system for managing packaging metadata which I don't want to migrate from. How much trouble will I get into if I want to generate the metadata on the fly each time (as I currently do for my debs)? - How much pain is it to roll a nix package "by hand" (basically with the dpkg-deb equivalent tool rather than the dpkg-buildpackage equivalent tool)? - Nix isn't supported in Artifactory (RTFACT-19998 has been open since 2019). Nominally, I can use the dumb WebDAV option, but is that going to affect my user experience and/or will it be a maintenance headache? - What is the apt/nix interop story? I would likely need it to be bidirectional, so that my nix workspace could depend on system debs that I don't want to port over, but also potentially have "gateway" debs which able to do the opposite, of depending on the nix workspace from a deb shim, and installing/updating it in the postinst. I've researched all of these in the past at various points and had a lot of trouble finding clear, straight answers on them. I'd love to hear from other users or nix experts, though, who could help calm our fears on this.

q3k 6y ago
> What is the story in nix for packaging untagged branches of software, or reasoning about "snapshots" where pools of unreleased repos/packages are able to be treated as a single versionable unit? A Nix derivation (a.k.a. 'packaging unit') can be built from sources either fetched remotely, or from a local directory, or a combination of both; plus any other derivation that it depends on. Any version semantics come from how you implement them, abstracted and formalized in any way you want. Nix is a programming language, you are free to do what you want. > Does the nix hash account for dependencies only changing the hash for ABI-impacting changes such as when a header file changes? Or does it change dependent hashes always? Or never? By default, any change to any dependency will cause all dependent derivations to be rebuilt, and their hashes to also change. You can 'break' this by introducing stable ABI barriers preventing rebuilds and performing runtime loads of dependencies from .so (or whatever, executing some binaries from $PATH). This is for instance how my firefox can handle any OpenGL library that it can use at runtime, without having a firefox rebuilt for every possible GPU driver out there. > I have an existing system for managing packaging metadata which I don't want to migrate from. How much trouble will I get into if I want to generate the metadata on the fly each time (as I currently do for my debs)? You will have to generate nix source code, or importable nix source code, from that system, and pull that into your local nix source tree. All nix evaluation is done from purely defined locally available sources, or sources fetched from remote resources, but marked with some consistent hash. > How much pain is it to roll a nix package "by hand" (basically with the dpkg-deb equivalent tool rather than the dpkg-buildpackage equivalent tool)? Nix does not work this way. Nix derivations are a recipe on how to build a package, you have to specify that in one way or another so that Nix can perform the build for you, and reason about its status within a larger build graph. You can 'cheat' building derivations by eg. importing binary builds and only patching them up to work under Nix(OS), but generally you shouldn't. > Nix isn't supported in Artifactory (RTFACT-19998 has been open since 2019). Nominally, I can use the dumb WebDAV option, but is that going to affect my user experience and/or will it be a maintenance headache? You will need to run a Nix cache, that CI will push into and other consumers (all having a checkout of your nix definitions) will pull from instead of building everything from scratch themselves. > What is the apt/nix interop story? I would likely need it to be bidirectional, so that my nix workspace could depend on system debs that I don't want to port over, but also potentially have "gateway" debs which able to do the opposite, of depending on the nix workspace from a deb shim, and installing/updating it in the postinst. Not aware of anything out of the box. Nix builds only run on Nix (as they all need dependencies from /nix/store/..., and that is populated by Nix itself), so don't expect to easily run them on Debian as normal dpkgs without some disgusting home-rolled hacks. You could technically consume binary .debs from Debian within nix derivations (see previous point), but that's ugly. Generally, just install Nix on all your hosts.
- georgyo 6y ago
  > Not aware of anything out of the box. Nix builds only run on Nix (as they all need dependencies from /nix/store/..., and that is populated by Nix itself), so don't expect to easily run them on Debian without some disgusting home-rolled hacks. You could technically consume binary .debs from Debian within nix derivations (see previous point), but that's ugly. This is not true and no hacks are needed. Since nix only depends on /nix it can be used in combination with ANY linux OS. In fact there is an official debian binary for nix: https://packages.debian.org/bullseye/nix-bin https://packages.debian.org/bullseye/nix-bin
  - q3k 6y ago
    Yeah, I just added a comment that this is only true if you want to not run Nix on Debian, eg. use Nix builds are pure dpkgs that you install to 'unaware' Debian hosts.
  - danieldk 6y ago
    This is not true and no hacks are needed. Since nix only depends on /nix it can be used in combination with ANY linux OS. Just so that noone runs into issues. This is not completely true. E.g. you cannot use Nix on Fedora Silverblue, because it also uses an immutable filesystem and you cannot make top-level directories like /nix without ugly hacks. Also, multi-user Nix does not work on most distributions that enforce an SELinux policy (e.g. Fedora/CentOS/RHEL). Incorrect SELinux contexts get set on store files/directories. Single-user Nix typically works fine though. (These issues can be worked around with rootless Nix, but that is much more high-friction than a globally installed Nix.)
- mikepurvis 6y ago
  > By default, any change to any dependency will cause all dependent derivations to be rebuilt, and their hashes to also change. You can 'break' this by introducing stable ABI barriers preventing rebuilds and performing runtime loads of dependencies from .so (or whatever, executing some binaries from $PATH). Does this imply full control over hash holding/breaking for derivation authors, or is it like a single field that's basically just meant to be used for the soname? I would love to see more docs detailing how this works and what the user extension points are for it. Most of what I've found is pretty high level.
georgyo 6y ago
That's a lot of questions - What is the story in nix for packaging untagged branches of software If you want to do unreleased software, you would create overlays that added them. You're overlays most likely would not need to define the build, just create an override of the one in nixpkgs with a new source: https://discourse.nixos.org/t/how-to-override-package-version/2889 https://discourse.nixos.org/t/how-to-override-package-versio... - Does the nix hash account for dependencies only changing the hash for ABI-impacting changes such as when a header file changes? The nix hash changes whenever anything in the derivation or it dependencies changes. It has no concept of what an ABI-impacting change would be. - I have an existing system for managing packaging metadata which I don't want to migrate from. How much trouble will I get into if I want to generate the metadata on the fly each time (as I currently do for my debs)? The nix code _is_ the metadata. There isn't an equivalent Packages like there is for Debian. - How much pain is it to roll a nix package "by hand" (basically with the dpkg-deb equivalent tool rather than the dpkg-buildpackage equivalent tool)? Don't fully understand what you are asking here, but the equivalent tool of dpkg-deb would be nix-build. - Nix isn't supported in Artifactory (RTFACT-19998 has been open since 2019). Nominally, I can use the dumb WebDAV option, but is that going to affect my user experience and/or will it be a maintenance headache? Nix only writes new files to a remote store. As such nix caches can be served by anything that can serve files. I personally upload to S3 and have artifactory use that as a remote. - What is the apt/nix interop story? This is a complex question. At work I use nix on debian hosts, and it works very well. Nix packages have _no dependencies_ at all on the host OS. IE binary installed from nix will use required libraries from nix as well. However nix creates environments were only the packages explicitly installed show up. As such if you install firefox only bin/firefox will be added to that environment, despite using all of its dependencies from nix. Nix hard references all it dependencies to the /nix directory.
- mikepurvis 6y ago
  > Nix only writes new files to a remote store. As such nix caches can be served by anything that can serve files. I personally upload to S3 and have artifactory use that as a remote. What about the maintenance side, though, like being able to clean/reap old builds from the cache, reason about which ones are part of important/supported chains vs throwaway builds from merge pipelines, etc?
duijf 6y ago
> What is the story in nix for packaging untagged branches of software Most Nix packages define their sources using `src`. You can pass in whatever you want. E.g. fetch a tarball from GitHub with `fetchUrl`, fetch from a git repository, or point to a local directory. > or reasoning about "snapshots" where pools of unreleased repos/packages are able to be treated as a single versionable unit? This is basically what the Nixpkgs collection is minus the "unreleased" part. You can extend Nixpkgs with the packages that you care about using overlays. You have control over the stuff that you put in the overlay, so you can put in unreleased software as well like I described above. > - I have an existing system for managing packaging metadata which I don't want to migrate from. How much trouble will I get into if I want to generate the metadata on the fly each time (as I currently do for my debs)? This is difficult to answer without knowing more details. - You can generate the Nix source files based on the metadata like the sibling comment pointed out. - Or you could read the metadata with `builtins.fromJSON` and generate Nix derivations programmatically from within Nix. > - What is the apt/nix interop story? If you don't want to bother packaging certain applications using Nix (or work towards that gradually), you can. Nothing prevents you from referencing things outside of `/nix/store` in a Nix package. You can launch stuff from `/usr/bin` from a Nix binary if you are so inclined. Libraries are going to be tricky though. As for the other way around, I'm aware of two options: - At Channable, we would package the Nix closure of a package in a `.deb`. This works, but you run into trouble when multiple debs need the same store path. You can work around this by packaging the Nix closure under e.g. `/var/lib/<your_package>/nix/store` and bind mounting to `/nix/store` before launching. - Eventually we realized we didn't want the `/nix/store` stuff in our `.deb` packages. IIRC we created a `postinst` script which ran `nix-env` to realize a store path from our cache + create a GC root so it doesn't get garbage collected. Some of that stuff may have changed since I left. Feel free to contact me if you'd like to discuss further. Info is in my profile.
- mikepurvis 6y ago
  Thanks for the response! > This is difficult to answer without knowing more details. The situation specifically is the ROS ecosystem, where metadata is managed in these package.xml files: https://github.com/ros2/rclcpp/blob/master/rclcpp/package.xml https://github.com/ros2/rclcpp/blob/master/rclcpp/package.xm... The federated nature of the ecosystem has led to a culture where it's very normal to be building dozens of these at once, in the same workspace together, often from multiple repos (the repo above has four in it). So there are several build tools which automate the work of examining a source workspace and building all the packages within it in the correct topological order, in parallel, while respecting build_depend tags. The newest of these tools (colcon) has actually made the package.xml optional in many cases, as it can examine CMakelists, setup.py, BUILD, etc, and discover for itself what the dependencies are. Your "distribution" of ROS is formed by listing all the packages and repos in this big file, for which there is other tooling to manage pulling dependency sources, whatever: https://github.com/ros/rosdistro/blob/master/foxy/distribution.yaml https://github.com/ros/rosdistro/blob/master/foxy/distributi... Anyway, so the existing ROS/nix efforts (1) seem to basically consume all of this package/distribution metadata at once and generate a giant parallel structure of nix definitions (eg https://github.com/lopsided98/nix-ros-overlay/blob/master/distros/foxy/rclcpp/default.nix https://github.com/lopsided98/nix-ros-overlay/blob/master/di...), which I fear would be completely opaque to my users. And any system which required a flag day where everyone leaves behind these existing workflows would be an immediate non-starter. I think the ideal scenario (and what it would look like if I built this myself based on debs) would be that you could source the "base" workspace as usual (enter the nix-shell?), and check out source, build groups of packages as usual with colcon, but there'd be an extra plugin/verb/flag for it, which would make it build each package as a nix package instead of into the usual installspace. The verb would generate the nix definitions on the fly, and probably handle the invocation and build-parallelism side of it as well. [1]: https://github.com/acowley/ros2nix https://github.com/acowley/ros2nix, https://github.com/lopsided98/nix-ros-overlay https://github.com/lopsided98/nix-ros-overlay

q3k 6y ago

> What is the story in nix for packaging untagged branches of software, or reasoning about "snapshots" where pools of unreleased repos/packages are able to be treated as a single versionable unit? A Nix derivation (a.k.a. 'packaging unit') can be built from sources either fetched remotely, or from a local directory, or a combination of both; plus any other derivation that it depends on. Any version semantics come from how you implement them, abstracted and formalized in any way you want. Nix is a programming language, you are free to do what you want. > Does the nix hash account for dependencies only changing the hash for ABI-impacting changes such as when a header file changes? Or does it change dependent hashes always? Or never? By default, any change to any dependency will cause all dependent derivations to be rebuilt, and their hashes to also change. You can 'break' this by introducing stable ABI barriers preventing rebuilds and performing runtime loads of dependencies from .so (or whatever, executing some binaries from $PATH). This is for instance how my firefox can handle any OpenGL library that it can use at runtime, without having a firefox rebuilt for every possible GPU driver out there. > I have an existing system for managing packaging metadata which I don't want to migrate from. How much trouble will I get into if I want to generate the metadata on the fly each time (as I currently do for my debs)? You will have to generate nix source code, or importable nix source code, from that system, and pull that into your local nix source tree. All nix evaluation is done from purely defined locally available sources, or sources fetched from remote resources, but marked with some consistent hash. > How much pain is it to roll a nix package "by hand" (basically with the dpkg-deb equivalent tool rather than the dpkg-buildpackage equivalent tool)? Nix does not work this way. Nix derivations are a recipe on how to build a package, you have to specify that in one way or another so that Nix can perform the build for you, and reason about its status within a larger build graph. You can 'cheat' building derivations by eg. importing binary builds and only patching them up to work under Nix(OS), but generally you shouldn't. > Nix isn't supported in Artifactory (RTFACT-19998 has been open since 2019). Nominally, I can use the dumb WebDAV option, but is that going to affect my user experience and/or will it be a maintenance headache? You will need to run a Nix cache, that CI will push into and other consumers (all having a checkout of your nix definitions) will pull from instead of building everything from scratch themselves. > What is the apt/nix interop story? I would likely need it to be bidirectional, so that my nix workspace could depend on system debs that I don't want to port over, but also potentially have "gateway" debs which able to do the opposite, of depending on the nix workspace from a deb shim, and installing/updating it in the postinst. Not aware of anything out of the box. Nix builds only run on Nix (as they all need dependencies from /nix/store/..., and that is populated by Nix itself), so don't expect to easily run them on Debian as normal dpkgs without some disgusting home-rolled hacks. You could technically consume binary .debs from Debian within nix derivations (see previous point), but that's ugly. Generally, just install Nix on all your hosts.

georgyo 6y ago
> Not aware of anything out of the box. Nix builds only run on Nix (as they all need dependencies from /nix/store/..., and that is populated by Nix itself), so don't expect to easily run them on Debian without some disgusting home-rolled hacks. You could technically consume binary .debs from Debian within nix derivations (see previous point), but that's ugly. This is not true and no hacks are needed. Since nix only depends on /nix it can be used in combination with ANY linux OS. In fact there is an official debian binary for nix: https://packages.debian.org/bullseye/nix-bin https://packages.debian.org/bullseye/nix-bin
- q3k 6y ago
  Yeah, I just added a comment that this is only true if you want to not run Nix on Debian, eg. use Nix builds are pure dpkgs that you install to 'unaware' Debian hosts.
- danieldk 6y ago
  This is not true and no hacks are needed. Since nix only depends on /nix it can be used in combination with ANY linux OS. Just so that noone runs into issues. This is not completely true. E.g. you cannot use Nix on Fedora Silverblue, because it also uses an immutable filesystem and you cannot make top-level directories like /nix without ugly hacks. Also, multi-user Nix does not work on most distributions that enforce an SELinux policy (e.g. Fedora/CentOS/RHEL). Incorrect SELinux contexts get set on store files/directories. Single-user Nix typically works fine though. (These issues can be worked around with rootless Nix, but that is much more high-friction than a globally installed Nix.)
mikepurvis 6y ago
> By default, any change to any dependency will cause all dependent derivations to be rebuilt, and their hashes to also change. You can 'break' this by introducing stable ABI barriers preventing rebuilds and performing runtime loads of dependencies from .so (or whatever, executing some binaries from $PATH). Does this imply full control over hash holding/breaking for derivation authors, or is it like a single field that's basically just meant to be used for the soname? I would love to see more docs detailing how this works and what the user extension points are for it. Most of what I've found is pretty high level.

georgyo 6y ago

That's a lot of questions - What is the story in nix for packaging untagged branches of software If you want to do unreleased software, you would create overlays that added them. You're overlays most likely would not need to define the build, just create an override of the one in nixpkgs with a new source: https://discourse.nixos.org/t/how-to-override-package-version/2889 https://discourse.nixos.org/t/how-to-override-package-versio... - Does the nix hash account for dependencies only changing the hash for ABI-impacting changes such as when a header file changes? The nix hash changes whenever anything in the derivation or it dependencies changes. It has no concept of what an ABI-impacting change would be. - I have an existing system for managing packaging metadata which I don't want to migrate from. How much trouble will I get into if I want to generate the metadata on the fly each time (as I currently do for my debs)? The nix code _is_ the metadata. There isn't an equivalent Packages like there is for Debian. - How much pain is it to roll a nix package "by hand" (basically with the dpkg-deb equivalent tool rather than the dpkg-buildpackage equivalent tool)? Don't fully understand what you are asking here, but the equivalent tool of dpkg-deb would be nix-build. - Nix isn't supported in Artifactory (RTFACT-19998 has been open since 2019). Nominally, I can use the dumb WebDAV option, but is that going to affect my user experience and/or will it be a maintenance headache? Nix only writes new files to a remote store. As such nix caches can be served by anything that can serve files. I personally upload to S3 and have artifactory use that as a remote. - What is the apt/nix interop story? This is a complex question. At work I use nix on debian hosts, and it works very well. Nix packages have _no dependencies_ at all on the host OS. IE binary installed from nix will use required libraries from nix as well. However nix creates environments were only the packages explicitly installed show up. As such if you install firefox only bin/firefox will be added to that environment, despite using all of its dependencies from nix. Nix hard references all it dependencies to the /nix directory.

mikepurvis 6y ago
> Nix only writes new files to a remote store. As such nix caches can be served by anything that can serve files. I personally upload to S3 and have artifactory use that as a remote. What about the maintenance side, though, like being able to clean/reap old builds from the cache, reason about which ones are part of important/supported chains vs throwaway builds from merge pipelines, etc?

duijf 6y ago

> What is the story in nix for packaging untagged branches of software Most Nix packages define their sources using `src`. You can pass in whatever you want. E.g. fetch a tarball from GitHub with `fetchUrl`, fetch from a git repository, or point to a local directory. > or reasoning about "snapshots" where pools of unreleased repos/packages are able to be treated as a single versionable unit? This is basically what the Nixpkgs collection is minus the "unreleased" part. You can extend Nixpkgs with the packages that you care about using overlays. You have control over the stuff that you put in the overlay, so you can put in unreleased software as well like I described above. > - I have an existing system for managing packaging metadata which I don't want to migrate from. How much trouble will I get into if I want to generate the metadata on the fly each time (as I currently do for my debs)? This is difficult to answer without knowing more details. - You can generate the Nix source files based on the metadata like the sibling comment pointed out. - Or you could read the metadata with `builtins.fromJSON` and generate Nix derivations programmatically from within Nix. > - What is the apt/nix interop story? If you don't want to bother packaging certain applications using Nix (or work towards that gradually), you can. Nothing prevents you from referencing things outside of `/nix/store` in a Nix package. You can launch stuff from `/usr/bin` from a Nix binary if you are so inclined. Libraries are going to be tricky though. As for the other way around, I'm aware of two options: - At Channable, we would package the Nix closure of a package in a `.deb`. This works, but you run into trouble when multiple debs need the same store path. You can work around this by packaging the Nix closure under e.g. `/var/lib/<your_package>/nix/store` and bind mounting to `/nix/store` before launching. - Eventually we realized we didn't want the `/nix/store` stuff in our `.deb` packages. IIRC we created a `postinst` script which ran `nix-env` to realize a store path from our cache + create a GC root so it doesn't get garbage collected. Some of that stuff may have changed since I left. Feel free to contact me if you'd like to discuss further. Info is in my profile.

mikepurvis 6y ago
Thanks for the response! > This is difficult to answer without knowing more details. The situation specifically is the ROS ecosystem, where metadata is managed in these package.xml files: https://github.com/ros2/rclcpp/blob/master/rclcpp/package.xml https://github.com/ros2/rclcpp/blob/master/rclcpp/package.xm... The federated nature of the ecosystem has led to a culture where it's very normal to be building dozens of these at once, in the same workspace together, often from multiple repos (the repo above has four in it). So there are several build tools which automate the work of examining a source workspace and building all the packages within it in the correct topological order, in parallel, while respecting build_depend tags. The newest of these tools (colcon) has actually made the package.xml optional in many cases, as it can examine CMakelists, setup.py, BUILD, etc, and discover for itself what the dependencies are. Your "distribution" of ROS is formed by listing all the packages and repos in this big file, for which there is other tooling to manage pulling dependency sources, whatever: https://github.com/ros/rosdistro/blob/master/foxy/distribution.yaml https://github.com/ros/rosdistro/blob/master/foxy/distributi... Anyway, so the existing ROS/nix efforts (1) seem to basically consume all of this package/distribution metadata at once and generate a giant parallel structure of nix definitions (eg https://github.com/lopsided98/nix-ros-overlay/blob/master/distros/foxy/rclcpp/default.nix https://github.com/lopsided98/nix-ros-overlay/blob/master/di...), which I fear would be completely opaque to my users. And any system which required a flag day where everyone leaves behind these existing workflows would be an immediate non-starter. I think the ideal scenario (and what it would look like if I built this myself based on debs) would be that you could source the "base" workspace as usual (enter the nix-shell?), and check out source, build groups of packages as usual with colcon, but there'd be an extra plugin/verb/flag for it, which would make it build each package as a nix package instead of into the usual installspace. The verb would generate the nix definitions on the fly, and probably handle the invocation and build-parallelism side of it as well. [1]: https://github.com/acowley/ros2nix https://github.com/acowley/ros2nix, https://github.com/lopsided98/nix-ros-overlay https://github.com/lopsided98/nix-ros-overlay