5 ms·

Nixhub: Search Historical Versions of Nix Packages

68 points by clessg 3y ago

moojd 3y ago
Oh man I've needed this for a while to the point where I was about to build it my self so thanks! It would be great though if it also listed the tagged nixpkgs version in addition to the hash if applicable.
jljljl 3y ago
Hey! Author of this post here, happy to answer any questions about Nixhub!
- nathan_phoenix 3y ago
  First off, amazing tool and thanks for doing the nix community a favor! If you could expand more on the architecture of the tool I'd be very curious! Like what goes all on from parsing Hydra builds to getting a useful package versions DB and what were the challenges while building it? Also, how are you merging all the nodejs packages (nodejs_20, nodejs_18, ...) into one “nodejs”? I'm also in the process of building something very similar (didn't know the idea was that popular haha), but I'm not parsing Hydra builds but the output from `nix-env -qa --json` to get all the package versions and it's proving more involved than I anticipated (:
  - jljljl 3y ago
    Thank you! We're planning a longer post that explains the architecture and process of building the index. An extremely basic description of what we do can be found here: https://www.jetpack.io/blog/0-5-0-install-nix-packages-by-version/#:~:text=Our%20Solution%3A%20a%20Devbox%20Package%20Index https://www.jetpack.io/blog/0-5-0-install-nix-packages-by-ve... First, we processed build outputs from Hydra to map package names + version numbers to commits in the Nixpkgs repo. We then created a service that stores this index as a prefix tree, making it easy to search for a specific package and version in Nix. There's some extra cleanup required to make the index really usable, which we'll share in the follow up post For the merged packages -- we have a hardcoded "canonical names", which map the various attribute paths like `nodejs_20` and `nodejs_18` to a single, easier to search name (like `nodejs`). We originally designed this so that Devbox users (who may not be familiar with Nix naming conventions) can just type `devbox add nodejs@20` and get the right package version.
    - nathan_phoenix 3y ago
      Thanks for the answer! Looking forward to the followup post :D And yeah, nix naming conventions are sometimes a mess. Your approach does seem much nicer. Last question if you have a bit more time. How come that you don't have every version of a package available? Is it because you only parse the Hydra stable 23 builds (which I guess aren't updated on every version) or because you parse them only periodically like Lazamar's Nix Package Versions?
      - jljljl 3y ago
        We're parsing periodically (though our index is more up to date than Nix Package Versions at the moment). We've also truncated pre-Flakes commits, though we are planning to include those in the future
        nathan_phoenix 3y ago
        Looking forward to that for completeness sake! Btw, regarding a bit older pre-Flakes commits, at least some packages will error out while nix-installing on M1 (or newer) macbooks because at that point nix packages didn't anticipate macbooks ever ditching x86 for aarch... Hope this info helps you out, at least took me some time to realize that the random obscure error was caused by this. Can be almost always fixed by appending "--system darwin-x86_64" to the nix command to use Apple's Rosetta binary translation, cheers.
noahmasur 3y ago
Roughly equivalent tool that I've used in the past: https://lazamar.co.uk/nix-versions/ https://lazamar.co.uk/nix-versions/
stabbles 3y ago
It's interesting how Nix's selling point is that its non-standard file system structure allows multiple versions/flavors of the same package to be installed at the same time, yet if I understand correctly, it often has one version per package per Nix version? It's more like install multiple versions of the same package at a different time? Compare this to the model we have in Spack (which is inspired by nix), where there's one recipe per package for many versions, and you can define different build flavors, conditional dependencies (and conditional pretty much everything). The difference is that Spack has a dependency solver and Nix apparently does not? For example the curl package [1] has tons of versions, and its `tls` package variant has conditional values depending on the version: variant("tls", values=("gnutls", conditional("mbedtls", when="@7.46:"), ...), multi=True) And dependencies are specified as depends_on("gnutls", when="tls=gnutls") with when("tls=mbedtls"): depends_on("mbedtls@2:", when="@7.79:") depends_on("mbedtls@:2", when="@:7.78") This allows you both to install all sorts of curls, like `spack install curl tls=mbedtls` (gives you something >= 7.46), or `spack install curl@7.77 tls=mbedtls` gives you a curl with an old mbedtls, etc. [1] https://github.com/spack/spack/blob/develop/var/spack/repos/builtin/packages/curl/package.py https://github.com/spack/spack/blob/develop/var/spack/repos/...
- _7tgr 3y ago
  Probably because it's pretty easy to just use an older package from a different version of Nixpkgs (just use an override that defines the attribute for a package in your current Nixpkgs as the same package in an older Nixpkgs). It'll transparently pull the older version from the binary cache, too, if available. The only difficulty is finding the right Nixpkgs commit hash, which this solves. I think I find the Nix way to be a bit cleaner, though I'm unaware of the full set of pros/cons. Worth noting that the Nix package definition for curl at [1] is a little easier to read for me since it doesn't have the duplicated code for each version. [1] https://github.com/NixOS/nixpkgs/blob/master/pkgs/tools/networking/curl/default.nix https://github.com/NixOS/nixpkgs/blob/master/pkgs/tools/netw...
- nathan_phoenix 3y ago
  > yet if I understand correctly, it often has one version per package per Nix version? Both yeah and no. Taking the example of nodejs, you have all the supported major releases like nodejs_16, nodejs_18, etc. for each Nix revision[1], but within each revision you only have a specific version of the package. E.g. nodejs_16 is 16.3.1 for revision at date X. There are benefits and drawbacks of managing a package repo like that, with revisions which implicitely pin all packages and it's dependencies at once, but it's too long of a topic to write about here. > For example the curl package [1] has tons of versions [...] So does it in nix (curlMinimal, curlWithGnuTls, curlHTTP3, etc)[2]. In that regard nix and Spack are similar. [1]: A revision is like a shapshot of all the packages at a given point in time. [2]: https://search.nixos.org/packages?channel=23.05&from=0&size=50&sort=relevance&type=packages&query=curl https://search.nixos.org/packages?channel=23.05&from=0&size=...
hamandcheese 3y ago
For most people, depending on historic versions of nixpkgs is an anti-pattern. The main reason I say this is security. You may want an old version of, say, ruby, but you probably don't want the old, insecure libraries it was built against (i.e. glibc, openssl). It is often just as easy to use overrides to build a different version of a piece of software, and in the rare case that doesn't work you can fall back to a custom derivation (drawing inspiration from the old version of nixpkgs, perhaps). I worry about tools like this which make doing the wrong thing too easy.
- wredue 3y ago
  Yeah. People like to push that there are “legitimate reasons” to want old versions, but it usually comes down to businesses. I would suggest instead that there are no legitimate reasons, and that businesses who are requiring old software are at best being negligent. I’ve worked with people that refuse to modernize and update and it’s always the same stupid thing: “cost”. We really do need strong regulations on businesses that negligently refuse keep reasonably up to date.
  - haolez 3y ago
    You are assuming that software always improves as its development moves forward. It might not. It might even get more insecure as time passes.
    - lallysingh 3y ago
      Or is always backwards compatible. Upgrades aren't free.
      - wredue 3y ago
        1) In general, older code tends to have security patches 2) it costs money to run a business, and saving money through gross negligence, leading to real people having real impacts through massive data leaks while suffering zero repercussions for that needs to end. “It costs money to keep software up to date” is a shit argument for why businesses should not be managing their software.

68 points by clessg 3y ago

Oh man I've needed this for a while to the point where I was about to build it my self so thanks! It would be great though if it also listed the tagged nixpkgs version in addition to the hash if applicable.

jljljl 3y ago

Hey! Author of this post here, happy to answer any questions about Nixhub!

nathan_phoenix 3y ago
First off, amazing tool and thanks for doing the nix community a favor! If you could expand more on the architecture of the tool I'd be very curious! Like what goes all on from parsing Hydra builds to getting a useful package versions DB and what were the challenges while building it? Also, how are you merging all the nodejs packages (nodejs_20, nodejs_18, ...) into one “nodejs”? I'm also in the process of building something very similar (didn't know the idea was that popular haha), but I'm not parsing Hydra builds but the output from `nix-env -qa --json` to get all the package versions and it's proving more involved than I anticipated (:
- jljljl 3y ago
  Thank you! We're planning a longer post that explains the architecture and process of building the index. An extremely basic description of what we do can be found here: https://www.jetpack.io/blog/0-5-0-install-nix-packages-by-version/#:~:text=Our%20Solution%3A%20a%20Devbox%20Package%20Index https://www.jetpack.io/blog/0-5-0-install-nix-packages-by-ve... First, we processed build outputs from Hydra to map package names + version numbers to commits in the Nixpkgs repo. We then created a service that stores this index as a prefix tree, making it easy to search for a specific package and version in Nix. There's some extra cleanup required to make the index really usable, which we'll share in the follow up post For the merged packages -- we have a hardcoded "canonical names", which map the various attribute paths like `nodejs_20` and `nodejs_18` to a single, easier to search name (like `nodejs`). We originally designed this so that Devbox users (who may not be familiar with Nix naming conventions) can just type `devbox add nodejs@20` and get the right package version.
  - nathan_phoenix 3y ago
    Thanks for the answer! Looking forward to the followup post :D And yeah, nix naming conventions are sometimes a mess. Your approach does seem much nicer. Last question if you have a bit more time. How come that you don't have every version of a package available? Is it because you only parse the Hydra stable 23 builds (which I guess aren't updated on every version) or because you parse them only periodically like Lazamar's Nix Package Versions?
    - jljljl 3y ago
      We're parsing periodically (though our index is more up to date than Nix Package Versions at the moment). We've also truncated pre-Flakes commits, though we are planning to include those in the future
      - nathan_phoenix 3y ago
        Looking forward to that for completeness sake! Btw, regarding a bit older pre-Flakes commits, at least some packages will error out while nix-installing on M1 (or newer) macbooks because at that point nix packages didn't anticipate macbooks ever ditching x86 for aarch... Hope this info helps you out, at least took me some time to realize that the random obscure error was caused by this. Can be almost always fixed by appending "--system darwin-x86_64" to the nix command to use Apple's Rosetta binary translation, cheers.

noahmasur 3y ago

Roughly equivalent tool that I've used in the past: https://lazamar.co.uk/nix-versions/ https://lazamar.co.uk/nix-versions/

stabbles 3y ago

It's interesting how Nix's selling point is that its non-standard file system structure allows multiple versions/flavors of the same package to be installed at the same time, yet if I understand correctly, it often has one version per package per Nix version? It's more like install multiple versions of the same package at a different time? Compare this to the model we have in Spack (which is inspired by nix), where there's one recipe per package for many versions, and you can define different build flavors, conditional dependencies (and conditional pretty much everything). The difference is that Spack has a dependency solver and Nix apparently does not? For example the curl package [1] has tons of versions, and its `tls` package variant has conditional values depending on the version: variant("tls", values=("gnutls", conditional("mbedtls", when="@7.46:"), ...), multi=True) And dependencies are specified as depends_on("gnutls", when="tls=gnutls") with when("tls=mbedtls"): depends_on("mbedtls@2:", when="@7.79:") depends_on("mbedtls@:2", when="@:7.78") This allows you both to install all sorts of curls, like `spack install curl tls=mbedtls` (gives you something >= 7.46), or `spack install curl@7.77 tls=mbedtls` gives you a curl with an old mbedtls, etc. [1] https://github.com/spack/spack/blob/develop/var/spack/repos/builtin/packages/curl/package.py https://github.com/spack/spack/blob/develop/var/spack/repos/...

_7tgr 3y ago
Probably because it's pretty easy to just use an older package from a different version of Nixpkgs (just use an override that defines the attribute for a package in your current Nixpkgs as the same package in an older Nixpkgs). It'll transparently pull the older version from the binary cache, too, if available. The only difficulty is finding the right Nixpkgs commit hash, which this solves. I think I find the Nix way to be a bit cleaner, though I'm unaware of the full set of pros/cons. Worth noting that the Nix package definition for curl at [1] is a little easier to read for me since it doesn't have the duplicated code for each version. [1] https://github.com/NixOS/nixpkgs/blob/master/pkgs/tools/networking/curl/default.nix https://github.com/NixOS/nixpkgs/blob/master/pkgs/tools/netw...
nathan_phoenix 3y ago
> yet if I understand correctly, it often has one version per package per Nix version? Both yeah and no. Taking the example of nodejs, you have all the supported major releases like nodejs_16, nodejs_18, etc. for each Nix revision[1], but within each revision you only have a specific version of the package. E.g. nodejs_16 is 16.3.1 for revision at date X. There are benefits and drawbacks of managing a package repo like that, with revisions which implicitely pin all packages and it's dependencies at once, but it's too long of a topic to write about here. > For example the curl package [1] has tons of versions [...] So does it in nix (curlMinimal, curlWithGnuTls, curlHTTP3, etc)[2]. In that regard nix and Spack are similar. [1]: A revision is like a shapshot of all the packages at a given point in time. [2]: https://search.nixos.org/packages?channel=23.05&from=0&size=50&sort=relevance&type=packages&query=curl https://search.nixos.org/packages?channel=23.05&from=0&size=...

hamandcheese 3y ago

For most people, depending on historic versions of nixpkgs is an anti-pattern. The main reason I say this is security. You may want an old version of, say, ruby, but you probably don't want the old, insecure libraries it was built against (i.e. glibc, openssl). It is often just as easy to use overrides to build a different version of a piece of software, and in the rare case that doesn't work you can fall back to a custom derivation (drawing inspiration from the old version of nixpkgs, perhaps). I worry about tools like this which make doing the wrong thing too easy.

wredue 3y ago
Yeah. People like to push that there are “legitimate reasons” to want old versions, but it usually comes down to businesses. I would suggest instead that there are no legitimate reasons, and that businesses who are requiring old software are at best being negligent. I’ve worked with people that refuse to modernize and update and it’s always the same stupid thing: “cost”. We really do need strong regulations on businesses that negligently refuse keep reasonably up to date.
- haolez 3y ago
  You are assuming that software always improves as its development moves forward. It might not. It might even get more insecure as time passes.
  - lallysingh 3y ago
    Or is always backwards compatible. Upgrades aren't free.
    - wredue 3y ago
      1) In general, older code tends to have security patches 2) it costs money to run a business, and saving money through gross negligence, leading to real people having real impacts through massive data leaks while suffering zero repercussions for that needs to end. “It costs money to keep software up to date” is a shit argument for why businesses should not be managing their software.