3 ms·

First off, amazing tool and thanks for doing the nix community a favor! If you could expand more on the architecture of the tool I'd be very curious! Like what

First off, amazing tool and thanks for doing the nix community a favor! If you could expand more on the architecture of the tool I'd be very curious! Like what goes all on from parsing Hydra builds to getting a useful package versions DB and what were the challenges while building it? Also, how are you merging all the nodejs packages (nodejs_20, nodejs_18, ...) into one “nodejs”? I'm also in the process of building something very similar (didn't know the idea was that popular haha), but I'm not parsing Hydra builds but the output from `nix-env -qa --json` to get all the package versions and it's proving more involved than I anticipated (:

jljljl 3y ago
Thank you! We're planning a longer post that explains the architecture and process of building the index. An extremely basic description of what we do can be found here: https://www.jetpack.io/blog/0-5-0-install-nix-packages-by-version/#:~:text=Our%20Solution%3A%20a%20Devbox%20Package%20Index https://www.jetpack.io/blog/0-5-0-install-nix-packages-by-ve... First, we processed build outputs from Hydra to map package names + version numbers to commits in the Nixpkgs repo. We then created a service that stores this index as a prefix tree, making it easy to search for a specific package and version in Nix. There's some extra cleanup required to make the index really usable, which we'll share in the follow up post For the merged packages -- we have a hardcoded "canonical names", which map the various attribute paths like `nodejs_20` and `nodejs_18` to a single, easier to search name (like `nodejs`). We originally designed this so that Devbox users (who may not be familiar with Nix naming conventions) can just type `devbox add nodejs@20` and get the right package version.
- nathan_phoenix 3y ago
  Thanks for the answer! Looking forward to the followup post :D And yeah, nix naming conventions are sometimes a mess. Your approach does seem much nicer. Last question if you have a bit more time. How come that you don't have every version of a package available? Is it because you only parse the Hydra stable 23 builds (which I guess aren't updated on every version) or because you parse them only periodically like Lazamar's Nix Package Versions?
  - jljljl 3y ago
    We're parsing periodically (though our index is more up to date than Nix Package Versions at the moment). We've also truncated pre-Flakes commits, though we are planning to include those in the future
    - nathan_phoenix 3y ago
      Looking forward to that for completeness sake! Btw, regarding a bit older pre-Flakes commits, at least some packages will error out while nix-installing on M1 (or newer) macbooks because at that point nix packages didn't anticipate macbooks ever ditching x86 for aarch... Hope this info helps you out, at least took me some time to realize that the random obscure error was caused by this. Can be almost always fixed by appending "--system darwin-x86_64" to the nix command to use Apple's Rosetta binary translation, cheers.

jljljl 3y ago

Thank you! We're planning a longer post that explains the architecture and process of building the index. An extremely basic description of what we do can be found here: https://www.jetpack.io/blog/0-5-0-install-nix-packages-by-version/#:~:text=Our%20Solution%3A%20a%20Devbox%20Package%20Index https://www.jetpack.io/blog/0-5-0-install-nix-packages-by-ve... First, we processed build outputs from Hydra to map package names + version numbers to commits in the Nixpkgs repo. We then created a service that stores this index as a prefix tree, making it easy to search for a specific package and version in Nix. There's some extra cleanup required to make the index really usable, which we'll share in the follow up post For the merged packages -- we have a hardcoded "canonical names", which map the various attribute paths like `nodejs_20` and `nodejs_18` to a single, easier to search name (like `nodejs`). We originally designed this so that Devbox users (who may not be familiar with Nix naming conventions) can just type `devbox add nodejs@20` and get the right package version.

nathan_phoenix 3y ago
Thanks for the answer! Looking forward to the followup post :D And yeah, nix naming conventions are sometimes a mess. Your approach does seem much nicer. Last question if you have a bit more time. How come that you don't have every version of a package available? Is it because you only parse the Hydra stable 23 builds (which I guess aren't updated on every version) or because you parse them only periodically like Lazamar's Nix Package Versions?
- jljljl 3y ago
  We're parsing periodically (though our index is more up to date than Nix Package Versions at the moment). We've also truncated pre-Flakes commits, though we are planning to include those in the future
  - nathan_phoenix 3y ago
    Looking forward to that for completeness sake! Btw, regarding a bit older pre-Flakes commits, at least some packages will error out while nix-installing on M1 (or newer) macbooks because at that point nix packages didn't anticipate macbooks ever ditching x86 for aarch... Hope this info helps you out, at least took me some time to realize that the random obscure error was caused by this. Can be almost always fixed by appending "--system darwin-x86_64" to the nix command to use Apple's Rosetta binary translation, cheers.