5 ms·

The birth of a package manager

65 points by wofo 4y ago

wofo 4y ago
OP here, happy to see this reached the frontpage! For those like me who always look at the comments to decide if the article is worth their time: this is about an open source package manager written in Rust, which I had the pleasure to work on during my last contract. There were a bunch of interesting technical challenges (mostly around performance), which required some creativity to solve. If you like that kind of stuff, you might enjoy reading the article ;)
_frkl 4y ago
Thanks for sharing, very cool! There doesn't seem to be a whole lot of documentation on how to use it as a library, but looking at the code all seems to be nice and tidy, and it doesn't look too difficult to adapt to a use-case or two I have in mind. Any plans to publish a release version to crates.io, or is it mainly intended as an internal tool for prefix.dev?
- wofo 4y ago
  As far as I know the plan is indeed to publish a version to crates.io, and the only reason it hasn't been done yet is that it is still early days. You might want to jump in their discord server to ask, though, if you want to be sure (or maybe you can even create an issue on GitHub)
  - baszalmstra 4y ago
    Rattler author here! I just published v0.1.0 to crates.
    - _frkl 4y ago
      Yay! :) Thanks heaps! It's funny, 'rattler' is exactly what I need at the moment. I was about to write a rust app that downloads 'micromamba' and runs it as a subprocess. But this is so much better for what I try to do, being able to include everything in a single binary and calling rust code directly...
      - baszalmstra 4y ago
        Great to hear! You peaked my interest now! I would love to know more about your usecase! Would you be open to join our discord to talk about it?
        _frkl 4y ago
        Sure, will do!
pxc 4y ago
The caching trick for libsolv here could be a game changer for performance in a lot of Linux distro package managers, where libsolv's completeness comes at a substantial performance cost. I've never looked too deeply into how conda works because I'm not that into Python, but I guess I should. It seems there are some valuable general lessons here— I wish that more package managers, and in particular, Nix, maintained more version combinations in their repos. Searching through Nixpkgs commits to find compatible versions of things because you have to rely on the implicit versioning of the monorepo sucks. Adding dependency resolution to Nix as it stands would normally add a big performance cost. Something like this would help with that problem a lot, and Nix already usually runs with a persistent daemon. Maybe a similar technique could fit in well here.
- droelf 4y ago
  Yes, we definitely believe and rely a lot on proper "SAT" solving in the conda ecosystem (old versions are also never deleted, for reproducibility reasons, which makes this problem actually quite interesting and large). We are also quite curious to learn more about the Nix approach. Building packages is still a bit too painful / annoying in the conda ecosystem, which is something we really want to fix, and I think we can learn a lesson or two in the Nix ecosystem. And by the way, the conda ecosystem is not at all Python specific. It's a common misconception, but they are all general binary packages and can be used for Ruby, R, Rust, C++, Zig, ... whatever – just like Nix. Rattler, mamba and conda all handle them the same nice way.
  - pxc 4y ago
    I knew that conda's connection to Python was mostly historical rather than technological, since it handled native deps and so it had to be able to work with C as well. I didn't realize so much work had been done packaging all of those other language ecosystems, though!
- chriswarbo 4y ago
  I'm not sure how 'dependency resolution' would work in a system like Nix; since Nix has no notion of "package", "dependency", "version", etc. > Searching through Nixpkgs commits to find compatible versions of things because you have to rely on the implicit versioning of the monorepo sucks. I get what you're saying, but I gave up on thinking of Nix/Nixpkgs in terms of "package management". It's much closer to a build system, like GNU Make; rather than, say, APT/Yum. When it comes to versions of things in Nixpkgs: most things are defined as a function of other things; i.e. the arguments to that function are (roughly) its "dependencies". If we want some different version, we can call those functions with different arguments (or use the various `override` helpers included in their return values). Once we've overridden the things we care about, we can usually take the fixed-point/closure of that package set (whether it's the whole of Nixpkgs, or just e.g. python3Packages, or whatever); that way, our changes get propagated through all of the function calls to ensure everything's referencing our versions. Admittedly, this can be quite tedious!
epage 4y ago
Was glad to see this blurb > One interesting avenue of future work is to try to replace libsolv by a solver written in Rust, such as PubGrub. That way we could get rid of a bunch of unsafe code we are using to interface with libsolv through Rust’s FFI. For people who would be interested in learning more, check out this conference presentation: https://youtu.be/XyJpYFnl18U https://youtu.be/XyJpYFnl18U Using pubgrub in cargo is something we talk about on a regularly basis. Its our equivalent of rustc chalk or polonius: we keep limping along, always finding a new hack to extend the life of the current package resolver due to the large lift to change out the resolver but also knowing how much the current one slows us down and holds us back. I'm excited for any forward movement on pubgrub in any related project.
- droelf 4y ago
  Another rattler author here! Indeed, pubgrub-rs looks like one hell of a good alternative to libsolv. We've already tried quite a bit to use it in rattler, but the many "variant" packages in the conda ecosystem gave it a bit of a hard time. We still think that we can make it work. And thanks for linking a PackagingCon talk – I am actually the an organizer of that conference and happy to report that it's coming back this year in October (https://packaging-con.org https://packaging-con.org has the details).
  - epage 4y ago
    Oo, don't want to miss it next year! Do you have a mastadon account or an email list? I'm trying to keep to my "no twitter" streak.
    - droelf 4y ago
      I just made a mastodon account for PackagingCon on Fosstodon, will update you here once it's approved.
- munificent 4y ago
  If you prefer text over video, here's Natalie's original article about PubGrub: https://nex3.medium.com/pubgrub-2fb6470504f https://nex3.medium.com/pubgrub-2fb6470504f She invented this for Dart's package manager (pub), which is where the name comes from.
calvinmorrison 4y ago
I thought I'd get started on Rust last night with a sort of simple getopt's CLI tool. I immediately ran into some whacky dependency/library issue. Google revealed an open bug report. I put rust down and maybe I'm stupid but, I was following a 'get started with rust' type of tutorial. I think I will not look again for some years.
- brabel 4y ago
  That's surprising, the Rust ecosystem is already getting quite mature for most things. Can you share the bug you ran into?
  - calvinmorrison 4y ago
    something about linking to rustc https://github.com/rust-lang/rust/issues/27812 https://github.com/rust-lang/rust/issues/27812 i didn't look any futher.
ris 4y ago
Ah yes if there's one thing the world needs it's another package manager to fight with the other seventeen on your system.

65 points by wofo 4y ago

wofo 4y ago

OP here, happy to see this reached the frontpage! For those like me who always look at the comments to decide if the article is worth their time: this is about an open source package manager written in Rust, which I had the pleasure to work on during my last contract. There were a bunch of interesting technical challenges (mostly around performance), which required some creativity to solve. If you like that kind of stuff, you might enjoy reading the article ;)

_frkl 4y ago

Thanks for sharing, very cool! There doesn't seem to be a whole lot of documentation on how to use it as a library, but looking at the code all seems to be nice and tidy, and it doesn't look too difficult to adapt to a use-case or two I have in mind. Any plans to publish a release version to crates.io, or is it mainly intended as an internal tool for prefix.dev?

wofo 4y ago
As far as I know the plan is indeed to publish a version to crates.io, and the only reason it hasn't been done yet is that it is still early days. You might want to jump in their discord server to ask, though, if you want to be sure (or maybe you can even create an issue on GitHub)
- baszalmstra 4y ago
  Rattler author here! I just published v0.1.0 to crates.
  - _frkl 4y ago
    Yay! :) Thanks heaps! It's funny, 'rattler' is exactly what I need at the moment. I was about to write a rust app that downloads 'micromamba' and runs it as a subprocess. But this is so much better for what I try to do, being able to include everything in a single binary and calling rust code directly...
    - baszalmstra 4y ago
      Great to hear! You peaked my interest now! I would love to know more about your usecase! Would you be open to join our discord to talk about it?
      - _frkl 4y ago
        Sure, will do!

pxc 4y ago

The caching trick for libsolv here could be a game changer for performance in a lot of Linux distro package managers, where libsolv's completeness comes at a substantial performance cost. I've never looked too deeply into how conda works because I'm not that into Python, but I guess I should. It seems there are some valuable general lessons here— I wish that more package managers, and in particular, Nix, maintained more version combinations in their repos. Searching through Nixpkgs commits to find compatible versions of things because you have to rely on the implicit versioning of the monorepo sucks. Adding dependency resolution to Nix as it stands would normally add a big performance cost. Something like this would help with that problem a lot, and Nix already usually runs with a persistent daemon. Maybe a similar technique could fit in well here.

droelf 4y ago
Yes, we definitely believe and rely a lot on proper "SAT" solving in the conda ecosystem (old versions are also never deleted, for reproducibility reasons, which makes this problem actually quite interesting and large). We are also quite curious to learn more about the Nix approach. Building packages is still a bit too painful / annoying in the conda ecosystem, which is something we really want to fix, and I think we can learn a lesson or two in the Nix ecosystem. And by the way, the conda ecosystem is not at all Python specific. It's a common misconception, but they are all general binary packages and can be used for Ruby, R, Rust, C++, Zig, ... whatever – just like Nix. Rattler, mamba and conda all handle them the same nice way.
- pxc 4y ago
  I knew that conda's connection to Python was mostly historical rather than technological, since it handled native deps and so it had to be able to work with C as well. I didn't realize so much work had been done packaging all of those other language ecosystems, though!
chriswarbo 4y ago
I'm not sure how 'dependency resolution' would work in a system like Nix; since Nix has no notion of "package", "dependency", "version", etc. > Searching through Nixpkgs commits to find compatible versions of things because you have to rely on the implicit versioning of the monorepo sucks. I get what you're saying, but I gave up on thinking of Nix/Nixpkgs in terms of "package management". It's much closer to a build system, like GNU Make; rather than, say, APT/Yum. When it comes to versions of things in Nixpkgs: most things are defined as a function of other things; i.e. the arguments to that function are (roughly) its "dependencies". If we want some different version, we can call those functions with different arguments (or use the various `override` helpers included in their return values). Once we've overridden the things we care about, we can usually take the fixed-point/closure of that package set (whether it's the whole of Nixpkgs, or just e.g. python3Packages, or whatever); that way, our changes get propagated through all of the function calls to ensure everything's referencing our versions. Admittedly, this can be quite tedious!

epage 4y ago

Was glad to see this blurb > One interesting avenue of future work is to try to replace libsolv by a solver written in Rust, such as PubGrub. That way we could get rid of a bunch of unsafe code we are using to interface with libsolv through Rust’s FFI. For people who would be interested in learning more, check out this conference presentation: https://youtu.be/XyJpYFnl18U https://youtu.be/XyJpYFnl18U Using pubgrub in cargo is something we talk about on a regularly basis. Its our equivalent of rustc chalk or polonius: we keep limping along, always finding a new hack to extend the life of the current package resolver due to the large lift to change out the resolver but also knowing how much the current one slows us down and holds us back. I'm excited for any forward movement on pubgrub in any related project.

droelf 4y ago
Another rattler author here! Indeed, pubgrub-rs looks like one hell of a good alternative to libsolv. We've already tried quite a bit to use it in rattler, but the many "variant" packages in the conda ecosystem gave it a bit of a hard time. We still think that we can make it work. And thanks for linking a PackagingCon talk – I am actually the an organizer of that conference and happy to report that it's coming back this year in October (https://packaging-con.org https://packaging-con.org has the details).
- epage 4y ago
  Oo, don't want to miss it next year! Do you have a mastadon account or an email list? I'm trying to keep to my "no twitter" streak.
  - droelf 4y ago
    I just made a mastodon account for PackagingCon on Fosstodon, will update you here once it's approved.
munificent 4y ago
If you prefer text over video, here's Natalie's original article about PubGrub: https://nex3.medium.com/pubgrub-2fb6470504f https://nex3.medium.com/pubgrub-2fb6470504f She invented this for Dart's package manager (pub), which is where the name comes from.

calvinmorrison 4y ago

I thought I'd get started on Rust last night with a sort of simple getopt's CLI tool. I immediately ran into some whacky dependency/library issue. Google revealed an open bug report. I put rust down and maybe I'm stupid but, I was following a 'get started with rust' type of tutorial. I think I will not look again for some years.

brabel 4y ago
That's surprising, the Rust ecosystem is already getting quite mature for most things. Can you share the bug you ran into?
- calvinmorrison 4y ago
  something about linking to rustc https://github.com/rust-lang/rust/issues/27812 https://github.com/rust-lang/rust/issues/27812 i didn't look any futher.

ris 4y ago

Ah yes if there's one thing the world needs it's another package manager to fight with the other seventeen on your system.