10 ms·

Rip – Rust crate to resolve and install Python packages

135 points by droelf 3y ago

ArchOversight 3y ago
How well does this work for projects that include dependencies that are still reliant on the old setup.py and don't have wheels published?
- droelf 3y ago
  It doesn't work well for those yet. rip only deals with wheel files for now. The way (afaik) pip works for source dist packages is that it locally builds the wheel, and then extracts the metadata from that locally built wheel. This is also the approach that we want to take with rip. As you can imagine, the performance will be quite bad (but nothing we can do about that). Ideally, almost all packages should ship wheels.
  - glandium 3y ago
    > all packages should ship wheels. And then a new python version is released and virtually no packages with native modules have wheels for that new python version. (too few use abi3) Also, good luck finding wheels for e.g. s390x linux.
  - jborean93 3y ago
    > Ideally, almost all packages should ship wheels. It would be ideal but it's not always possible. I maintain a GSSAPI/KRB5 library that wraps the C libs but due to PyPI wheel policies I cannot upload a wheel without embedding those C libs which then opens up a whole bunch of problems around deps and lib conflicts :(
  - stuaxo 3y ago
    Ideally - however there are points where it's not possible yet. Bindings like pycairo are a good example. On windows a WHL can be provided. On Linux it can't - Cairo (which pycairo binds to) is shipped by the distro, and distros are free to enable or disable different features. Since the C bindings are linked to the cairo shared library, and that includes all the backends you can't build that and know if would work in a distro. There are two use cases for pycairo - in one you might not care what the system provides and if the pycairo who provided its own Cairo shared object, that would be fine. In the second use case you want to use the system provides Cairo, e.g. to work with Gtk. It's impossible to resolve this workout changing how Cairo itself works and it's API. The result is no pycairo whl on Linux, and users who have to install all the dependencies (which once go to pango you pull in Gtk, X, freetype etc). Cairo isn't the only example but it's one I know. A lot of these are libraries that bind system libraries and predate virtualenv / venv - when they were created compiling things was fine.
- westurner 3y ago
  pypa/cibuildwheel: https://github.com/pypa/cibuildwheel https://github.com/pypa/cibuildwheel : > Example setup: To build manylinux, musllinux, macOS, and Windows wheels on GitHub Actions, you could use this .github/workflows/wheels.yml
  - ArchOversight 3y ago
    That's great and all for packages I control, but I publish wheels already.
    - westurner 3y ago
      https://pythonwheels.com/ https://pythonwheels.com/ now lists only three packages that are not yet published to PyPI as wheels. But do you or others build them with SLSA 3? https://slsa.dev/get-started#slsa-3 https://slsa.dev/get-started#slsa-3
      - ArchOversight 3y ago
        There's thousands more packages than listed on that website. Also now you are moving the goalposts. Python wheels exist but not all Python packages are distributed as wheels. If I am going through the effort to build wheels myself, then I don't need a resolver since I can just make sure to pick all the versions I need. Then I don't need rip to install anything.
westurner 3y ago
https://github.com/mamba-org/rattler https://github.com/mamba-org/rattler : > Rattler: Rust crates for fast handling of conda packages https://github.com/prefix-dev/pixi https://github.com/prefix-dev/pixi : > https://pixi.sh/ https://pixi.sh/
- deleted 3y ago
  [deleted]
- droelf 3y ago
  Yeah, rattler are our low-level crates to handle conda packages (similar to how rip is a low-level crate to handle PyPI packages). Both build on top of resolvo (our SAT solver, based on MiniSAT and libsolv) and both are used (or going to be used) in pixi, which is our package manager.
- ahurmazda 3y ago
  If I understand it correctly, rip::pip as to what rattler::conda? Finally pixi brings them altogether as the pkg manager to rule them all. Lofty goal but I remain optimistic that something better (esp in python land) is just around the corner.
  - droelf 3y ago
    yep, that's it!
droelf 3y ago
Blog post author here – happy to take any questions. We're looking to unify the conda + PyPI ecosystems and build a really nice cross-platform, language independent package manager (which is called pixi).
- singhrac 3y ago
  Any chance you guys, Charlie Marsh/Astral, and Armin Ronacher will ever team up? I’m really excited about next-gen tooling but would love a cargo-like “one true path”. At the very least keep the tools compatible?
  - droelf 3y ago
    Yeah, I think it would be awesome to work on top of the same foundations (rip & resolvo being two of those, and konstin's, who works with Charlie at astral, pep440_rs and pep508_rs being two others). We'll have some more conversations around this in the upcoming weeks I hope.
- Uptrenda 3y ago
  This is really impressive work tbh. Good job.
- out-of-ideas 3y ago
  as an arch user; i gotta say `aur/rip` is already in use; and i can only say good luck if you keep to a 3char name
  - dcow 3y ago
    Last updated: 2015. Sourceforge. Perhaps packages that are borderline irrelevant should not squat on prime name real estate. Plus cant rip ship as rip-py and still install as rip on the bet that nobody is actually going to be trying to use these tools simultaneously?
    - tubs 3y ago
      Agreed no one uses those defunct tools like awk or roff, let's re-use them! Who cares if every build script in the world breaks, progress waits for no one.
dimator 3y ago
Are there any gains to be had in just using this for pypi packages (no conda)? Solving for a monorepo with hundreds of packages is the cause of much time lost.
- droelf 3y ago
  Our solver implements "state-of-the-art" CDCL (conflict driven clause learning). So there is a chance that the performance is improved over "resolvelib" which is used by pip. Once we are feature complete enough we will work on some more benchmarking.
thatxliner 3y ago
I sure hope there’s a pnpm-style content addressable storage and space-saving capabilities
- droelf 3y ago
  Yeah, we're working on that and use a similar model like pnpm in pixi / rattler already. In fact, for a long time conda & mamba have created "hard-links" to a central package cache for the mentioned space savings.
d-k-bo 3y ago
> The above invocation will automatically download the latest version of pixi, extract it, and move the pixi binary to ~/.pixi/bin. If this directory does not already exist, the script will create it. Please don't pollute my home directory. Follow the XDG base directory specification [1]. [1]: https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html https://specifications.freedesktop.org/basedir-spec/basedir-...
- sideeffffect 3y ago
  There are even libraries for that! https://github.com/dirs-dev/directories-rs https://github.com/dirs-dev/directories-rs
Already__Taken 3y ago
Been quite happy with PDM last 6 months, that's pretty fast, how does this stack up?

135 points by droelf 3y ago

How well does this work for projects that include dependencies that are still reliant on the old setup.py and don't have wheels published?

droelf 3y ago
It doesn't work well for those yet. rip only deals with wheel files for now. The way (afaik) pip works for source dist packages is that it locally builds the wheel, and then extracts the metadata from that locally built wheel. This is also the approach that we want to take with rip. As you can imagine, the performance will be quite bad (but nothing we can do about that). Ideally, almost all packages should ship wheels.
- glandium 3y ago
  > all packages should ship wheels. And then a new python version is released and virtually no packages with native modules have wheels for that new python version. (too few use abi3) Also, good luck finding wheels for e.g. s390x linux.
- jborean93 3y ago
  > Ideally, almost all packages should ship wheels. It would be ideal but it's not always possible. I maintain a GSSAPI/KRB5 library that wraps the C libs but due to PyPI wheel policies I cannot upload a wheel without embedding those C libs which then opens up a whole bunch of problems around deps and lib conflicts :(
- stuaxo 3y ago
  Ideally - however there are points where it's not possible yet. Bindings like pycairo are a good example. On windows a WHL can be provided. On Linux it can't - Cairo (which pycairo binds to) is shipped by the distro, and distros are free to enable or disable different features. Since the C bindings are linked to the cairo shared library, and that includes all the backends you can't build that and know if would work in a distro. There are two use cases for pycairo - in one you might not care what the system provides and if the pycairo who provided its own Cairo shared object, that would be fine. In the second use case you want to use the system provides Cairo, e.g. to work with Gtk. It's impossible to resolve this workout changing how Cairo itself works and it's API. The result is no pycairo whl on Linux, and users who have to install all the dependencies (which once go to pango you pull in Gtk, X, freetype etc). Cairo isn't the only example but it's one I know. A lot of these are libraries that bind system libraries and predate virtualenv / venv - when they were created compiling things was fine.
westurner 3y ago
pypa/cibuildwheel: https://github.com/pypa/cibuildwheel https://github.com/pypa/cibuildwheel : > Example setup: To build manylinux, musllinux, macOS, and Windows wheels on GitHub Actions, you could use this .github/workflows/wheels.yml
- ArchOversight 3y ago
  That's great and all for packages I control, but I publish wheels already.
  - westurner 3y ago
    https://pythonwheels.com/ https://pythonwheels.com/ now lists only three packages that are not yet published to PyPI as wheels. But do you or others build them with SLSA 3? https://slsa.dev/get-started#slsa-3 https://slsa.dev/get-started#slsa-3
    - ArchOversight 3y ago
      There's thousands more packages than listed on that website. Also now you are moving the goalposts. Python wheels exist but not all Python packages are distributed as wheels. If I am going through the effort to build wheels myself, then I don't need a resolver since I can just make sure to pick all the versions I need. Then I don't need rip to install anything.

westurner 3y ago

https://github.com/mamba-org/rattler https://github.com/mamba-org/rattler : > Rattler: Rust crates for fast handling of conda packages https://github.com/prefix-dev/pixi https://github.com/prefix-dev/pixi : > https://pixi.sh/ https://pixi.sh/

deleted 3y ago
[deleted]
droelf 3y ago
Yeah, rattler are our low-level crates to handle conda packages (similar to how rip is a low-level crate to handle PyPI packages). Both build on top of resolvo (our SAT solver, based on MiniSAT and libsolv) and both are used (or going to be used) in pixi, which is our package manager.
ahurmazda 3y ago
If I understand it correctly, rip::pip as to what rattler::conda? Finally pixi brings them altogether as the pkg manager to rule them all. Lofty goal but I remain optimistic that something better (esp in python land) is just around the corner.
- droelf 3y ago
  yep, that's it!

droelf 3y ago

Blog post author here – happy to take any questions. We're looking to unify the conda + PyPI ecosystems and build a really nice cross-platform, language independent package manager (which is called pixi).

singhrac 3y ago
Any chance you guys, Charlie Marsh/Astral, and Armin Ronacher will ever team up? I’m really excited about next-gen tooling but would love a cargo-like “one true path”. At the very least keep the tools compatible?
- droelf 3y ago
  Yeah, I think it would be awesome to work on top of the same foundations (rip & resolvo being two of those, and konstin's, who works with Charlie at astral, pep440_rs and pep508_rs being two others). We'll have some more conversations around this in the upcoming weeks I hope.
Uptrenda 3y ago
This is really impressive work tbh. Good job.
out-of-ideas 3y ago
as an arch user; i gotta say `aur/rip` is already in use; and i can only say good luck if you keep to a 3char name
- dcow 3y ago
  Last updated: 2015. Sourceforge. Perhaps packages that are borderline irrelevant should not squat on prime name real estate. Plus cant rip ship as rip-py and still install as rip on the bet that nobody is actually going to be trying to use these tools simultaneously?
  - tubs 3y ago
    Agreed no one uses those defunct tools like awk or roff, let's re-use them! Who cares if every build script in the world breaks, progress waits for no one.

dimator 3y ago

Are there any gains to be had in just using this for pypi packages (no conda)? Solving for a monorepo with hundreds of packages is the cause of much time lost.

droelf 3y ago
Our solver implements "state-of-the-art" CDCL (conflict driven clause learning). So there is a chance that the performance is improved over "resolvelib" which is used by pip. Once we are feature complete enough we will work on some more benchmarking.

thatxliner 3y ago

I sure hope there’s a pnpm-style content addressable storage and space-saving capabilities

droelf 3y ago
Yeah, we're working on that and use a similar model like pnpm in pixi / rattler already. In fact, for a long time conda & mamba have created "hard-links" to a central package cache for the mentioned space savings.

d-k-bo 3y ago

> The above invocation will automatically download the latest version of pixi, extract it, and move the pixi binary to ~/.pixi/bin. If this directory does not already exist, the script will create it. Please don't pollute my home directory. Follow the XDG base directory specification [1]. [1]: https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html https://specifications.freedesktop.org/basedir-spec/basedir-...

sideeffffect 3y ago
There are even libraries for that! https://github.com/dirs-dev/directories-rs https://github.com/dirs-dev/directories-rs

Already__Taken 3y ago

Been quite happy with PDM last 6 months, that's pretty fast, how does this stack up?