13 ms·

Uv: Python packaging in Rust

647 points by samwho 3y ago

0cf8612b2e1e 3y ago
Coupling with Rye is very interesting, as that or the PyBi idea seem inevitable to take over. Managing different interpreters is just too much effort which each tool is handling differently. Of course, the PSF will remain silent on the issue and let yet another tool exist in the ecosystem. I really do not care who wins, but we need a standard to be decided.
- OJFord 3y ago
  I hadn't heard of Rye before I don't think, but sounds like a strange arrangement to me - basically the same project & idea, so the new one to start switched the old one to use it 'under the hood', 'took it over', and will replace it 'when the time is right'? Why.. any of that, why not just buy it and let it be uv instead of starting uv, or just leave it alone?
  - the_mitsuhiko 3y ago
    I wrote down a small little FAQ at the bottom, hope it answers some questions: https://lucumr.pocoo.org/2024/2/15/rye-grows-with-uv/ https://lucumr.pocoo.org/2024/2/15/rye-grows-with-uv/
    - OJFord 3y ago
      That is helpful, and fair enough, thanks!
  - deleted 3y ago
    [deleted]
- theyinwhy 3y ago
  Why do people here claim that the PSF has any leverage to stop tools being developed?
  - 0cf8612b2e1e 3y ago
    I don’t want the PSF to declare tools cannot be made. I want the PSF to pick a winner so we can finally consolidate on a workflow. Just today, I was helping a beginner setup Python on their laptop. A nightmare collection of tooling complexity (so you have Python, but you need to worry about virtual environments, to do that, I use this combination of pipx+virtualenvwrapper, but to do actual projects, you need Poetry, but…) all the while noting that this is an opinionated workflow that I happen to utilize because of the tradeoffs that matter to me. Ask someone else, and they will definitely have a different configuration. Entire first lesson was just to get Python installed with tons of elided complexity. I should be able to point to a document on Python.org that says, “This is the way.” Yet for decades, they have refused to take a stand.
    - theyinwhy 3y ago
      Isn't it your job as an engineer to pick the right toolchain for the job?
ericra 3y ago
While I don't see the immediate need to switch from using pip, I am such a fan of ruff that I'm always happy to check out what you guys come out with. Please keep doing what you're doing!
drcongo 3y ago
I've been dying to see what they came up with next given how in love I am with Ruff. In my wildest dreams I didn't expect it to be a package resolver. I'm very excited about this. edit: As a side note, if anyone from Astral happens by, any chance of an RSS feed on your blog?
- singhrac 3y ago
  I was kind of hoping it was a better type checker, something like mypy + pyright. To be fair that's an incredible amount of work, so maybe we'll see that later. Still very excited.
  - drcongo 3y ago
    That was actually my first guess, and while I certainly would love to see that, packaging is such a headache for so many people that this could be amazing. Even with poetry I still occasionally get mysterious errors and wildly wrong resolver claims about versions not existing.
    - catlover76 3y ago
      [dead]
  - fydorm 3y ago
    Why include mypy at all? Pyright is amazing and all you need.
marmaduke 3y ago
For envs which are pets, who cares, but this will be great for those cattle situations.
SushiHippie 3y ago
Isn't this basically what pixi wants to be? Wouldn't it be better to work together? https://github.com/prefix-dev/pixi/ https://github.com/prefix-dev/pixi/
- theLiminator 3y ago
  Pixi is even more ambitious but with a different philosophy (I think? Specifically thinking about how pixi went with their own lock and manifest format as well as first class conda support). I'd definitely prefer if they worked together instead or instead dedicated their time to type checking python which imo there still isn't a great solution for.
  - SushiHippie 3y ago
    Especially the conda support is IMO a cool thing. As conda/pip interop is just not great. And even micromamba (C++ implementation of conda) is relatively slow to resolve compared to pip. But agreed either work together or create a type checker. I use mypy currently but it definitely slows down my editor.
    - theLiminator 3y ago
      Yeah, pixi has decent mixed support for conda/pypi, it currently solves and installs conda (then locks it) and then solves and installs pypi. I think it's on their roadmap to solve them together, which would be a killer feature.
    - davidhalter 3y ago
      I have been working on a faster type checker for 3.5 years now. It's coming, I promise :)
  - fydorm 3y ago
    Have you tried Pyright? It made me actually enjoy Python development.
    - 3y ago
epage 3y ago
Congrats! > Similarly, uv does not yet generate a platform-agnostic lockfile. This matches pip-tools, but differs from Poetry and PDM, making uv a better fit for projects built around the pip and pip-tools workflows. Do you expect to make the higher level workflow independent of requirements.txt / support a platform-agnostic lockfile? Being attached to Rye makes me think "no". Without being platform agnostic, to me this is dead-on-arrival and unable to meet the "Cargo for Python" aim. > uv supports alternate resolution strategies. By default, uv follows the standard Python dependency resolution strategy of preferring the latest compatible version of each package. But by passing --resolution=lowest, library authors can test their packages against the lowest-compatible version of their dependencies. (This is similar to Go's Minimal version selection.) > uv allows for resolutions against arbitrary target Python versions. While pip and pip-tools always resolve against the currently-installed Python version (generating, e.g., a Python 3.12-compatible resolution when running under Python 3.12), uv accepts a --python-version parameter, enabling you to generate, e.g., Python 3.7-compatible resolutions even when running under newer versions. This is great to see though! I can understand it being a flag on these lower level, directly invoked dependency resolution operations. While you aren't onto the higher level operations yet, I think it'd be useful to see if there is any cross-ecosystem learning we can do for my MSRV RFC: https://github.com/rust-lang/rfcs/pull/3537 https://github.com/rust-lang/rfcs/pull/3537 How are you handling pre-releases in you resolution? Unsure how much of that is specified in PEPs. Its something that Cargo is weak in today but we're slowly improving.
- OJFord 3y ago
  That is great. Fuzzing would be cool too - just completely randomise the versions within the claimed compatibility constraints.
- charliermarsh 3y ago
  Thanks Ed! Your work as always is a big source of inspiration. > Do you expect to make the higher level workflow independent of requirements.txt / support a platform-agnostic lockfile? Being attached to Rye makes me think "no". Yes, we absolutely do. We don't do this today, the initial scope is intentionally limited. But in the next phase of the project, we want to extend to multi-platform and multi-version resolution. > How are you handling pre-releases in you resolution? Unsure how much of that is specified in PEPs. Its something that Cargo is weak in today but we're slowly improving. This is something we talked with Jacob about quite a bit. Turns out (as you know) it's a very hard problem. For the initial release, we added a constraint: our default behavior is that if you want to use a pre-release, you _have_ to specify the package as a first-party dependency and use a pre-release marker in the version specifier. (We also support globally enabling and disabling pre-releases.) So, we basically don't support "transitive" pre-releases right now -- but we give you a dedicated error message if your resolution fails for that reason.
notatallshaw 3y ago
There are a couple of promising tools written in Rust looking to replace Pip for most users. Rip (https://github.com/prefix-dev/rip/issues https://github.com/prefix-dev/rip/issues) which is more of a library for other rust tools to be on top of like Pixi (which is looking to replace both Pip and Conda). And now uv, which seems to be looking to replace Pip, Pip-Tools, and eventually Poetry and PDM. A lot of the explosion in tools in the Python world is coming from the desire for better workflows. But it has been enabled by the fact that build configuration and calling has been standardized and tool makers are able to follow standards instead of reverse engineering easy install or setup tools. I know a lot of people are put off by there being so many tools, but I think in a few years the dust will settle and there will emerge a best practice work flow that most users can follow. As a primarily Python developer and someone who occasionally contributes to Pip to solve complex dependency resolution issues it does make me wonder if I should hang my hat on that and learn enough rust to contribute to one of these projects eventually.
- ShamelessC 3y ago
  My experience with Rust developers who dwell in Python land is that they fundamentally disagree with most of the language that is Python and think they know better than incumbents what belongs and what doesn't.
  - drawnwren 3y ago
    To be fair, python package management is in such a poor state that I would expect any outside opinion to not be worse.
  - klardotsh 3y ago
    The Rust ecosystem gets so much right that honestly, even as a career-long Python developer myself (and Rust for many years, but that's less of my point), they honestly probably do know how to build a good devxp better than much of the Python ecosystem. Put other ways: the Python ecosystem has had 30+ years to figure out how to make packaging not suck. It has continually failed - failed less and less over time, sure, but the story is still generally speaking a nightmare ("throw it all in an OCI container" is an extremely reasonable solution to Python packaging, still, in 2024). I welcome advances, especially those inspired by tooling from languages that focused heavily on developer experience.
eigenvalue 3y ago
Looks awesome. I find that pip is usually pretty fast for me, and when it's not, it is mostly because it has to download so much data or wait for native libraries to compile in the background (or anything involving cuda which always seems to take forever). What really needs some help with speed is conda, which is just so absurdly slow for literally anything, even on ridiculously powerful machines.
- paulddraper 3y ago
  Yeah, I'm curious how much uv is actually faster. npm -> Yarn was life changing performance-wise. I wonder what pip -> uv is.
  - the_mitsuhiko 3y ago
    From my testing in rye it’s significantly faster in day to day. There are numbers in the blog post obviously but it’s the feeling you get using it locally that makes it much more fun to use.
  - laborcontract 3y ago
    Bun, ruff/uv, polars.. all have been major qol improvements. I’m loving the pace of release with this crop of speed obsessed projects and I cannot wait for astral to tackle typing.
    - laborcontract 3y ago
      Ok, here are some benchmarks installing an application that has a couple build-heavy dependencies: Installation times (MM:SS): # No cache - uv: 01:05 - pip: 01:56 # Cache: - uv: 00:02 - pip: 00:42
  - eigenvalue 3y ago
    npm seems to have gotten a lot faster lately. All that competition from yarn and now bun seems to have pushed them to focus on optimization.
rengler33 3y ago
This is great. Just adopted the pip-tools approach on a project and that has been great. Excited to give this a try.
maxvt 3y ago
I'd be curious to see how "packaging for production" is addressed. As described, this is a tool for development purposes: > Think: a single binary that bootstraps your Python installation and gives you everything you need to be productive with Python, bundling not only pip, pip-tools, and virtualenv, but also pipx, tox, poetry, pyenv, ruff, and more. A lot of that dev tooling is not needed in prod, in fact it is a liability (features available for misuse, code/image size, higher resource requirements). Would there be a "uv-prod" to only deal with the minimum subset of environment setup / container building that enables production use? Would uv build packages that are runnable in prod but themselves don't contain uv? It'd be interesting to hear the plans.
- radus 3y ago
  My approach for this is to use multi-stage dockerfiles. Build tools live and operate in a build stage, and you copy the /venv into your runtime stage. I think this same approach should work with uv so long as it doesn't bundle itself into the venv.
duped 3y ago
Do you plan to support platform-agnostic lockfiles? From the wording it sounds like "yes" but also "this is a feature not a bug." Which is it? Checksum verification and portable lockfiles are kind of table stakes for me, which is why I use poetry everywhere possible. I can't give up correctness for speed. sidenote: this would be super useful as a library so I could plug it into build systems. Managing python dependencies in standards-compliant ways is a pain, so shelling out to tools like poetry and pip are used even though it would be nicer to write that code eg in Rust and just use the parts you care about.
- charliermarsh 3y ago
  Yes, we absolutely want to support platform-agnostic lockfiles. Very important thing. The "feature not a bug" tone is probably because we intentionally limited the scope of this release to _not_ support platform-agnostic resolution, since it adds a lot of complexity (and we were still able to ship something that's immediately useful for those that rely on pip and pip-tools today).
captn3m0 3y ago
So what's the plan for monetization? NPM getting acquired by GitHub was an anomaly, and I'm wary of letting VC-backed companies becoming ecosystem lynchpins.
- sophacles 3y ago
  It's MIT and Apache dual licensed. If the company fails, you fork and move on. If the company never makes a dime, that's OK for me: some folks got paid to make open source tooling.
  - anze3db 3y ago
    What they are doing is already discouraging people from contributing to the projects their tools are replacing[0]. If they go out of business and stop supporting their tools, it might leave the Python community in a bad place. [0] https://www.youtube.com/watch?v=XzW4-KEB664 https://www.youtube.com/watch?v=XzW4-KEB664
    - woodruffw 3y ago
      I'm not sure this is a reasonable framing: LLVM stole much of GCC's thunder by being significantly easier to contribute to (and extend externally), but I don't think it would be accurate to say that LLVM is "discouraging" people from contributing to GCC.
      - ForHackernews 3y ago
        The slow death of GCC has been unquestionably bad for copyleft software.
        woodruffw 3y ago
        Why is it LLVM's job to be good for copyleft software?
        3y ago
alfalfasprout 3y ago
While we don't particularly care about pip lockfiles, etc. as we're moving to the conda/mamba ecosystem a faster pip is most welcome. We invoke it a lot programmatically and it's very, very slow.
iwebdevfromhome 3y ago
Very nice! Not long ago we got rid of a bunch of tools at work in favor of ruff. I'm looking forward to this. Specially when the time comes to be able to replace poetry with it, hoping it is a good implementation.
murkt 3y ago
An ability to override dependencies, finally!
amelius 3y ago
Iirc, some pip packages require compilation which depends on entire toolchains with e.g. gcc, g++, and with dependencies like gtk, Qt, etc. How do they intend to make that less error prone and thus more user-friendly?
- cavallo71 3y ago
  That's exactly what conda/mamba/micromamba is covering: the whole toolchain.
  - amelius 3y ago
    I've used conda/mamba, but often ended up with broken installs (random coredumps) that were mysteriously fixed by just using pip3 instead.
  - ZeroCool2u 3y ago
    That's what they attempt to cover, but they're not super successful at it.
- mixmastamyk 3y ago
  Wheels have been a thing for a decade+. If not available from the developer the complexity doesn't reduce much.
  - wiredfool 3y ago
    That’s the really pointy end of packaging. Pillow (python +c+ External dlls) produces something like 50 wheels per release, and we’ve still got platforms where you’ve got to compile your own. Since they tend to be lower volume, they also tend to have more interesting compiler issues.
    - amelius 3y ago
      Yes, I'm using Python on an nVidia Jetson system with PyTorch, which depends on CUDA, and all my other dependencies are quite challenging to get working in concert. Every time pip3 starts compiling under the hood, I'm just praying that it will work.
ofek 3y ago
Exciting stuff! I view Hatch [1] as becoming the Cargo for Python because it's already close and has an existing (and growing) user base but I can definitely see depending on this for resolution and potentially not even using pip after it becomes more stable. [1]: https://hatch.pypa.io/latest/ https://hatch.pypa.io/latest/
woodruffw 3y ago
This is very exciting! Congratulations to the Astral team on their work here. I have historically expressed trepidation towards external attempts to "fix" Python packaging, largely because each attempt has historically accreted another layer of incompatibilities and hacks onto the giant pile of hacks that are already needed to keep the ecosystem running. As such, it makes me happy to see that compatibility is a priority here: the Astral team has gone out of their way to emphasize both formal (PEP) and informal (pip CLI) compatibility with existing tooling and standards. (Separately, I share concerns about VC funding and sustainability. But everything I've seen so far indicates that Charlie and the other folks at Astral are not in it to screw or otherwise leverage the Python community. So I choose to be optimistic here.)
- eviks 3y ago
  Prioritizing compatibility with a giant incompatible pile of historical hacks is how you grow the pile
  - woodruffw 3y ago
    I don't think that's true, at least in the case of Python packaging. The pile has historically grown because of informal or non-existent standards, along with multiple independent implementations attempting to align their observed (but not specified) behaviors.
  - fwip 3y ago
    Well, they do explicitly list certain historical features that they don't intend to support, like installation from eggs or editable installs. So they're doing some work to trim the pile while they're there.
    - SonOfLilit 3y ago
      They do support editable installs, but only from local directories. So you have to perform git clone yourself and then install.
SamEdwardes 3y ago
Sign me up! A single binary to manage Python is something I have been hoping for.
chrisshroba 3y ago
Obligatory two xkcd's, but I'm super excited about this project! Definitely going to try it out. https://xkcd.com/1987/ https://xkcd.com/1987/ https://xkcd.com/927/ https://xkcd.com/927/
nindalf 3y ago
Feel like I called this 11 days ago - https://news.ycombinator.com/item?id=39251014 https://news.ycombinator.com/item?id=39251014 > I had to guess, that’s the path that the Astral team would take as well - expand ruff’s capabilities so it can do everything a Python developer needs. So the vision that Armin is describing here might be achieved by ruff eventually. They’d have an advantage that they’re not a single person maintenance team, but the disadvantage of needing to show a return to their investors.
12_throw_away 3y ago
This article says they don't support "platform-agnostic lockfiles" YET. And platform-agnostic lockfiles have been mentioned in several comments here, times too. I haven't encountered these before (with python, anyway) - how do they work platform-specific dependencies, e.g., binary wheels? For instance, there are >30 dists for numpy 1.26.4 [1], with different artifacts for different platforms and python interpreters. Which of those hashes ends up in a platform-agnostic lockfile? Do you have to stick with sdists only? [1] https://pypi.org/project/numpy/#files https://pypi.org/project/numpy/#files
- orf 3y ago
  All of them - poetry adds them as an array, and the right one is installed for the platform.
- jvolkman 3y ago
  PDM and Poetry would include hashes for all possible wheels, and would also follow transitive paths for platforms other than the current host. E.g. if my host is Linux and I depend on foo, and foo depends on bar only on windows, and bar depends on baz, my lock file will still contain baz.
gary_0 3y ago
Something I've been waiting to see from language package managers is more attention paid to security. I believe both cargo and pip packages can run arbitrary code as the local user the instant they are installed, and malicious packages have existed in the wild for years. I also recall a blog post where someone was scanning PyPI for malicious Python packages, only to realize that `pip download` also executed code. Just downloading a library to a local source code directory should not cause arbitrary code to run on your system, and there should be safeguards in place so developers are not one typo away from a malicious package pwning their entire system. Supply chain attacks remain a huge area of concern. Instead, when I "ctrl+f security" the homepages of any of these new packaging systems, I get 0 results. Not good.
- woodruffw 3y ago
  > I also recall a blog post where someone was scanning PyPI for malicious Python packages, only to realize that `pip download` also executed code. I think you're thinking of this post[1]. The code being searched for wasn't malicious, just buggy :-) [1]: https://moyix.blogspot.com/2022/09/someones-been-messing-with-my-subnormals.html https://moyix.blogspot.com/2022/09/someones-been-messing-wit...
  - gary_0 3y ago
    Thanks, that's the one! I was actually just trying to find it. Heh, I forgot about the anime catgirl part.
the__alchemist 3y ago
Very cool! Of note, I made something along these lines a few years ago, although with a slightly broader scope to also include managing and installing python versions. I abandoned it due to lack of free time, and edge cases breaking things. The major challenge is that Python packages that aren't wheels can do surprising things due to setup.py running arbitrary code. (https://github.com/David-OConnor/pyflow https://github.com/David-OConnor/pyflow) For some more context, at the time, poetry and pipenv were both buggy, and had some limitations their teams listed as "wont-fix". The sort of thing where you hit a breaking bug immediately on certain systems. These have since been fixed.
singhrac 3y ago
Something that is super super important to me is editable installs, and local references to those editable installs. Often I have several editable packages set up in a virtualenv which I think of as like my "base" virtualenv. One reason I struggled with Rye earlier is that it kind of felt like this wasn't entirely supported, possibly because I couldn't parse the information on the workspaces or virtual projects page. Maybe someone else figured this out? It does seem common enough that I would be surprised if it wasn't supported.
- charliermarsh 3y ago
  I think having a good story for local workspaces is gonna be a key piece of this next phase of the project, where we grow uv from a "pip replacement" to a "Poetry replacement" -- or, phrased differently, to something that looks more like Cargo. (Cargo does a bunch of nice things here that I want to learn from.)
  - singhrac 3y ago
    I think Rye actually does handle this mostly correctly (as the sibling comment said). I got through some of it here: https://github.com/mitsuhiko/rye/issues/671 https://github.com/mitsuhiko/rye/issues/671. I think it's very close to what I actually want (maybe not what Armin wants with multiversion).
- Zizizizz 3y ago
  This does editable installs, I tried it myself yesterday
lijok 3y ago
Gonna put my pessimist hat on and say, great - another python package manager. If it was at least a drop-in replacement for pip, we could switch to it as we did to ruff just for the speed gains. We need a permanent solve for python package management, governed by the python steering council.
- raccoonDivider 3y ago
  Isn't it a replacement for pip? I've seen people on Twitter saying that they migrated very quickly. TFA: > TL;DR: uv is an extremely fast Python package installer and resolver, written in Rust, and designed as a drop-in replacement for pip and pip-tools workflows.
  - lijok 3y ago
    I tested it on my repo, multiple errors, it's not a drop-in replacement. (.venv) lijok@mbp icebreaker % python -m uv pip install --upgrade uv setuptools wheel error: unexpected argument '--upgrade' found tip: to pass '--upgrade' as a value, use '-- --upgrade' Usage: uv pip install [OPTIONS] <PACKAGE|--requirement <REQUIREMENT>|--editable <EDITABLE>> For more information, try '--help'. (.venv) lijok@mbp icebreaker % python -m uv pip install -- --upgrade uv setuptools wheel error: Failed to parse `--upgrade` Caused by: Expected package name starting with an alphanumeric character, found '-' --upgrade ^ I'm sure it can be made to work - but it's definitely not drop in
    - deleted 3y ago
      [deleted]
    - drcongo 3y ago
      What happens if you don't prefix with `python -m`? There's no mention of doing that in the blog post and it certainly feels wrong to call a rust package with python -m
      - aragilar 3y ago
        That's specifying which python to use. The issue which is printed is there's no support for `--upgrade`, so it's not a drop in replacement for pip.
hprotagonist 3y ago
A VC-backed pip-and-more doesn't make sense to me. It's 2024: what's the revenue model when the free money printer's on the fritz?
- dj_gitmo 3y ago
  That was one of my first questions, but Anaconda exists https://www.anaconda.com/download/ https://www.anaconda.com/download/ Python is used by loads of scientists, academics, and other non software engineers. Those people need an easy way to use python.
  - nateglims 3y ago
    I knew they sold something, but I am amazed to learn they have 300 employees.
    - BiteCode_dev 3y ago
      Anaconda is very big in corporate envs. If you want to have a platform that allow to manage Python on all machines, including allowed packages and version, integrated with ldap, with auditing capabilities, they are pretty much the only game in town. And big companies want that.
      - deleted 3y ago
        [deleted]
      - throwaway2037 3y ago
        I agree. Real question: Why didn't ActiveState (ActivePython) win this battle? In my corporate experience, they are invisible.
        pjmlp 3y ago
        They were quite big in the 2000, on my bubble they faded away as .NET and Java with their IDEs came into the picture, alongside Tcl, Perl and Python improving their Windows support story. They were never big outside Windows shops.
ZeroCool2u 3y ago
So uv uses pubgrub-rs, a rust implementation of the pubgrub version solving algorithm originally written for the dart language. I suppose I shouldn't be surprised, but I always loved Pub the Dart/Flutter packaging tool. It's the only thing out there that comes close to cargo that I know of. Fun to see these chains of inspiration reach across languages.
endgame 3y ago
How does a language ecosystem that bakes "there should be one-- and preferably only one --obvious way to do it" into the interpreter* as a statement of values end up with such a convoluted packaging story? * Try running `python -c 'import this' | sed -n 15p`
- stared 3y ago
  Well, there are languages better at incorporating the Zen of Python than Python itself, vide (full disclosure: my blog post) https://p.migdal.pl/blog/2020/03/types-test-typescript/ https://p.migdal.pl/blog/2020/03/types-test-typescript/. When it comes to package managers, as you noted, the situation is even more ironic, as nicely depicted in https://xkcd.com/1987/ https://xkcd.com/1987/.
  - surfingdino 3y ago
    What's your view on how Golang deals with this problem? Serious question.
    - zadokshi 3y ago
      Golang users don’t sit around arguing about package managers, nor do they have trouble with trying to make sure they have the right version of go installed. Early versions of go didn’t solve things well, but go has had this fixed for a long time now. Any arguments about packaging issues are likely related to older out of date problems people had. The only real problem people have these days is around private non open source packages. You can do it, but you need to learn how to set environment variables.
  - mixmastamyk 3y ago
    One way to do things becomes impossible to after thirty years unless one wants to make large, breaking changes. After Python 3: >>> from __future__ import braces File "<stdin>", line 1 SyntaxError: not a chance
    - 3y ago
marclave 3y ago
> Like pip-compile, uv generates a platform-specific requirements.txt file (unlike, e.g., poetry and pdm, which generate platform-agnostic poetry.lock and pdm.lock files). As such, uv's requirements.txt files may not be portable across platforms and Python versions. really curious on the reasoning behind this decision :)
- Too 3y ago
  Hard to give a concrete example but you can end up in dependency deadlocks, with combination of packages that require new features vs packages that don’t work on newer versions. Mostly it’s useful right after a new release where prebuilt wheels aren’t available for all packages yet and you have users who may care or not about this. If you only have single requirements file, you get forced to choose which platform to support. With multiple, you can break out if needed. These breaking changes and deadlocks are rare. It’s still good to have an escape hatch.
- anentropic 3y ago
  https://github.com/astral-sh/uv?tab=readme-ov-file#multi-version-resolution https://github.com/astral-sh/uv?tab=readme-ov-file#multi-ver... says: > uv does not yet produce a machine-agnostic lockfile. so maybe the non-portable requirements.txt is just a first milestone
bagels 3y ago
What is wrong with pip that people thought we needed conda, and all these other managers?
- duped 3y ago
  If you want to learn every lesson a package manager author has learned in the last twenty years by way of counterexample, use pip. Global dependencies by default (fixed by venvs). Shipping precompiled binaries that are not portable across platforms or python versions (kinda fixed by wheels, I forget the PEP number). No lock files with checksum verification by default (kinda fixed by requirements.txt with require hashes, but not really). Also, there's a bunch of weird shit that can happen if the pip version and python version disagree due setup tools. It's got a great interface for installing stuff. But the likelihood that what it installs breaks something on your machine is pretty high the longer you use it. Edit: the real, fundamental problem with pip is that it can't be saved. Too many dockerfiles and server provisioning scripts are relying on it - it's interface, semantics, CLI output, etc. It's not 100% broken, which is why it's still useful. But any true fix has to be done by creating a separate tool.
- okanat 3y ago
  Imperative dependency (mis)management. Pip resolves what needs to be installed while it installs things depending on the setup script of the package. It requires a full execution and it is neither repeatable nor reproducible. So it is impossible to know what needs to be pulled from Pypi ahead of time. Pip is also strongly tied to the specific Python version. If you upgraded Python interpreter on a system, congrats all virtual environments and all PyPi packages are broken now. Hopefully you saved the minimum set of requirements and they don't have some catastrophic dependency breakage. You run pip install and pray. What? Native binaries? Building them on the fly while installing a pip package? Oh you're brave.
  - mixmastamyk 3y ago
    > upgraded Python ... all virtual environments and all PyPi packages are broken now. Misleading. Continue to use the older version, or packages will need to be installed in a new folder for the new version. Recreate venvs and/or simply reinstall any --user.
ddanieltan 3y ago
Any plans to adopt Rye's approach of using standard Python builds to cover the installing of different Python versions? I feel uv should provide a way to install different python versions to truly cover an end-to-end tool. The current approach of searching for existing virtualenvs of conda envs helps but I was hoping to completely remove the need for another package/dependency manager. (Taken from the docs) If a --python-version is provided to pip compile (e.g., --python-version=3.7), uv will search for a Python interpreter matching that version in the following order: - An activated virtual environment based on the VIRTUAL_ENV environment variable. - An activated Conda environment based on the CONDA_PREFIX environment variable. - A virtual environment at .venv in the current directory, or in the nearest parent directory. - The Python interpreter available as, e.g., python3.7 on macOS and Linux. On Windows, uv will use the same mechanism as py --list-paths to discover all available Python interpreters, and will select the first interpreter matching the requested version. - The Python interpreter available as python3 on macOS and Linux, or python.exe on Windows.
deleted 3y ago
[deleted]
nmca 3y ago
Initially I was confused by the business model here, but I eventually figured it out --- improved python tooling is going to accelerate AI so much that they can just rely on the future gratitude of the machine god.
capital_guy 3y ago
A lot of uncalled for pessimism here in the comments. If python folks reading haven’t used ruff yet, i can highly recommend it. The astral folks have proven themselves to me already. Looking forward to more and more of their rust built python tooling.
davidthewatson 3y ago
I've complained about the resemblance to the Ruby community's packaging conflict a decade ago almost two years ago now. I may have been in the minority, but was apparently not alone: https://news.ycombinator.com/item?id=32116649 https://news.ycombinator.com/item?id=32116649 Python packaging has gotten asymptotically worse in the last 90 days from my experience having used the language for 20 years now and working on a multi-platform triumvirate of Lin-Mac-Win that has not changed in twenty years. I can honestly say that, given how daily failures in the packaging ecosystem such as pip failing to build egg, pkg-config, and similar problems, I'm happy to see a solution from rust, because I sent an intern into the polars underworld 2 years ago to great success and have only grown my own reliance on rust infrastructure more broadly since. This may very well be the only packaging innovation in python that I actually look forward to trying tomorrow. I'd already be working in rust full-time if I could only get past the syntax that reminds me too much of perl after bathing in the semantic whitespace light for this long.
posix_monad 3y ago
Why is this written in Rust and not Python? Does this show deficiencies of Python as a general purpose language?
- liveoneggs 3y ago
  I don't know if the world needs extra examples to prove that python is very slow but this is certainly one more
- mixmastamyk 3y ago
  This company is rust-focused and they compete with fast implementations. Python has always leaned towards prototyping and reimplentation when performance is an issue. They also seem to have money for rapid development which trumps poor community projects.
- kzrdude 3y ago
  That has been discussed before, same question for ruff, rye etc. Maybe it shows the challenges with using python for deployment and distribution of binaries. uv, rye they want to install and manage python, so they try to solve the bootstrapping problem - should a tool that installs python require python to be installed, etc?
KETpXDDzR 3y ago
The most time poetry spends in resolving is downloading all the different versions of a package. Unfortunately, the dependency info is stored in the Python wheel. I can't imagine how uv wants to make the downloads faster.

647 points by samwho 3y ago

Coupling with Rye is very interesting, as that or the PyBi idea seem inevitable to take over. Managing different interpreters is just too much effort which each tool is handling differently. Of course, the PSF will remain silent on the issue and let yet another tool exist in the ecosystem. I really do not care who wins, but we need a standard to be decided.

OJFord 3y ago
I hadn't heard of Rye before I don't think, but sounds like a strange arrangement to me - basically the same project & idea, so the new one to start switched the old one to use it 'under the hood', 'took it over', and will replace it 'when the time is right'? Why.. any of that, why not just buy it and let it be uv instead of starting uv, or just leave it alone?
- the_mitsuhiko 3y ago
  I wrote down a small little FAQ at the bottom, hope it answers some questions: https://lucumr.pocoo.org/2024/2/15/rye-grows-with-uv/ https://lucumr.pocoo.org/2024/2/15/rye-grows-with-uv/
  - OJFord 3y ago
    That is helpful, and fair enough, thanks!
- deleted 3y ago
  [deleted]
theyinwhy 3y ago
Why do people here claim that the PSF has any leverage to stop tools being developed?
- 0cf8612b2e1e 3y ago
  I don’t want the PSF to declare tools cannot be made. I want the PSF to pick a winner so we can finally consolidate on a workflow. Just today, I was helping a beginner setup Python on their laptop. A nightmare collection of tooling complexity (so you have Python, but you need to worry about virtual environments, to do that, I use this combination of pipx+virtualenvwrapper, but to do actual projects, you need Poetry, but…) all the while noting that this is an opinionated workflow that I happen to utilize because of the tradeoffs that matter to me. Ask someone else, and they will definitely have a different configuration. Entire first lesson was just to get Python installed with tons of elided complexity. I should be able to point to a document on Python.org that says, “This is the way.” Yet for decades, they have refused to take a stand.
  - theyinwhy 3y ago
    Isn't it your job as an engineer to pick the right toolchain for the job?

ericra 3y ago

While I don't see the immediate need to switch from using pip, I am such a fan of ruff that I'm always happy to check out what you guys come out with. Please keep doing what you're doing!

drcongo 3y ago

I've been dying to see what they came up with next given how in love I am with Ruff. In my wildest dreams I didn't expect it to be a package resolver. I'm very excited about this. edit: As a side note, if anyone from Astral happens by, any chance of an RSS feed on your blog?

singhrac 3y ago
I was kind of hoping it was a better type checker, something like mypy + pyright. To be fair that's an incredible amount of work, so maybe we'll see that later. Still very excited.
- drcongo 3y ago
  That was actually my first guess, and while I certainly would love to see that, packaging is such a headache for so many people that this could be amazing. Even with poetry I still occasionally get mysterious errors and wildly wrong resolver claims about versions not existing.
  - catlover76 3y ago
    [dead]
- fydorm 3y ago
  Why include mypy at all? Pyright is amazing and all you need.

marmaduke 3y ago

For envs which are pets, who cares, but this will be great for those cattle situations.

SushiHippie 3y ago

Isn't this basically what pixi wants to be? Wouldn't it be better to work together? https://github.com/prefix-dev/pixi/ https://github.com/prefix-dev/pixi/

theLiminator 3y ago
Pixi is even more ambitious but with a different philosophy (I think? Specifically thinking about how pixi went with their own lock and manifest format as well as first class conda support). I'd definitely prefer if they worked together instead or instead dedicated their time to type checking python which imo there still isn't a great solution for.
- SushiHippie 3y ago
  Especially the conda support is IMO a cool thing. As conda/pip interop is just not great. And even micromamba (C++ implementation of conda) is relatively slow to resolve compared to pip. But agreed either work together or create a type checker. I use mypy currently but it definitely slows down my editor.
  - theLiminator 3y ago
    Yeah, pixi has decent mixed support for conda/pypi, it currently solves and installs conda (then locks it) and then solves and installs pypi. I think it's on their roadmap to solve them together, which would be a killer feature.
  - davidhalter 3y ago
    I have been working on a faster type checker for 3.5 years now. It's coming, I promise :)
- fydorm 3y ago
  Have you tried Pyright? It made me actually enjoy Python development.
  - 3y ago

epage 3y ago

Congrats! > Similarly, uv does not yet generate a platform-agnostic lockfile. This matches pip-tools, but differs from Poetry and PDM, making uv a better fit for projects built around the pip and pip-tools workflows. Do you expect to make the higher level workflow independent of requirements.txt / support a platform-agnostic lockfile? Being attached to Rye makes me think "no". Without being platform agnostic, to me this is dead-on-arrival and unable to meet the "Cargo for Python" aim. > uv supports alternate resolution strategies. By default, uv follows the standard Python dependency resolution strategy of preferring the latest compatible version of each package. But by passing --resolution=lowest, library authors can test their packages against the lowest-compatible version of their dependencies. (This is similar to Go's Minimal version selection.) > uv allows for resolutions against arbitrary target Python versions. While pip and pip-tools always resolve against the currently-installed Python version (generating, e.g., a Python 3.12-compatible resolution when running under Python 3.12), uv accepts a --python-version parameter, enabling you to generate, e.g., Python 3.7-compatible resolutions even when running under newer versions. This is great to see though! I can understand it being a flag on these lower level, directly invoked dependency resolution operations. While you aren't onto the higher level operations yet, I think it'd be useful to see if there is any cross-ecosystem learning we can do for my MSRV RFC: https://github.com/rust-lang/rfcs/pull/3537 https://github.com/rust-lang/rfcs/pull/3537 How are you handling pre-releases in you resolution? Unsure how much of that is specified in PEPs. Its something that Cargo is weak in today but we're slowly improving.

OJFord 3y ago
That is great. Fuzzing would be cool too - just completely randomise the versions within the claimed compatibility constraints.
charliermarsh 3y ago
Thanks Ed! Your work as always is a big source of inspiration. > Do you expect to make the higher level workflow independent of requirements.txt / support a platform-agnostic lockfile? Being attached to Rye makes me think "no". Yes, we absolutely do. We don't do this today, the initial scope is intentionally limited. But in the next phase of the project, we want to extend to multi-platform and multi-version resolution. > How are you handling pre-releases in you resolution? Unsure how much of that is specified in PEPs. Its something that Cargo is weak in today but we're slowly improving. This is something we talked with Jacob about quite a bit. Turns out (as you know) it's a very hard problem. For the initial release, we added a constraint: our default behavior is that if you want to use a pre-release, you _have_ to specify the package as a first-party dependency and use a pre-release marker in the version specifier. (We also support globally enabling and disabling pre-releases.) So, we basically don't support "transitive" pre-releases right now -- but we give you a dedicated error message if your resolution fails for that reason.

notatallshaw 3y ago

There are a couple of promising tools written in Rust looking to replace Pip for most users. Rip (https://github.com/prefix-dev/rip/issues https://github.com/prefix-dev/rip/issues) which is more of a library for other rust tools to be on top of like Pixi (which is looking to replace both Pip and Conda). And now uv, which seems to be looking to replace Pip, Pip-Tools, and eventually Poetry and PDM. A lot of the explosion in tools in the Python world is coming from the desire for better workflows. But it has been enabled by the fact that build configuration and calling has been standardized and tool makers are able to follow standards instead of reverse engineering easy install or setup tools. I know a lot of people are put off by there being so many tools, but I think in a few years the dust will settle and there will emerge a best practice work flow that most users can follow. As a primarily Python developer and someone who occasionally contributes to Pip to solve complex dependency resolution issues it does make me wonder if I should hang my hat on that and learn enough rust to contribute to one of these projects eventually.

ShamelessC 3y ago
My experience with Rust developers who dwell in Python land is that they fundamentally disagree with most of the language that is Python and think they know better than incumbents what belongs and what doesn't.
- drawnwren 3y ago
  To be fair, python package management is in such a poor state that I would expect any outside opinion to not be worse.
- klardotsh 3y ago
  The Rust ecosystem gets so much right that honestly, even as a career-long Python developer myself (and Rust for many years, but that's less of my point), they honestly probably do know how to build a good devxp better than much of the Python ecosystem. Put other ways: the Python ecosystem has had 30+ years to figure out how to make packaging not suck. It has continually failed - failed less and less over time, sure, but the story is still generally speaking a nightmare ("throw it all in an OCI container" is an extremely reasonable solution to Python packaging, still, in 2024). I welcome advances, especially those inspired by tooling from languages that focused heavily on developer experience.

eigenvalue 3y ago

Looks awesome. I find that pip is usually pretty fast for me, and when it's not, it is mostly because it has to download so much data or wait for native libraries to compile in the background (or anything involving cuda which always seems to take forever). What really needs some help with speed is conda, which is just so absurdly slow for literally anything, even on ridiculously powerful machines.

paulddraper 3y ago
Yeah, I'm curious how much uv is actually faster. npm -> Yarn was life changing performance-wise. I wonder what pip -> uv is.
- the_mitsuhiko 3y ago
  From my testing in rye it’s significantly faster in day to day. There are numbers in the blog post obviously but it’s the feeling you get using it locally that makes it much more fun to use.
- laborcontract 3y ago
  Bun, ruff/uv, polars.. all have been major qol improvements. I’m loving the pace of release with this crop of speed obsessed projects and I cannot wait for astral to tackle typing.
  - laborcontract 3y ago
    Ok, here are some benchmarks installing an application that has a couple build-heavy dependencies: Installation times (MM:SS): # No cache - uv: 01:05 - pip: 01:56 # Cache: - uv: 00:02 - pip: 00:42
- eigenvalue 3y ago
  npm seems to have gotten a lot faster lately. All that competition from yarn and now bun seems to have pushed them to focus on optimization.

rengler33 3y ago

This is great. Just adopted the pip-tools approach on a project and that has been great. Excited to give this a try.

maxvt 3y ago

I'd be curious to see how "packaging for production" is addressed. As described, this is a tool for development purposes: > Think: a single binary that bootstraps your Python installation and gives you everything you need to be productive with Python, bundling not only pip, pip-tools, and virtualenv, but also pipx, tox, poetry, pyenv, ruff, and more. A lot of that dev tooling is not needed in prod, in fact it is a liability (features available for misuse, code/image size, higher resource requirements). Would there be a "uv-prod" to only deal with the minimum subset of environment setup / container building that enables production use? Would uv build packages that are runnable in prod but themselves don't contain uv? It'd be interesting to hear the plans.

radus 3y ago
My approach for this is to use multi-stage dockerfiles. Build tools live and operate in a build stage, and you copy the /venv into your runtime stage. I think this same approach should work with uv so long as it doesn't bundle itself into the venv.

duped 3y ago

Do you plan to support platform-agnostic lockfiles? From the wording it sounds like "yes" but also "this is a feature not a bug." Which is it? Checksum verification and portable lockfiles are kind of table stakes for me, which is why I use poetry everywhere possible. I can't give up correctness for speed. sidenote: this would be super useful as a library so I could plug it into build systems. Managing python dependencies in standards-compliant ways is a pain, so shelling out to tools like poetry and pip are used even though it would be nicer to write that code eg in Rust and just use the parts you care about.

charliermarsh 3y ago
Yes, we absolutely want to support platform-agnostic lockfiles. Very important thing. The "feature not a bug" tone is probably because we intentionally limited the scope of this release to _not_ support platform-agnostic resolution, since it adds a lot of complexity (and we were still able to ship something that's immediately useful for those that rely on pip and pip-tools today).

captn3m0 3y ago

So what's the plan for monetization? NPM getting acquired by GitHub was an anomaly, and I'm wary of letting VC-backed companies becoming ecosystem lynchpins.

sophacles 3y ago
It's MIT and Apache dual licensed. If the company fails, you fork and move on. If the company never makes a dime, that's OK for me: some folks got paid to make open source tooling.
- anze3db 3y ago
  What they are doing is already discouraging people from contributing to the projects their tools are replacing[0]. If they go out of business and stop supporting their tools, it might leave the Python community in a bad place. [0] https://www.youtube.com/watch?v=XzW4-KEB664 https://www.youtube.com/watch?v=XzW4-KEB664
  - woodruffw 3y ago
    I'm not sure this is a reasonable framing: LLVM stole much of GCC's thunder by being significantly easier to contribute to (and extend externally), but I don't think it would be accurate to say that LLVM is "discouraging" people from contributing to GCC.
    - ForHackernews 3y ago
      The slow death of GCC has been unquestionably bad for copyleft software.
      - woodruffw 3y ago
        Why is it LLVM's job to be good for copyleft software?
        3y ago

alfalfasprout 3y ago

While we don't particularly care about pip lockfiles, etc. as we're moving to the conda/mamba ecosystem a faster pip is most welcome. We invoke it a lot programmatically and it's very, very slow.

iwebdevfromhome 3y ago

Very nice! Not long ago we got rid of a bunch of tools at work in favor of ruff. I'm looking forward to this. Specially when the time comes to be able to replace poetry with it, hoping it is a good implementation.

murkt 3y ago

An ability to override dependencies, finally!

amelius 3y ago

Iirc, some pip packages require compilation which depends on entire toolchains with e.g. gcc, g++, and with dependencies like gtk, Qt, etc. How do they intend to make that less error prone and thus more user-friendly?

cavallo71 3y ago
That's exactly what conda/mamba/micromamba is covering: the whole toolchain.
- amelius 3y ago
  I've used conda/mamba, but often ended up with broken installs (random coredumps) that were mysteriously fixed by just using pip3 instead.
- ZeroCool2u 3y ago
  That's what they attempt to cover, but they're not super successful at it.
mixmastamyk 3y ago
Wheels have been a thing for a decade+. If not available from the developer the complexity doesn't reduce much.
- wiredfool 3y ago
  That’s the really pointy end of packaging. Pillow (python +c+ External dlls) produces something like 50 wheels per release, and we’ve still got platforms where you’ve got to compile your own. Since they tend to be lower volume, they also tend to have more interesting compiler issues.
  - amelius 3y ago
    Yes, I'm using Python on an nVidia Jetson system with PyTorch, which depends on CUDA, and all my other dependencies are quite challenging to get working in concert. Every time pip3 starts compiling under the hood, I'm just praying that it will work.

ofek 3y ago

Exciting stuff! I view Hatch [1] as becoming the Cargo for Python because it's already close and has an existing (and growing) user base but I can definitely see depending on this for resolution and potentially not even using pip after it becomes more stable. [1]: https://hatch.pypa.io/latest/ https://hatch.pypa.io/latest/

woodruffw 3y ago

This is very exciting! Congratulations to the Astral team on their work here. I have historically expressed trepidation towards external attempts to "fix" Python packaging, largely because each attempt has historically accreted another layer of incompatibilities and hacks onto the giant pile of hacks that are already needed to keep the ecosystem running. As such, it makes me happy to see that compatibility is a priority here: the Astral team has gone out of their way to emphasize both formal (PEP) and informal (pip CLI) compatibility with existing tooling and standards. (Separately, I share concerns about VC funding and sustainability. But everything I've seen so far indicates that Charlie and the other folks at Astral are not in it to screw or otherwise leverage the Python community. So I choose to be optimistic here.)

eviks 3y ago
Prioritizing compatibility with a giant incompatible pile of historical hacks is how you grow the pile
- woodruffw 3y ago
  I don't think that's true, at least in the case of Python packaging. The pile has historically grown because of informal or non-existent standards, along with multiple independent implementations attempting to align their observed (but not specified) behaviors.
- fwip 3y ago
  Well, they do explicitly list certain historical features that they don't intend to support, like installation from eggs or editable installs. So they're doing some work to trim the pile while they're there.
  - SonOfLilit 3y ago
    They do support editable installs, but only from local directories. So you have to perform git clone yourself and then install.

SamEdwardes 3y ago

Sign me up! A single binary to manage Python is something I have been hoping for.

chrisshroba 3y ago

Obligatory two xkcd's, but I'm super excited about this project! Definitely going to try it out. https://xkcd.com/1987/ https://xkcd.com/1987/ https://xkcd.com/927/ https://xkcd.com/927/

nindalf 3y ago

Feel like I called this 11 days ago - https://news.ycombinator.com/item?id=39251014 https://news.ycombinator.com/item?id=39251014 > I had to guess, that’s the path that the Astral team would take as well - expand ruff’s capabilities so it can do everything a Python developer needs. So the vision that Armin is describing here might be achieved by ruff eventually. They’d have an advantage that they’re not a single person maintenance team, but the disadvantage of needing to show a return to their investors.

12_throw_away 3y ago

This article says they don't support "platform-agnostic lockfiles" YET. And platform-agnostic lockfiles have been mentioned in several comments here, times too. I haven't encountered these before (with python, anyway) - how do they work platform-specific dependencies, e.g., binary wheels? For instance, there are >30 dists for numpy 1.26.4 [1], with different artifacts for different platforms and python interpreters. Which of those hashes ends up in a platform-agnostic lockfile? Do you have to stick with sdists only? [1] https://pypi.org/project/numpy/#files https://pypi.org/project/numpy/#files

orf 3y ago
All of them - poetry adds them as an array, and the right one is installed for the platform.
jvolkman 3y ago
PDM and Poetry would include hashes for all possible wheels, and would also follow transitive paths for platforms other than the current host. E.g. if my host is Linux and I depend on foo, and foo depends on bar only on windows, and bar depends on baz, my lock file will still contain baz.

gary_0 3y ago

Something I've been waiting to see from language package managers is more attention paid to security. I believe both cargo and pip packages can run arbitrary code as the local user the instant they are installed, and malicious packages have existed in the wild for years. I also recall a blog post where someone was scanning PyPI for malicious Python packages, only to realize that `pip download` also executed code. Just downloading a library to a local source code directory should not cause arbitrary code to run on your system, and there should be safeguards in place so developers are not one typo away from a malicious package pwning their entire system. Supply chain attacks remain a huge area of concern. Instead, when I "ctrl+f security" the homepages of any of these new packaging systems, I get 0 results. Not good.

woodruffw 3y ago
> I also recall a blog post where someone was scanning PyPI for malicious Python packages, only to realize that `pip download` also executed code. I think you're thinking of this post[1]. The code being searched for wasn't malicious, just buggy :-) [1]: https://moyix.blogspot.com/2022/09/someones-been-messing-with-my-subnormals.html https://moyix.blogspot.com/2022/09/someones-been-messing-wit...
- gary_0 3y ago
  Thanks, that's the one! I was actually just trying to find it. Heh, I forgot about the anime catgirl part.

the__alchemist 3y ago

Very cool! Of note, I made something along these lines a few years ago, although with a slightly broader scope to also include managing and installing python versions. I abandoned it due to lack of free time, and edge cases breaking things. The major challenge is that Python packages that aren't wheels can do surprising things due to setup.py running arbitrary code. (https://github.com/David-OConnor/pyflow https://github.com/David-OConnor/pyflow) For some more context, at the time, poetry and pipenv were both buggy, and had some limitations their teams listed as "wont-fix". The sort of thing where you hit a breaking bug immediately on certain systems. These have since been fixed.

singhrac 3y ago

Something that is super super important to me is editable installs, and local references to those editable installs. Often I have several editable packages set up in a virtualenv which I think of as like my "base" virtualenv. One reason I struggled with Rye earlier is that it kind of felt like this wasn't entirely supported, possibly because I couldn't parse the information on the workspaces or virtual projects page. Maybe someone else figured this out? It does seem common enough that I would be surprised if it wasn't supported.

charliermarsh 3y ago
I think having a good story for local workspaces is gonna be a key piece of this next phase of the project, where we grow uv from a "pip replacement" to a "Poetry replacement" -- or, phrased differently, to something that looks more like Cargo. (Cargo does a bunch of nice things here that I want to learn from.)
- singhrac 3y ago
  I think Rye actually does handle this mostly correctly (as the sibling comment said). I got through some of it here: https://github.com/mitsuhiko/rye/issues/671 https://github.com/mitsuhiko/rye/issues/671. I think it's very close to what I actually want (maybe not what Armin wants with multiversion).
Zizizizz 3y ago
This does editable installs, I tried it myself yesterday

lijok 3y ago

Gonna put my pessimist hat on and say, great - another python package manager. If it was at least a drop-in replacement for pip, we could switch to it as we did to ruff just for the speed gains. We need a permanent solve for python package management, governed by the python steering council.

raccoonDivider 3y ago
Isn't it a replacement for pip? I've seen people on Twitter saying that they migrated very quickly. TFA: > TL;DR: uv is an extremely fast Python package installer and resolver, written in Rust, and designed as a drop-in replacement for pip and pip-tools workflows.
- lijok 3y ago
  I tested it on my repo, multiple errors, it's not a drop-in replacement. (.venv) lijok@mbp icebreaker % python -m uv pip install --upgrade uv setuptools wheel error: unexpected argument '--upgrade' found tip: to pass '--upgrade' as a value, use '-- --upgrade' Usage: uv pip install [OPTIONS] <PACKAGE|--requirement <REQUIREMENT>|--editable <EDITABLE>> For more information, try '--help'. (.venv) lijok@mbp icebreaker % python -m uv pip install -- --upgrade uv setuptools wheel error: Failed to parse `--upgrade` Caused by: Expected package name starting with an alphanumeric character, found '-' --upgrade ^ I'm sure it can be made to work - but it's definitely not drop in
  - deleted 3y ago
    [deleted]
  - drcongo 3y ago
    What happens if you don't prefix with `python -m`? There's no mention of doing that in the blog post and it certainly feels wrong to call a rust package with python -m
    - aragilar 3y ago
      That's specifying which python to use. The issue which is printed is there's no support for `--upgrade`, so it's not a drop in replacement for pip.

hprotagonist 3y ago

A VC-backed pip-and-more doesn't make sense to me. It's 2024: what's the revenue model when the free money printer's on the fritz?

dj_gitmo 3y ago
That was one of my first questions, but Anaconda exists https://www.anaconda.com/download/ https://www.anaconda.com/download/ Python is used by loads of scientists, academics, and other non software engineers. Those people need an easy way to use python.
- nateglims 3y ago
  I knew they sold something, but I am amazed to learn they have 300 employees.
  - BiteCode_dev 3y ago
    Anaconda is very big in corporate envs. If you want to have a platform that allow to manage Python on all machines, including allowed packages and version, integrated with ldap, with auditing capabilities, they are pretty much the only game in town. And big companies want that.
    - deleted 3y ago
      [deleted]
    - throwaway2037 3y ago
      I agree. Real question: Why didn't ActiveState (ActivePython) win this battle? In my corporate experience, they are invisible.
      - pjmlp 3y ago
        They were quite big in the 2000, on my bubble they faded away as .NET and Java with their IDEs came into the picture, alongside Tcl, Perl and Python improving their Windows support story. They were never big outside Windows shops.

ZeroCool2u 3y ago

So uv uses pubgrub-rs, a rust implementation of the pubgrub version solving algorithm originally written for the dart language. I suppose I shouldn't be surprised, but I always loved Pub the Dart/Flutter packaging tool. It's the only thing out there that comes close to cargo that I know of. Fun to see these chains of inspiration reach across languages.

endgame 3y ago

How does a language ecosystem that bakes "there should be one-- and preferably only one --obvious way to do it" into the interpreter* as a statement of values end up with such a convoluted packaging story? * Try running `python -c 'import this' | sed -n 15p`

stared 3y ago
Well, there are languages better at incorporating the Zen of Python than Python itself, vide (full disclosure: my blog post) https://p.migdal.pl/blog/2020/03/types-test-typescript/ https://p.migdal.pl/blog/2020/03/types-test-typescript/. When it comes to package managers, as you noted, the situation is even more ironic, as nicely depicted in https://xkcd.com/1987/ https://xkcd.com/1987/.
- surfingdino 3y ago
  What's your view on how Golang deals with this problem? Serious question.
  - zadokshi 3y ago
    Golang users don’t sit around arguing about package managers, nor do they have trouble with trying to make sure they have the right version of go installed. Early versions of go didn’t solve things well, but go has had this fixed for a long time now. Any arguments about packaging issues are likely related to older out of date problems people had. The only real problem people have these days is around private non open source packages. You can do it, but you need to learn how to set environment variables.
- mixmastamyk 3y ago
  One way to do things becomes impossible to after thirty years unless one wants to make large, breaking changes. After Python 3: >>> from __future__ import braces File "<stdin>", line 1 SyntaxError: not a chance
  - 3y ago

marclave 3y ago

> Like pip-compile, uv generates a platform-specific requirements.txt file (unlike, e.g., poetry and pdm, which generate platform-agnostic poetry.lock and pdm.lock files). As such, uv's requirements.txt files may not be portable across platforms and Python versions. really curious on the reasoning behind this decision :)

Too 3y ago
Hard to give a concrete example but you can end up in dependency deadlocks, with combination of packages that require new features vs packages that don’t work on newer versions. Mostly it’s useful right after a new release where prebuilt wheels aren’t available for all packages yet and you have users who may care or not about this. If you only have single requirements file, you get forced to choose which platform to support. With multiple, you can break out if needed. These breaking changes and deadlocks are rare. It’s still good to have an escape hatch.
anentropic 3y ago
https://github.com/astral-sh/uv?tab=readme-ov-file#multi-version-resolution https://github.com/astral-sh/uv?tab=readme-ov-file#multi-ver... says: > uv does not yet produce a machine-agnostic lockfile. so maybe the non-portable requirements.txt is just a first milestone

bagels 3y ago

What is wrong with pip that people thought we needed conda, and all these other managers?

duped 3y ago
If you want to learn every lesson a package manager author has learned in the last twenty years by way of counterexample, use pip. Global dependencies by default (fixed by venvs). Shipping precompiled binaries that are not portable across platforms or python versions (kinda fixed by wheels, I forget the PEP number). No lock files with checksum verification by default (kinda fixed by requirements.txt with require hashes, but not really). Also, there's a bunch of weird shit that can happen if the pip version and python version disagree due setup tools. It's got a great interface for installing stuff. But the likelihood that what it installs breaks something on your machine is pretty high the longer you use it. Edit: the real, fundamental problem with pip is that it can't be saved. Too many dockerfiles and server provisioning scripts are relying on it - it's interface, semantics, CLI output, etc. It's not 100% broken, which is why it's still useful. But any true fix has to be done by creating a separate tool.
okanat 3y ago
Imperative dependency (mis)management. Pip resolves what needs to be installed while it installs things depending on the setup script of the package. It requires a full execution and it is neither repeatable nor reproducible. So it is impossible to know what needs to be pulled from Pypi ahead of time. Pip is also strongly tied to the specific Python version. If you upgraded Python interpreter on a system, congrats all virtual environments and all PyPi packages are broken now. Hopefully you saved the minimum set of requirements and they don't have some catastrophic dependency breakage. You run pip install and pray. What? Native binaries? Building them on the fly while installing a pip package? Oh you're brave.
- mixmastamyk 3y ago
  > upgraded Python ... all virtual environments and all PyPi packages are broken now. Misleading. Continue to use the older version, or packages will need to be installed in a new folder for the new version. Recreate venvs and/or simply reinstall any --user.

ddanieltan 3y ago

Any plans to adopt Rye's approach of using standard Python builds to cover the installing of different Python versions? I feel uv should provide a way to install different python versions to truly cover an end-to-end tool. The current approach of searching for existing virtualenvs of conda envs helps but I was hoping to completely remove the need for another package/dependency manager. (Taken from the docs) If a --python-version is provided to pip compile (e.g., --python-version=3.7), uv will search for a Python interpreter matching that version in the following order: - An activated virtual environment based on the VIRTUAL_ENV environment variable. - An activated Conda environment based on the CONDA_PREFIX environment variable. - A virtual environment at .venv in the current directory, or in the nearest parent directory. - The Python interpreter available as, e.g., python3.7 on macOS and Linux. On Windows, uv will use the same mechanism as py --list-paths to discover all available Python interpreters, and will select the first interpreter matching the requested version. - The Python interpreter available as python3 on macOS and Linux, or python.exe on Windows.

deleted 3y ago

[deleted]

nmca 3y ago

Initially I was confused by the business model here, but I eventually figured it out --- improved python tooling is going to accelerate AI so much that they can just rely on the future gratitude of the machine god.

capital_guy 3y ago

A lot of uncalled for pessimism here in the comments. If python folks reading haven’t used ruff yet, i can highly recommend it. The astral folks have proven themselves to me already. Looking forward to more and more of their rust built python tooling.

davidthewatson 3y ago

I've complained about the resemblance to the Ruby community's packaging conflict a decade ago almost two years ago now. I may have been in the minority, but was apparently not alone: https://news.ycombinator.com/item?id=32116649 https://news.ycombinator.com/item?id=32116649 Python packaging has gotten asymptotically worse in the last 90 days from my experience having used the language for 20 years now and working on a multi-platform triumvirate of Lin-Mac-Win that has not changed in twenty years. I can honestly say that, given how daily failures in the packaging ecosystem such as pip failing to build egg, pkg-config, and similar problems, I'm happy to see a solution from rust, because I sent an intern into the polars underworld 2 years ago to great success and have only grown my own reliance on rust infrastructure more broadly since. This may very well be the only packaging innovation in python that I actually look forward to trying tomorrow. I'd already be working in rust full-time if I could only get past the syntax that reminds me too much of perl after bathing in the semantic whitespace light for this long.

posix_monad 3y ago

Why is this written in Rust and not Python? Does this show deficiencies of Python as a general purpose language?

liveoneggs 3y ago
I don't know if the world needs extra examples to prove that python is very slow but this is certainly one more
mixmastamyk 3y ago
This company is rust-focused and they compete with fast implementations. Python has always leaned towards prototyping and reimplentation when performance is an issue. They also seem to have money for rapid development which trumps poor community projects.
kzrdude 3y ago
That has been discussed before, same question for ruff, rye etc. Maybe it shows the challenges with using python for deployment and distribution of binaries. uv, rye they want to install and manage python, so they try to solve the bootstrapping problem - should a tool that installs python require python to be installed, etc?

KETpXDDzR 3y ago

The most time poetry spends in resolving is downloading all the different versions of a package. Unfortunately, the dependency info is stored in the Python wheel. I can't imagine how uv wants to make the downloads faster.