5 ms·
uv: Deduplicate all files in the wheel cache
- dboreham 2mo agoPeople say uv is good because it's fast, but honestly I don't care about that. We switched to using it for distribution of our Python-based tools because it makes it very convenient to install directly from a git repository and then to subsequently update from same repository. No need to build a package.
- Neywiny 2mo agoI tried it out successfully for the first time the other day (had 1 false start some months ago). This is after 10 ish years of system wide installs or venvs. I didn't find it fast at all. Every time I went to run the script it spent multiple seconds checking dependencies. Then one time it updated one, which luckily didn't break anything but I did get concerned. I'm sure there're some flags I didn't know to use but uvx was not great. On the other hand, it did seem to install the packages faster than pip.
- nmstoker 2mo agoYMMV depending on specific needs but generally even on Windows on an average machine with an imperfect setup I've found it exceptionally fast, often well inside the sub-second range, to the point that I do sometimes worry if it really ran. Sounds like it's worth another look at your settings to make sure they are right.
- intoXbox 2mo agoI do avoid uv run for this reason but it’s useful for managing python projects. The speed claims actually have a lot to do with efficient caching, and I run several projects on the same Python (patch) version with similar packages on the same system
- coldtea 2mo ago>I didn't find it fast at all. Compared to what? Fast is relative. Compared to pip it's miles ahead.
- Neywiny 2mo agoCompared to just running the script with system-wide installs or in a venv, as I mentioned originally
- deleted 2mo ago[deleted]
- zanie 2mo agoIf you want to open an issue with verbose logs (`-vv`) I'd be happy to look into it. You can set `UV_LOG_CONTEXT=1` to emit timing information. If you're running Python scripts, I'd recommend using `uv lock --script <path>` to generate a lockfile — we don't do that by default for scripts yet but that will avoid unexpected upgrades. (I work on uv)
- nchammas 2mo agoMaybe I've misunderstood you, but that's not a special advantage of uv. pip has been able to do this for many years. pip install git+https://github.com/some-org/repo
- deleted 2mo ago[deleted]
- stephenlf 2mo agouv is the backbone of any modern Python library. https://stephenlf.dev/blog/python-library-in-2026/ https://stephenlf.dev/blog/python-library-in-2026/
- blactuary 2mo agoThis is exactly the kind of brief summary I was looking for to get my team more organized, thanks for this
- apple1417 2mo agoI have never gotten the speed argument either. Yes it's noticeably faster - but in all my projects it's 1s vs 0.1s, not enough to make a meaningful difference. Even in their benchmarks, the worst number they ever give for pip is 7s [1] (and those numbers are 2+ years old, with python's improvements pip should be doing a little better now). I just can't say I've ever cared about installing dependencies, something you do once, taking 7s longer. Nor do I care how long CI takes, it running in the background is the point. The one use I can give it is for running scripts with inline dependencies. I have found it a noticeable improvement over pipx there. But that's more due to needing to parse dependencies every single launch, if it's something I use regularly I end up just installing them normally instead, and it's faster than either. [1] https://github.com/astral-sh/uv/blob/main/BENCHMARKS.md https://github.com/astral-sh/uv/blob/main/BENCHMARKS.md
- maeln 2mo agoFor me, the speed is the least interesting part of `uv`. What I like about it is what it bundle into a single bin : Managing installed python version, automatically creating local .venv for the project (I don't like software like pipenv who install the venv who knows where in a global directory of their choosing), support of pyproject.toml, including of the python version declared in it, and running script with dependencies. None of it is unique to `uv` I believe, but it does it all, reliably, is easy to install, and easy to use. In terms of DX, for my use cases, it beat poetry, pipenv, pyenv (for python version management) and just using pip
- BrokenCogs 2mo agoFor a uv noob like me: what do you mean by bundling it all into a single binary? How does that work?
- deleted 2mo ago[deleted]
- MayeulC 2mo agoUV can be built (or downloaded) as a single optionally statically linked with the libc executable file, with no other dependencies. That makes it very easy to distribute on weird systems.
- dimatura 2mo agoIt's not such a big deal compared to pip for a typical python package, but a couple of use cases where the speed is a nice feature are tool running with uvx and inline dependencies in scripts - you'll still see something is installed the first time you run the tool/script, but usually it's fast enough to not notice. One could also make a comparison to another popular tool, conda, which is glacially slow, but that would arguably be unfair since conda does things uv doesn't. More generally, it's still a nice thing, though. It's just a little less friction. Even a few seconds slower can nudge you towards different behavior
- TacticalCoder 2mo ago> deduplication at the file level: every file is now stored under its BLAKE3 hash Blake3 is really a wonderfully fast cryptographic hash. I use it for my own "deduplication / integrity / berzerker" utility (which I made before LLMs were a thing). If I've got a file named: DSC98731-b3-7b39197a22.JPG then: - if that file doesn't checksum back to 7b39197a22 there's a file integrity problem (amazing and it already helped me troubleshoot issues) - if any other file has the same Blake3 7b39197a22 hash, it's a duplicate - if that 7b39197a22 checksum is in my database, "things can happen". For example my DB can say "any file with a Blake3 hash of 7b39197a22 can always be deleted" or "any file with a Blake3 hash of 887463c09e, if it's got a generic filename like "dscXXXXX" can always be renamed to "20260722jackJohnAtTheBeach-b3-778463c09e.jpg" (or whatever suits you). It's really great (and I know several here independently made similar schemes) and Blake3 is an amazing hash for those kind of use.
- Someone 2mo agoFor those wondering like me: Blake3 generates hashes of at least 224 bits, not, as a literal reading of that comment indicates, 40 bits (which would be bad for file deduplication, giving you a 50% hash collision after around a million files)
- TacticalCoder 2mo agoYup sorry if I mislead people. I usually use 10 hexdigits (40 bits) but no matter how many there are, it's the x bits of the beginning of the checksum that are verified against the hash (in the examples I gave 40 bits). I wasn't very clear.
- pama 2mo agoSo at 3 million different files you have a 98.3% chance of a hash collision. Wouldnt that cause problems in real datasets?
- dist-epoch 2mo agoIt's annoying that most file formats don't checksum their own content. Even formats which should know better, like SQLite, delegate that to the filesystem, most of which are also not checksumed and which delegate that further to the storage. PostgreSQL, which prides itself by it's quality and reliability, only turned on checksums by default in the last version, 18. This is one great benefit of using .zip files as file formats, you get this for free.
- as1297kj 2mo ago[flagged]
- embedding-shape 2mo ago> By using uv, you are literally supporting the people who want to make you unemployed for stock options. By using any FOSS or software produced by software developers and programmers, you are supporting the idea that computation can and should be automated, instead of having humans doing it. This is the origin of computing, and what we've been doing so far, and it continues to "eat the world" via automation, just like the past decades of it.
- LeBit 2mo agoWhat is the word? "Facetious"?
- maeln 2mo agoIt's so human that many software engineers started to worry about automation and being replaced when it started to be something that might affect them.
- ashg1-qwzudg 2mo agoAnd it is machine like that many software engineers still cheer on being replaced once it is happening. Software engineers are the socially dumbest creatures on the planet and deserve every bit of scorn that was heaped on them at high school. The industry used them for multi level marketing since about 2010 and now has instructed them to direct the MLM against themselves. And they comply!
- wiseowise 2mo ago“You hate society yet you still participate in it” with smug soyjack face isn’t the slam dunk you think it is.
- ashg1-qwzudg 2mo agoThat is a false equivalency. FOSS started as a counter movement against Microsoft in order to escape from surveillance and corporate control. The AI cartel is the new Microsoft. It is amazing how software developers have been brainwashed. Ted Ts'o, Google employee and pro-AI shill at Debian and LWN, recently argued that supporting AI is similar to supporting the Internet in 2000. That is another fallacy. The Internet is a common carrier and was supposed to lead to freedom in 2000. It has nothing to do with outsourcing your thinking to a bunch of wannabe trillionaires.
- CivBase 2mo agoA 10% reduction in cache size in exchange for a 4% slowdown doesn't seem obviously worthwhile to me, especially when it comes at an increase in complexity.
- colechristensen 2mo agoWhereas I made a mistake and bought an underspeced MacBook and my 512GB disk is constantly on the brink of full as I reclaim the last 40 GB over and over from different caches, downloads, and wherever else and had to simply give up on several projects because of their disk usage. Disk isn't free, especially now.
- CivBase 2mo agoI don't disagree. But if you're in that position, surely you'd be better off emptying the cache and disabling it?
- colechristensen 2mo agoThe things which use my disk are things like package caches in several languages, build artifacts in both release and debug, multiplied by worktrees, things like iOS device support and release files + simulators, docker containers and related. I had to give up on things like Lean theorem prover projects and VM projects because the disk space wasn't there. The causes of disk waste are manifold and usually can't just be disabled and even when they can the result is constant rebuilding, redownloading, etc. Too many developer products act like disk and memory is free to waste and the consequences have gotten ridiculous.
- NoboruWataya 2mo agoExactly my experience, even worse because for me it's 2x256GB drives. Eventually I bit the bullet and upgraded one of them to 1TB, but not the one my root partition is mounted on (too lazy for that), so now I find myself moving various large directories across to my "extra" drive and symlinking them. Funnily enough I don't remember this ever being a problem back when 50GB was considered a lot.
- stephenlf 2mo agouv is the backbone of any modern Python library. I’m excited to see improvements. https://stephenlf.dev/blog/python-library-in-2026/ https://stephenlf.dev/blog/python-library-in-2026/
- 310298 2mo agoNonsense. uv is one of many existing build tools. All the items can be done by other programs. Funny to list OpenAI as an example at the end of the blog post ...
- tingletech 2mo agothey just implement the modern PEPs
- woodruffw 2mo agoWe do a lot more than that, e.g. how to cache distributions is not defined anywhere within PEPs (and should not be, since it’s a purely internal tool decision). But yes, it helps that uv implements the PEP for detached metadata, particularly during resolution.
- tingletech 2mo agoDon't get me wrong, I love `uv` -- I mean as far as being "the backbone of any modern Python library" -- shouldn't any modern python library work with any packing system that follows the PEPs?
- woodruffw 2mo agoIt depends on what you mean. The PEPs are essentially a behavioral core, and each tool (build backend, etc.) adds on top of that core. If you’re installing a package, any installer should work; if you’re developing a package, you may be subject to implementation details from your development tool of choice.
- 1mo ago
- notatallshaw 2mo agoAs a pip maintainer, I've long been looking at the tradeoffs of uv's cache, it's the biggest item that makes warm installs faster for uv vs. pip. As pip caches the original distributions and then has to unzip them each time, uv caches the unzipped distribution and hard links to it if it can. But it has always had two major issues: 1. No way to reproduce exact distributions for a "download" command (there is no uv equivalent of "pip download") 2. For people with a lot of different environments the cache grows significantly more than pip I'm interested to see, at least anecdotally, if this significantly improves 2, then we can perhaps have a two layer caching strategy without the significant disk space cost.
- user5994461 2mo agoAlso as a pip contributor. The only advantage of uv is to have support for parallel async extraction. If pip extracted multiple files/wheels in parallel without being blocked by the GIL, pip could easily match or outcompete uv. I'm personally not looking forward to any deduplication/hardlink in pip. Hardlinks are very dangerous and system dependent. It's very dangerous for empty files (init.py, empty.log yet not written). When the user edits one file, all files are modified simultaneously, all venv ever created by the user can be broken by editing one file, which is quite catastrophic. It's also dangerous for small files with repeated content, for example random settings files that would contain a "1" or "true". Again, when the user edits one file, all files are edited and they were supposed to be different! Hypothetically, a simple deduplication of binary files (.dll .so) should achieve 50% of the savings without significant drawbacks I'd venture to say that pip extraction is more optimized than uv in at least one way. We have optimization for empty files (0 bytes) because there is nothing to write and checksum. uv doesn't seem to have the same optimizations, though I could be wrong, I just had a cursory look and my rust is not great. uv should probably review their treatment of empty files, it's counter productive to do any file system operation open/read/write because there is no content, it might be counterproductive to use any cache/comparison/hardlink if it takes more operations than doing nothing.
- zanie 2mo ago(I work on uv) > The only advantage of uv is to have support for parallel async extraction. This isn't true, the biggest speed ups are for the warm cases where we've already unpacked the files into the cache, as notatallshaw mentions above. We can also make low-level optimizations during resolution, e.g., in version parsing, that are not possible in pure Python code. > If pip extracted multiple files/wheels in parallel without being blocked by the GIL, pip could easily match or outcompete uv. I'm a bit confused by these claims about the GIL? The expensive IO operations release the GIL. > When the user edits one file, all files are modified simultaneously This is why we default to reflinks or copy-on-write semantics when creating environments, not all file systems support it but it's becoming more common. > Hypothetically, a simple deduplication of binary files (.dll .so) should achieve 50% of the savings without significant drawbacks We also explored this (see https://github.com/astral-sh/uv/pull/19694 https://github.com/astral-sh/uv/pull/19694) and the linked pull request has a table comparing to this strategy. > We have optimization for empty files (0 bytes) because there is nothing to write and checksum. Interesting, I would be very surprised if this made a significant difference? but I'll take a look.
- mark_l_watson 2mo agoNice improvement. For me, uv is the ‘Quicklisp for Python.’ uv just let me enjoy using Python like Quicklisp just makes Common Lisp nicer to use. I have always been a Lisp devotee, but a few years ago when I started using uv, I then started seeing Python as a language I could really enjoy using so I put effort into making my Python dev setup nearly frictionless.