6 ms·
Uv saves Home Assistant 215 compute hours per month
- ZoomerCretin 3y agoWould the authors not have been better off adding their performance improvements to pip rather than creating their own tool? I remember this being a complaint from Node maintainers when Bun was released: they had multiple open performance-related issues with no one available to work on them.
- dmix 3y agoMost innovations happens outside large organizations, then either replaces it or the ideas get stolen/absorbed into the parent. All the downsides and barriers of operating in a large legacy system are pretty well known, even if the maintainers are receptive, radical change will be DOA.
- stouset 3y agoI’m guessing “rewrite it in Rust” would have been an unpalatable PR for the pip maintainers to merge.
- dmix 3y agoYeah "lets rewrite in Rust" has become a meme with negative connotations that will attract ridiculed or be ignored, but Rust clearly excels in this sort of tooling. Whether it makes sense to rewrite an existing codebase or pigeonhole it on the edges is another question.
- jonathankoren 3y agoYeah but I suspect there’s something more going on than Rust magic. They’re claiming a savings of about an hour and a half. (4% of the original time) That sounds like an architecture change, or more likely a whole lot of caching.
- 01HNNWZ0MV43FF 3y agoI guess the uv authors don't want to maintain a better algorithm in Python then
- andrewjf 3y agoI don’t know how you can seriously imply that Python could be just as fast as Rust if the authors of uv cared enough. That seems like a disingenuous framing of why UV exists, let alone doesn’t pass the smell test that Python can just be as fast as a compiled, native machine code language. Python and Ruby and all that are just so god awful slow that it’s completely obvious that’s a rewrite in native code would have these sorts of benefits.
- eyegor 3y agoIf the first step of your build process is creating a new pip venv from scratch, it's entirely believable to me. Pip resolves packages sequentially, so there's a lot of low hanging fruit even without yoinking a state of the art constraint solver. As a concrete example, I have a raspberry pi 4 I use for "what's the most pathetic hardware a user might try". It takes 5-10 minutes to build up a fresh venv even when it has all of the packages cached. If I nuke the caches and make it download, it's 20+ minutes. For comparison a modern laptop takes ~1 minute.
- fbdab103 3y agoIs that on a SD card? My expectation is that SD cards are garbage and any number of IO access patterns break down vs a SSD.
- eyegor 3y agoI threw in a moderately fast USB drive (~90 mb/s after pi4 overhead) because I got sick of SD cards dying every 6-12 months. Pi4 is very speed limited. Even if you tried to use an ssd, I think the highest I've seen is around 250mb/s with an nvme drive that gets a few gb/s in a real computer. In theory it should get to like 500mb/s for sata limits.
- rtpg 3y agoIt's tricky to change up some parts of pip because it might break some existing users flows. It's really hard to test for this kind of infrastructure tool because it's so environment-dependent. The rewrite lets there be a clean slate. Meanwhile the target of being "compatible" with pip gives them a good idea of where to go, and they can focus a bit more on testing given the lessons learned.
- swyx 3y agoi think now their current positioning is that the competition actually helped demonstrate the value of some kinds of speed, and that finally got those performance issues prioritized enough to work on by people with the right knowledge. Node doesn't lack for people who want to contribute, its more about people who -can- contribute well, and yes as other comment noted, its often to easier to try things out outside the system rather than work within it.
- paxys 3y agouv isn't a performance improvement on pip but a ground-up rewrite in a completely different language.
- nojito 3y agoSource on the logic being different from pip? Last I checked it's nothing more than a port of pip to rust.
- btheunissen 3y agoI’m unclear on how you “port” a library from Python to Rust, it’s a ground up rewrite with a familiar interface to pip but fundamentally doesn’t share any code.
- gpm 3y agoIt's actually pretty easy to call python from rust and rust from python. An incremental port of a python library into rust would definitely be possible. I'm not sure it would be a good idea. Python patterns don't necessarily make good rust patterns. Even ignoring larger code structure (e.g. single ownership*) you'd practically have to do a second pass to stop using python compatible types (except in the exposed interface) if you wanted an idiomatic rust library at the end of it. But it's definitely possible. * Which you don't "have to" do in rust, you can use `Rc<RefCell>` everywhere and imitate python instead. It's just throwing some of the advantages of rust and adding some noise to type signatures.
- rovr138 3y agoIt is not. > Limitations > While uv supports a large subset of the pip interface, it does not support the entire feature set. In some cases, those differences are intentional; in others, they're a result of uv's early stage of development. > For details, see our pip compatibility guide[1] > Like pip-compile, uv generates a platform-specific requirements.txt file (unlike, e.g., poetry and pdm, which generate platform-agnostic poetry.lock and pdm.lock files). As such, uv's requirements.txt files may not be portable across platforms and Python versions. [1] - https://github.com/astral-sh/uv/blob/main/PIP_COMPATIBILITY.md https://github.com/astral-sh/uv/blob/main/PIP_COMPATIBILITY....
- elxx 3y agoThis is definitely a paid ad. Check out the company cited in the post. It's a VC funded attempt to extend/embrace/extinguish the current tooling. [0] https://astral.sh/about https://astral.sh/about
- Ar-Curunir 3y agoYes, and Linux was an attempt to extend/embrace/extinguish Unix.
- stavros 3y agoYou can't embrace/extend/extinguish OSS with other OSS. I have my doubts about Astral, but the software they make is OSS.
- acdha 3y agoAstral is a company but that doesn’t make this an ad without evidence you didn’t include. Do you personally know that this was anything other than organic praise from a happy user?
- charliermarsh 3y agoHappy to report that this definitely isn't a paid ad. The Home Assistant team did this on their own. I helped out by building uv for some of the architectures they needed: https://github.com/astral-sh/uv/pull/2417 https://github.com/astral-sh/uv/pull/2417
- jchw 3y agoMuch like how people underestimate the time and effort it takes to build mature software from scratch, people also seem to under-estimate greatly the complexity and effort involved in trying to make huge improvements to existing tools, both technical and political in nature. There's absolutely no chance that the vision the Bun folks had could be carried out in the Node.JS project in my opinion, the resulting attempt would have less impressive results and probably would leave everyone unsatisfied. One of the biggest benefits and most powerful features of open source is undoubtedly the fork, but just as important is knowing when not to do that, and that's definitely the case when the position you want to be is more different from any existing starting point than it is from scratch; I'd argue writing a 'fast' Pip-compatible package manager from first-principles is a perfect case for that. Re: politics. Note that sometimes forks and competitors are so successful, they make it possible for the upstream to "unblock" by virtue of convincing them to adjust their vision. For Node.JS, consider io.js and Yarn in particular.
- AtlasBarfed 3y agoSo .. like ... Why not dump python entirely if you are doing a python to rust port of a core aspect of python tool gain?
- Ar-Curunir 3y agoChanging your build tool to another tool (designed to be a drop-in replacement) is very different from changing your entire codebase...
- thorncorona 3y agoexisting code? swapping build tools is extremely different
- darby_eight 3y agoThere are obvious answers.... Why do you ask?
- datascienced 3y agoWell because best tool for the job. There is no natural law that everything must be bootstrapped. Languages use LLVM for example.
- eternityforest 3y agoBootstrapping is such an odd concept. Like, something that's self propagating is in a sense not fully reproducible and holonomic, unless you maintain the original non-bootstrapped version which would add tons of work. A compiler bug could put a bug in the compiler and propagate itself indefinitely. An implementation in some other languages that has an implementation in C means you could theoretically start from ASM and rebuild.
- porphyra 3y agobecause writing python is a lot easier for most people than writing rust but changing from pip to uv is like a one liner?
- tedivm 3y ago
- AceJohnny2 3y agoEvery time I hear "new system faster than old system!", my first thought is: what features does the new system not implement, what corner-cases does it ignore? What are the trade-offs?
- AceJohnny2 3y agoRust, the language, has rather successfully worked towards and demonstrated that, in many areas, you CAN have your pie and eat it too. But that doesn't carry over to applications, where it must be demonstrated again.
- hsbauauvhabzb 3y agoI’m doubtful the fact that uv is written in rust is directly responsible for the speed increase, it’s probably just a better dependency determination algorthm. Stop talking about fancy pies, you don’t need it even if it smells nice. What you actually need is to eat healthy and nutritious foods.
- kbenson 3y agoFeels like a mixed message, because my natural inclination would be to think of Rust as healthy and nutritious foods (it takes dedication and consistency to stick with it) and a better dependency algorithm as a fancy pie, since it's a on-off nice gain that feels good, but doesn't necessarily lead to additional benefits later.
- fnordpiglet 3y agoHaving done a fair amount of rust porting I imagine it’s a mix a both. Rust really is remarkably faster than Python at everything. But these results are indicative of a better algorithm as well.
- sedatk 3y agoI guess, building the tool takes longer.
- klysm 3y ago
- porphyra 3y agoAstral is really doing God's work in the land of Python. Ruff is an incredible time saver.
- thethimble 3y agoIt’s also interesting now that they’ve taken over rye as well. Really hopeful they develop a unified solution between these projects.
- stavros 3y agoHow do they make money? Tooling of this quality isn't cheap.
- acdha 3y agoYeah, I love their work but I am concerned that the VC death-spiral will be painful. I wonder whether they will offer support plans or similar, except that past experience makes me cynically assume that most developers will be “$5? haha, no!” until pivoting to complain about whatever they end up doing to make payroll.
- jojobas 3y ago>Our plan is to provide paid services that are better and easier to use than the alternatives by integrating our open-source offerings directly. Our goal is for these services to be as impactful as Ruff itself — but you may choose not to use them. Either way, Ruff will remain free and open-source, just as it is today. I guess the answer for now is "they don't".
- charliermarsh 3y agoThank you so much.
- armada651 3y agoI'm going to ask a stupid question, but why not just keep the packages installed?
- laborcontract 3y agoI've spent an embarrassing amount of time trying to figure out why Docker refuses to use my cached layers 80-90% of the time, even though every dependency is properly pinned and build commands properly ordered. I said f-it, gave up and replaced pip with uv. It's nowhere near the speed of a cached build but it's a free 3 minutes.
- jrockway 3y agoI guess I don't have enough context to really understand this. It takes 1.5 hours to calculate what dependencies Home Assistant needs? Wouldn't you normally amortize that out for each top-level dependency you add to the project and write it to a lock file or something, so that builds just become fetching the relevant artifacts? Even then, I'm not sure how it would take 1.5 hours even on pen and paper.
- ffsm8 3y ago> Wouldn't you normally amortize that out for each top-level dependency you add to the project and write it to a lock file or something Plain pip doesn't really have a lockfile. You can generate a freeze file, but that's something different in practice. I haven't really been in the Python ecosystem in several years, but my surprise was mostly how there was no mention of Poetry or pipenv. Didn't they address that need before? Are they already out of favor?
- danmur 3y agoI don't think this is anything like poetry. The dependency resolution doesnt seem to handle much variation in arch or Python version or other variables I at least have to deal with. I generally use poetry for dependency management (only runs in dev) then use pip or venv-update to install in prod from a generated requirements.txt. uv looks like a good and fast replacement for pip or venv-update but those only take a fraction of a second to run (on a noop, my main use case) so I personally haven't seen where uv would fit in my workflow yet. Maybe building in containers from scratch each time, though again pip is performant enough for me when not doing the hard work of dependency resolution, and for some reason it's not cached. I am not sure why dependency resolution outside of dev or build systems would be done. But maybe I have escaped the loop.
- deleted 3y ago[deleted]
- outlore 3y agoRuff and uv have been a godsend for my workflow. I wrote a little about it here https://news.ycombinator.com/item?id=39851280 https://news.ycombinator.com/item?id=39851280
- singhrac 3y agoThe comments here betray a significant misunderstanding of both the Python ecosystem and open source work. Getting to a consensus is hard work. They have a vision for how Python packaging and tooling could be better (I assume cargo-like), and a roadmap for improving that. They could have tried to convince the pip maintainers their roadmap was best, but with large fragmented communities like Python it is impossible to reason about how the community will receive changes to tooling. It’s better to build something and get feedback to prove the hypothesis. The choice of Rust is just because they like the developer ergonomics in that language (I suspect a big reason is that they wanted to ship a single binary independent of any Python env). CPython is written in C because… Guido started to write it in C. Many comments here don’t understand that their competitors are actually NOT just pip but the conda ecosystem including micromamba (written in C++) which does not attempt to maintain pip interop at all. Conda realized that there are millions of Python programmers who are not developers and could not debug a GCC error message when pip installing scipy. There are thousands of Python packages that are basically tested on whatever version of Python and libraries the author had installed at the time they wrote it, and literally millions of Github issues that are something like “this doesn’t work because I installed version Y of package X and package Z requires < Y” all vaguely for solveable reasons. Many of these include C/C++ code that builds via custom setup.py scripts. If you haven’t lost days of work to solving stupid dependency issues in Python I’m not sure you can relate.