4 ms·

Would the authors not have been better off adding their performance improvements to pip rather than creating their own tool? I remember this being a complaint f

Would the authors not have been better off adding their performance improvements to pip rather than creating their own tool? I remember this being a complaint from Node maintainers when Bun was released: they had multiple open performance-related issues with no one available to work on them.

dmix 3y ago
Most innovations happens outside large organizations, then either replaces it or the ideas get stolen/absorbed into the parent. All the downsides and barriers of operating in a large legacy system are pretty well known, even if the maintainers are receptive, radical change will be DOA.
stouset 3y ago
I’m guessing “rewrite it in Rust” would have been an unpalatable PR for the pip maintainers to merge.
- dmix 3y ago
  Yeah "lets rewrite in Rust" has become a meme with negative connotations that will attract ridiculed or be ignored, but Rust clearly excels in this sort of tooling. Whether it makes sense to rewrite an existing codebase or pigeonhole it on the edges is another question.
- jonathankoren 3y ago
  Yeah but I suspect there’s something more going on than Rust magic. They’re claiming a savings of about an hour and a half. (4% of the original time) That sounds like an architecture change, or more likely a whole lot of caching.
  - 01HNNWZ0MV43FF 3y ago
    I guess the uv authors don't want to maintain a better algorithm in Python then
    - andrewjf 3y ago
      I don’t know how you can seriously imply that Python could be just as fast as Rust if the authors of uv cared enough. That seems like a disingenuous framing of why UV exists, let alone doesn’t pass the smell test that Python can just be as fast as a compiled, native machine code language. Python and Ruby and all that are just so god awful slow that it’s completely obvious that’s a rewrite in native code would have these sorts of benefits.
  - eyegor 3y ago
    If the first step of your build process is creating a new pip venv from scratch, it's entirely believable to me. Pip resolves packages sequentially, so there's a lot of low hanging fruit even without yoinking a state of the art constraint solver. As a concrete example, I have a raspberry pi 4 I use for "what's the most pathetic hardware a user might try". It takes 5-10 minutes to build up a fresh venv even when it has all of the packages cached. If I nuke the caches and make it download, it's 20+ minutes. For comparison a modern laptop takes ~1 minute.
    - fbdab103 3y ago
      Is that on a SD card? My expectation is that SD cards are garbage and any number of IO access patterns break down vs a SSD.
      - eyegor 3y ago
        I threw in a moderately fast USB drive (~90 mb/s after pi4 overhead) because I got sick of SD cards dying every 6-12 months. Pi4 is very speed limited. Even if you tried to use an ssd, I think the highest I've seen is around 250mb/s with an nvme drive that gets a few gb/s in a real computer. In theory it should get to like 500mb/s for sata limits.
  - jerf 3y ago
    Your assignment, should you choose to accept it, is to write a small, simple Python program. Maybe something that builds a list of five or six elements, runs a simple map over it, then converts it to a dict somehow. Something simple. Then run it through gdb and watch every single CPU opcode that is executed for the code you wrote. Not Python opcode; CPU. Print something first and trigger off of that or something, I'm not worried about Python startup costs here. Then do the same with a roughly equivalent compiled program. Doesn't have to be Rust, C or whatever you know would do just fine. Yes. You can get a big speed improvement over pure Python simply by moving it to a compiled language and making effectively no other changes. Python is slow. This is not a value judgment. It is a big mistake to think that this is somehow an emotional judgment about Python stemming from anger or hatred or something and that anyone talking about Python's speed issues are just haters or something. I like Python just fine. But you will understand how it is not a value judgment when you are running the Python program through gdb. It is simply an engineering reality that anyone using Python needs to understand about Python. You can get yourself into a lot of trouble not understanding this about Python. Is there more things going on than a straight port? Possibly. But I don't find the difference unbelievable for it to be coming from a straight port with no major changes.
  - jejdirhrjeij 3y ago
    Converting python code to rust for data processing can get you 20x perf increase (single threaded).
rtpg 3y ago
It's tricky to change up some parts of pip because it might break some existing users flows. It's really hard to test for this kind of infrastructure tool because it's so environment-dependent. The rewrite lets there be a clean slate. Meanwhile the target of being "compatible" with pip gives them a good idea of where to go, and they can focus a bit more on testing given the lessons learned.
swyx 3y ago
i think now their current positioning is that the competition actually helped demonstrate the value of some kinds of speed, and that finally got those performance issues prioritized enough to work on by people with the right knowledge. Node doesn't lack for people who want to contribute, its more about people who -can- contribute well, and yes as other comment noted, its often to easier to try things out outside the system rather than work within it.
paxys 3y ago
uv isn't a performance improvement on pip but a ground-up rewrite in a completely different language.
- nojito 3y ago
  Source on the logic being different from pip? Last I checked it's nothing more than a port of pip to rust.
  - btheunissen 3y ago
    I’m unclear on how you “port” a library from Python to Rust, it’s a ground up rewrite with a familiar interface to pip but fundamentally doesn’t share any code.
    - gpm 3y ago
      It's actually pretty easy to call python from rust and rust from python. An incremental port of a python library into rust would definitely be possible. I'm not sure it would be a good idea. Python patterns don't necessarily make good rust patterns. Even ignoring larger code structure (e.g. single ownership*) you'd practically have to do a second pass to stop using python compatible types (except in the exposed interface) if you wanted an idiomatic rust library at the end of it. But it's definitely possible. * Which you don't "have to" do in rust, you can use `Rc<RefCell>` everywhere and imitate python instead. It's just throwing some of the advantages of rust and adding some noise to type signatures.
  - rovr138 3y ago
    It is not. > Limitations > While uv supports a large subset of the pip interface, it does not support the entire feature set. In some cases, those differences are intentional; in others, they're a result of uv's early stage of development. > For details, see our pip compatibility guide[1] > Like pip-compile, uv generates a platform-specific requirements.txt file (unlike, e.g., poetry and pdm, which generate platform-agnostic poetry.lock and pdm.lock files). As such, uv's requirements.txt files may not be portable across platforms and Python versions. [1] - https://github.com/astral-sh/uv/blob/main/PIP_COMPATIBILITY.md https://github.com/astral-sh/uv/blob/main/PIP_COMPATIBILITY....
elxx 3y ago
This is definitely a paid ad. Check out the company cited in the post. It's a VC funded attempt to extend/embrace/extinguish the current tooling. [0] https://astral.sh/about https://astral.sh/about
- Ar-Curunir 3y ago
  Yes, and Linux was an attempt to extend/embrace/extinguish Unix.
- stavros 3y ago
  You can't embrace/extend/extinguish OSS with other OSS. I have my doubts about Astral, but the software they make is OSS.
- acdha 3y ago
  Astral is a company but that doesn’t make this an ad without evidence you didn’t include. Do you personally know that this was anything other than organic praise from a happy user?
- charliermarsh 3y ago
  Happy to report that this definitely isn't a paid ad. The Home Assistant team did this on their own. I helped out by building uv for some of the architectures they needed: https://github.com/astral-sh/uv/pull/2417 https://github.com/astral-sh/uv/pull/2417
jchw 3y ago
Much like how people underestimate the time and effort it takes to build mature software from scratch, people also seem to under-estimate greatly the complexity and effort involved in trying to make huge improvements to existing tools, both technical and political in nature. There's absolutely no chance that the vision the Bun folks had could be carried out in the Node.JS project in my opinion, the resulting attempt would have less impressive results and probably would leave everyone unsatisfied. One of the biggest benefits and most powerful features of open source is undoubtedly the fork, but just as important is knowing when not to do that, and that's definitely the case when the position you want to be is more different from any existing starting point than it is from scratch; I'd argue writing a 'fast' Pip-compatible package manager from first-principles is a perfect case for that. Re: politics. Note that sometimes forks and competitors are so successful, they make it possible for the upstream to "unblock" by virtue of convincing them to adjust their vision. For Node.JS, consider io.js and Yarn in particular.

dmix 3y ago

Most innovations happens outside large organizations, then either replaces it or the ideas get stolen/absorbed into the parent. All the downsides and barriers of operating in a large legacy system are pretty well known, even if the maintainers are receptive, radical change will be DOA.

stouset 3y ago

I’m guessing “rewrite it in Rust” would have been an unpalatable PR for the pip maintainers to merge.

dmix 3y ago
Yeah "lets rewrite in Rust" has become a meme with negative connotations that will attract ridiculed or be ignored, but Rust clearly excels in this sort of tooling. Whether it makes sense to rewrite an existing codebase or pigeonhole it on the edges is another question.
jonathankoren 3y ago
Yeah but I suspect there’s something more going on than Rust magic. They’re claiming a savings of about an hour and a half. (4% of the original time) That sounds like an architecture change, or more likely a whole lot of caching.
- 01HNNWZ0MV43FF 3y ago
  I guess the uv authors don't want to maintain a better algorithm in Python then
  - andrewjf 3y ago
    I don’t know how you can seriously imply that Python could be just as fast as Rust if the authors of uv cared enough. That seems like a disingenuous framing of why UV exists, let alone doesn’t pass the smell test that Python can just be as fast as a compiled, native machine code language. Python and Ruby and all that are just so god awful slow that it’s completely obvious that’s a rewrite in native code would have these sorts of benefits.
- eyegor 3y ago
  If the first step of your build process is creating a new pip venv from scratch, it's entirely believable to me. Pip resolves packages sequentially, so there's a lot of low hanging fruit even without yoinking a state of the art constraint solver. As a concrete example, I have a raspberry pi 4 I use for "what's the most pathetic hardware a user might try". It takes 5-10 minutes to build up a fresh venv even when it has all of the packages cached. If I nuke the caches and make it download, it's 20+ minutes. For comparison a modern laptop takes ~1 minute.
  - fbdab103 3y ago
    Is that on a SD card? My expectation is that SD cards are garbage and any number of IO access patterns break down vs a SSD.
    - eyegor 3y ago
      I threw in a moderately fast USB drive (~90 mb/s after pi4 overhead) because I got sick of SD cards dying every 6-12 months. Pi4 is very speed limited. Even if you tried to use an ssd, I think the highest I've seen is around 250mb/s with an nvme drive that gets a few gb/s in a real computer. In theory it should get to like 500mb/s for sata limits.
- jerf 3y ago
  Your assignment, should you choose to accept it, is to write a small, simple Python program. Maybe something that builds a list of five or six elements, runs a simple map over it, then converts it to a dict somehow. Something simple. Then run it through gdb and watch every single CPU opcode that is executed for the code you wrote. Not Python opcode; CPU. Print something first and trigger off of that or something, I'm not worried about Python startup costs here. Then do the same with a roughly equivalent compiled program. Doesn't have to be Rust, C or whatever you know would do just fine. Yes. You can get a big speed improvement over pure Python simply by moving it to a compiled language and making effectively no other changes. Python is slow. This is not a value judgment. It is a big mistake to think that this is somehow an emotional judgment about Python stemming from anger or hatred or something and that anyone talking about Python's speed issues are just haters or something. I like Python just fine. But you will understand how it is not a value judgment when you are running the Python program through gdb. It is simply an engineering reality that anyone using Python needs to understand about Python. You can get yourself into a lot of trouble not understanding this about Python. Is there more things going on than a straight port? Possibly. But I don't find the difference unbelievable for it to be coming from a straight port with no major changes.
- jejdirhrjeij 3y ago
  Converting python code to rust for data processing can get you 20x perf increase (single threaded).

rtpg 3y ago

It's tricky to change up some parts of pip because it might break some existing users flows. It's really hard to test for this kind of infrastructure tool because it's so environment-dependent. The rewrite lets there be a clean slate. Meanwhile the target of being "compatible" with pip gives them a good idea of where to go, and they can focus a bit more on testing given the lessons learned.

swyx 3y ago

i think now their current positioning is that the competition actually helped demonstrate the value of some kinds of speed, and that finally got those performance issues prioritized enough to work on by people with the right knowledge. Node doesn't lack for people who want to contribute, its more about people who -can- contribute well, and yes as other comment noted, its often to easier to try things out outside the system rather than work within it.

paxys 3y ago

uv isn't a performance improvement on pip but a ground-up rewrite in a completely different language.

nojito 3y ago
Source on the logic being different from pip? Last I checked it's nothing more than a port of pip to rust.
- btheunissen 3y ago
  I’m unclear on how you “port” a library from Python to Rust, it’s a ground up rewrite with a familiar interface to pip but fundamentally doesn’t share any code.
  - gpm 3y ago
    It's actually pretty easy to call python from rust and rust from python. An incremental port of a python library into rust would definitely be possible. I'm not sure it would be a good idea. Python patterns don't necessarily make good rust patterns. Even ignoring larger code structure (e.g. single ownership*) you'd practically have to do a second pass to stop using python compatible types (except in the exposed interface) if you wanted an idiomatic rust library at the end of it. But it's definitely possible. * Which you don't "have to" do in rust, you can use `Rc<RefCell>` everywhere and imitate python instead. It's just throwing some of the advantages of rust and adding some noise to type signatures.
- rovr138 3y ago
  It is not. > Limitations > While uv supports a large subset of the pip interface, it does not support the entire feature set. In some cases, those differences are intentional; in others, they're a result of uv's early stage of development. > For details, see our pip compatibility guide[1] > Like pip-compile, uv generates a platform-specific requirements.txt file (unlike, e.g., poetry and pdm, which generate platform-agnostic poetry.lock and pdm.lock files). As such, uv's requirements.txt files may not be portable across platforms and Python versions. [1] - https://github.com/astral-sh/uv/blob/main/PIP_COMPATIBILITY.md https://github.com/astral-sh/uv/blob/main/PIP_COMPATIBILITY....

elxx 3y ago

This is definitely a paid ad. Check out the company cited in the post. It's a VC funded attempt to extend/embrace/extinguish the current tooling. [0] https://astral.sh/about https://astral.sh/about

Ar-Curunir 3y ago
Yes, and Linux was an attempt to extend/embrace/extinguish Unix.
stavros 3y ago
You can't embrace/extend/extinguish OSS with other OSS. I have my doubts about Astral, but the software they make is OSS.
acdha 3y ago
Astral is a company but that doesn’t make this an ad without evidence you didn’t include. Do you personally know that this was anything other than organic praise from a happy user?
charliermarsh 3y ago
Happy to report that this definitely isn't a paid ad. The Home Assistant team did this on their own. I helped out by building uv for some of the architectures they needed: https://github.com/astral-sh/uv/pull/2417 https://github.com/astral-sh/uv/pull/2417

jchw 3y ago

Much like how people underestimate the time and effort it takes to build mature software from scratch, people also seem to under-estimate greatly the complexity and effort involved in trying to make huge improvements to existing tools, both technical and political in nature. There's absolutely no chance that the vision the Bun folks had could be carried out in the Node.JS project in my opinion, the resulting attempt would have less impressive results and probably would leave everyone unsatisfied. One of the biggest benefits and most powerful features of open source is undoubtedly the fork, but just as important is knowing when not to do that, and that's definitely the case when the position you want to be is more different from any existing starting point than it is from scratch; I'd argue writing a 'fast' Pip-compatible package manager from first-principles is a perfect case for that. Re: politics. Note that sometimes forks and competitors are so successful, they make it possible for the upstream to "unblock" by virtue of convincing them to adjust their vision. For Node.JS, consider io.js and Yarn in particular.