4 ms·

10 years ago I was working at a place that was building Python systems that had dependency graphs too complicated for pip to handle. I was able to solve the pr

10 years ago I was working at a place that was building Python systems that had dependency graphs too complicated for pip to handle. I was able to solve the problem for my system with a "wheelhouse" system that could compute a list of wheels that could be installed to build it but the confidence of my team in Python had flagged. I had a sheaf of notes about the problem and figured out the math to build a proper dependency resolver for Python and tested out a lot of ideas such as being able to use http range requests to get the metadata out of wheels on PyPi without having to download the whole wheel. The problem I had no solution for though was "how to stop developers from trashing the environment that the dependency manager runs in." The data scientists I worked with had an astonishing target for wrecking anything at all. Myself I would have my poetry's environment got bad for reasons I didn't understand every few months ago. I also found the Python community just didn't care that pip didn't really work right. The most seductive form of blub is "I can accept using things that fail intermittently." I got a job coding Java and Javascript and never built the package manager. Then uv came along and managed to sell itself as "crazy fast" which did connect with people more than "correct". Written in rust, uv would have beaten my system in the fast department, and since it is a binary, there is no way anyone can screw up a Python it depends on -- as I see it, both technical and marketing genius!

robertlagrant 10d ago
Before uv came along, pipenv, Poetry, and (much older) Conda all were trying to solve this problem. It's a huge problem for Python that not all languages experience, because Python packages can contain all sorts. At one point if you wanted to install Scipy you had to drag in (and compile, if I remember correctly!) Fortran, of all things[0]. [0] https://stackoverflow.com/a/14822245/61938 https://stackoverflow.com/a/14822245/61938
- PaulHoule 10d ago
  Well Java has a kind of xenophobia that really resists bringing in foreign code but Python is at it's best when it accesses wrappers around C and Fortran. I used conda back in that period, it had a correct solver, and it was easy to manage my own packages, but it was slow in the technical sense of "it takes forever to build an environment" and slow in the business sense in that you got something curated which was not always the greatest or the latest but would, back in the day, "just work." Actually you could vendorize any software you need and have your own conda wheels, like I made wheels with different versions of CUDA drivers which are just DLLs so you could be running models with two versions of tensorflow that required two different versions of CUDA and never have to touch the NVIDIA installer. But today it is a more "just works" experience to use PyPi instead of conda so I don't use conda. Poetry was a big improvement over pip but I don't believe the resolver was 100% correct (like from looking at the source code) and performance was not that good, not so much because it was written in Python but because it did not have a proper cache, did not exploit concurrency. The Python way would be to use a world class SMT solver for the CPU intensive bit but when the bits hit the bus Rust is better at exploiting concurrency. So I am happy to have uv.

robertlagrant 10d ago

Before uv came along, pipenv, Poetry, and (much older) Conda all were trying to solve this problem. It's a huge problem for Python that not all languages experience, because Python packages can contain all sorts. At one point if you wanted to install Scipy you had to drag in (and compile, if I remember correctly!) Fortran, of all things[0]. [0] https://stackoverflow.com/a/14822245/61938 https://stackoverflow.com/a/14822245/61938

PaulHoule 10d ago
Well Java has a kind of xenophobia that really resists bringing in foreign code but Python is at it's best when it accesses wrappers around C and Fortran. I used conda back in that period, it had a correct solver, and it was easy to manage my own packages, but it was slow in the technical sense of "it takes forever to build an environment" and slow in the business sense in that you got something curated which was not always the greatest or the latest but would, back in the day, "just work." Actually you could vendorize any software you need and have your own conda wheels, like I made wheels with different versions of CUDA drivers which are just DLLs so you could be running models with two versions of tensorflow that required two different versions of CUDA and never have to touch the NVIDIA installer. But today it is a more "just works" experience to use PyPi instead of conda so I don't use conda. Poetry was a big improvement over pip but I don't believe the resolver was 100% correct (like from looking at the source code) and performance was not that good, not so much because it was written in Python but because it did not have a proper cache, did not exploit concurrency. The Python way would be to use a world class SMT solver for the CPU intensive bit but when the bits hit the bus Rust is better at exploiting concurrency. So I am happy to have uv.