6 ms·
All python packaging challenges are solved. Lesson learned is that there is not a single solution for all problems. getting more strings attached with VC funded
by runningmike 1y ago
All python packaging challenges are solved. Lesson learned is that there is not a single solution for all problems. getting more strings attached with VC funded companies and leaning on their infrastructure is a high risk for any FOSS community.
- tempest_ 1y agoI share your concern but I have saved so much time with uv already that I figure ill ride it till the VC enshitification kills the host. Hopefully at the point the community is centralized enough to move in one direction.
- deleted 1y ago[deleted]
- NegativeLatency 1y agoI've been heartened by the progress that opentofu has made, so I think if it gets enough momentum it could survive the inevitable money grab
- anitil 1y agoI agree, now I just use uv and forget about it. It does use up a fair bit of disk, but disk is cheap and the bootstrapping time reduction makes working with python a pleasure again
- nemosaltat 1y agoCouldn’t agree more and the `uv run executable.sh` that contains a shebang, imports and then python is just magical.
- frickinLasers 1y agoIs that much different than the python inline script format? https://peps.python.org/pep-0723/ https://peps.python.org/pep-0723/
- alisonatwork 1y agoI recently did the same at work, just converted all our pip stuff to use uv pip but otherwise no changes to the venv/requirements.txt workflow and everything just got much faster - it's a no-brainer. But the increased resource usage is real. Now around 10% of our builds get OOM killed because the build container isn't provisioned big enough to handle uv's excessive memory usage. I've considered reducing the number of available threads to try throttle the non-deterministic allocation behavior, but that would presumably make it slower too, so instead we just click the re-run job button. Even with that manual intervention 10% of the time, it is so much faster than pip it's worth it.
- zanie 1y agoPlease open an issue with some details about the memory usage. We're happy to investigate and feedback on how it's working in production is always helpful. (I work on uv)
- alisonatwork 1y agoLast time I looked into this I found this unresolved issue, which is pretty much the same thing: https://github.com/astral-sh/uv/issues/7004 https://github.com/astral-sh/uv/issues/7004 We run on-prem k8s and do the pip install stage in a 2CPU/4GB Gitlab runner, which feels like it should be sufficient for the uv:python3.12-bookworm image. We have about 100 deps that aside from numpy/pandas/pyarrow are pretty lightweight. No GPU stuff. I tried 2CPU/8GB runners but it still OOMed occasionally so didn't seem worth using up those resources for the normal case. I don't know enough about the uv internals to understand why it's so expensive, but it feels counter-intuitive because the whole venv is "only" around 500MB.
- zanie 1y agoThanks that's helpful. Did you try reducing the concurrency limit?
- zzzeek 1y agosorry, I guess you're new here? Here, try this Kool Aid. I think it will help you fit in. oh don't mind that "MongoDB" logo on the glass that's old
- JonChesterfield 1y agoI've been dealing with python vs debian for the last three hours and am deeply angry with the ecosystem. Solved it is not. Debian decided you should use venv for everything. But when packages are installed in a venv, random cmake nonsense does not find them. There are apt-get level packages, some things find those, others do not. Names are not consistent. There's a thing called pipx which my console recommended for much the same experience. Also the vestiges of 2 vs 3 are still kicking around in the forms of refusing to find a package based on the number being present or absent. Whatever c++headerparser might be, I'm left very sure that hacking python out of the build tree and leaving it on the trash heap of history is the proper thing to do.
- nemomarx 1y agofrom what I hear uv is the "solved" and venv by hand is the old way
- ghshephard 1y agouv is venv + insanely fast pip. I’ve used it every day for 5+ months and I still stare in amazement every time I use it. It’s probably the most joy I’ve ever gotten out of technology.
- typpilol 1y agoInstalling packages it the most joy you've ever gotten outta tech? Not a project you built, or something you're proud of? Installing packages?
- computershit 1y ago> All python packaging challenges are solved. This comes across as uninformed at best and ignorant at worst. Python still doesn't have a reliable way to handle native dependencies across different platforms. pip and setuptools cannot be the end all be all of this packaging ecosystem nor should they be.
- _the_inflator 1y ago„across different platforms“ First things first: Import path, os I love Python, the ZEN of it, and you really need to accept the fact that there are conventions - quite a lot and that bash or shell scripts are where the magic happens, like environmental variables, if you know how to secure your app. Even the self thing finally makes sense after years of bewilderment (“Wait: not even Java is that brutal to its users.”) Lately stumbled over poetry after really getting the gist out of venv and pip. Still hesitant, because Windows doesn’t play a role.
- bastawhiz 1y agoWell I started with pip because it's what I was told to use. But it was slow and had footguns. And then I started using virtualenv, but that only solved part of the problem. So I switched to conda, which sometimes worked but wrecked my shell profile and often leads to things mysteriously using the wrong version of a package. So someone told me to use pipenv, which was great until it was abandoned and picked up by someone who routinely broke the latest published version. So someone told me to use poetry, but it became unusably slow. So I switched back to pip with the built-in venv, but now I have the and problems I had before, with fewer features. So I switched to uv, because it actually worked. But the dependency I need is built and packaged differently for different operating systems and flavor of GPU, and now my coworkers can't get the project to install on their laptops. I'm so glad all the Python packaging challenges are "solved"
- aledalgrande 1y agoMan I used python sparingly over the years and I still had to deal with all those package manager changes. Worse than the JS bundling almost?
- seriocomic 1y agoI've walked the same rocky path and have the bleeding feet to show for it! My problem is that now my packaging/environment mental model is so muddled I frequently mix up the commands...
- anothernewdude 1y ago
- benreesman 1y agoTry doing CUDA stuff. It's a chemical fire. And the money would make solving it would fund arbitrary largesse towards OSS in perpetuity.
- deleted 1y ago[deleted]
- chaostheory 1y agoNo. This is the only thing that python still doesn’t have just working. Otherwise there would be no excitement for anything new in this space.
- Imustaskforhelp 1y agoUv truly is great, and I mean they are open source and we can always fork it just as how valkey forked redis And also if you mean that pyx might be hosted on uv, well I think the discussion can go towards that pyx should be made open source but honestly, I am pretty sure that someone might look at pyx and create a pyx api compliant hosted server or I am still curious as to how pyx works and what it actually truly does.
- dirkc 1y agoI see VC money as an artificial force propping up a project. It is not bad per se, but VC money is not a constant and it leaves a big drop at the end. If there is a big enough community that has grown around the project, that drop might be okay.
- miraculixx 1y ago+1
- mbonnet 1y agoIf Python packaging problems are solved, why is Python known for having the worst tooling ecosystem of any "modern" language?