4 ms·

This is especially true when you use a lot of tooling. I love jupyter, but installing it in a venv means pulling a lot of deps which will affect a lot what I ca

This is especially true when you use a lot of tooling. I love jupyter, but installing it in a venv means pulling a lot of deps which will affect a lot what I can install. Fortunately the Python community is much more serious about making deps that work together than the JS community, and the fact it works at all given the cartesian products of all the python modules is kind of a miracle and a testament to that. Unfortunately, that's a problem that is unlikely to be solved in the next decade, so we all live with it. The reverse problem is true for JS, and I see many projects shipping very heavy frontend code because despite all the tree shaking, they embed 5 times the same module with different versions in their bundle. That's one of the reasons for the bloated page epidemic. I guess it's a trade-off for all scripting languages: choosing between bloat or compat problem. Rust and Go don't care as much, and on top of that they can import code from 10 years ago and it sill works. However, and while I do know how hard it is to ship python code to the end user (at least if you don't use a web app), I don't think the version problem is the reason. We have zipapp and they work fine. No the main reason iscompiled extensions are very useful and popular, which means packaging is solving more than packaging python, but a ton of compiled languages at one. Take scipy: they have c, pascal and assembly in there. This can and will be improved though. In fact, thanks to wheels and indygreg/python-build-standalone, I think we will see a solution to this in the coming years. I'm even betting on astral to providing it.

dheera 2y ago
My ideal situation is that the system should maintain authoritative versions of every package and version that is ever requested, and they should not need to be shipped. Multiple versions of a package should coexist. /usr/lib/python3.12/torch/2.1.0/ /usr/lib/python3.12/torch/2.1.1/ /usr/lib/python3.12/torch/2.1.2/ When a package requests 2.1.1 it fetches it right out of there, installing from PyPI if it doesn't. The same should be true of JS and even C++. When a C++ app's deb package wants libusb==1.0.1 it should NOT overwrite libusb-1.0.0 that is on the system, it should coexist with it and link to the correct one so that another app that wants libusb-1.0.0 should still be able to use it. > Fortunately the Python community is much more serious about making deps that work together This is very not true at least in ML. I have to create a new conda environment for almost every ML paper that comes out. There are so many papers and code repos I test every week that refuse to work with the latest PyTorch, and some that require torch<2.0 or some bull. Also, xformers, apex, pytorch3d, and a number of other popular packages require that the cuda version that is included with the "torch" Python package matches the cuda version in /usr/local/cuda AND that your "CC" and "CXX" variables point to gcc-11 (NOT gcc-12), or else the pip install will fail. It's a fucking mess. Why can't gcc-12 compile gcc-11 code without complaining? Why does a Python package not ship binaries of all C/C++ parts for all common architectures compiled on a build farm?
- ericjmorey 2y ago
  ML researchers might be thinking that their paper will be obsolete next month so why bother taking time to make their coding environment reproducible.
  - setopt 2y ago
    It’s not the researcher’s fault if the libraries they use make breaking changes after a month; proof-of-concept code published with a paper is supposed to be static, and there’s often no incentive for the researcher to maintain it after publication. At this point, venvs are the best workaround, but we can still wish for something better. As someone commented further up, being able to “import pytorch==2.0” and have multiple library versions coexist would go a long way.
- BiteCode_dev 2y ago
  I'm assuming by system you mean OS, which is a terrible, terrible idea. Dev stack and system libs should not coexist, especially because system libs should be vetted by the OS vendor, but you can't ask them to do that for dev libs. > I have to create a new conda environment for almost every ML paper that comes out That's how it's supposed to work: one env per project. As for the rest, it's more telling about the C/C++ community building the things bellow the python wrappers.
  - dheera 2y ago
    > one env per project That causes 50 copies of the exact same version of a 1GB library to exist on my system that are all obtained from the same authority (PyPI). I have literally 50 copies of the entire set of CUDA libraries because every conda environment installs PyTorch and PyTorch includes its own CUDA. I'm not asking the OS to maintain this, but rather the package manager ("npm" or "pip" or similar) should do so on a system-wide basis. "python" and "pip" should allow for 1 copy per officially-released version of each package to live on the system, and multiple officially-released version numbers to coexist in /usr/lib. If a dev version is being used or any version that deviates from what is on PyPI, then that should live within the project.
ericjmorey 2y ago
I'm hopeful the uv will bring us closer to tooling on par with other language ecosystems. But it's very early on in the process.
- BiteCode_dev 2y ago
  Given the track record they got, I'm confident they will. But what I really hope is that they'll tackle the user app shipping problem eventually.
skeledrew 2y ago
I install most tooling, including Jupyter, using pipx. The only thing I then need to install in the project venvs is ipykernel (which I add as a dev dep), and then create a kernel config that allows Jupyter to be run using that venv.

dheera 2y ago

My ideal situation is that the system should maintain authoritative versions of every package and version that is ever requested, and they should not need to be shipped. Multiple versions of a package should coexist. /usr/lib/python3.12/torch/2.1.0/ /usr/lib/python3.12/torch/2.1.1/ /usr/lib/python3.12/torch/2.1.2/ When a package requests 2.1.1 it fetches it right out of there, installing from PyPI if it doesn't. The same should be true of JS and even C++. When a C++ app's deb package wants libusb==1.0.1 it should NOT overwrite libusb-1.0.0 that is on the system, it should coexist with it and link to the correct one so that another app that wants libusb-1.0.0 should still be able to use it. > Fortunately the Python community is much more serious about making deps that work together This is very not true at least in ML. I have to create a new conda environment for almost every ML paper that comes out. There are so many papers and code repos I test every week that refuse to work with the latest PyTorch, and some that require torch<2.0 or some bull. Also, xformers, apex, pytorch3d, and a number of other popular packages require that the cuda version that is included with the "torch" Python package matches the cuda version in /usr/local/cuda AND that your "CC" and "CXX" variables point to gcc-11 (NOT gcc-12), or else the pip install will fail. It's a fucking mess. Why can't gcc-12 compile gcc-11 code without complaining? Why does a Python package not ship binaries of all C/C++ parts for all common architectures compiled on a build farm?

ericjmorey 2y ago
ML researchers might be thinking that their paper will be obsolete next month so why bother taking time to make their coding environment reproducible.
- setopt 2y ago
  It’s not the researcher’s fault if the libraries they use make breaking changes after a month; proof-of-concept code published with a paper is supposed to be static, and there’s often no incentive for the researcher to maintain it after publication. At this point, venvs are the best workaround, but we can still wish for something better. As someone commented further up, being able to “import pytorch==2.0” and have multiple library versions coexist would go a long way.
BiteCode_dev 2y ago
I'm assuming by system you mean OS, which is a terrible, terrible idea. Dev stack and system libs should not coexist, especially because system libs should be vetted by the OS vendor, but you can't ask them to do that for dev libs. > I have to create a new conda environment for almost every ML paper that comes out That's how it's supposed to work: one env per project. As for the rest, it's more telling about the C/C++ community building the things bellow the python wrappers.
- dheera 2y ago
  > one env per project That causes 50 copies of the exact same version of a 1GB library to exist on my system that are all obtained from the same authority (PyPI). I have literally 50 copies of the entire set of CUDA libraries because every conda environment installs PyTorch and PyTorch includes its own CUDA. I'm not asking the OS to maintain this, but rather the package manager ("npm" or "pip" or similar) should do so on a system-wide basis. "python" and "pip" should allow for 1 copy per officially-released version of each package to live on the system, and multiple officially-released version numbers to coexist in /usr/lib. If a dev version is being used or any version that deviates from what is on PyPI, then that should live within the project.

ericjmorey 2y ago

I'm hopeful the uv will bring us closer to tooling on par with other language ecosystems. But it's very early on in the process.

BiteCode_dev 2y ago
Given the track record they got, I'm confident they will. But what I really hope is that they'll tackle the user app shipping problem eventually.

skeledrew 2y ago

I install most tooling, including Jupyter, using pipx. The only thing I then need to install in the project venvs is ipykernel (which I add as a dev dep), and then create a kernel config that allows Jupyter to be run using that venv.