4 ms·
The real issue is having to keep multiple copies of CUDA on a system to satisfy all the different moving parts that each want different CUDA versions and CUDNN
by dheera 3y ago
The real issue is having to keep multiple copies of CUDA on a system to satisfy all the different moving parts that each want different CUDA versions and CUDNN versions, and those different CUDA installers fight over and uninstall each others' nvidia driver versions because CUDA is shipped with the nvidia driver, bundled and listed as an apt dependency.
My system changes from nvidia-525 to nvidia-535 to nvidia-520 to nvidia-515 on a daily basis because I need to reinstall a different CUDA version just to try some new paper's code.
PyTorch did it right, it now ships with its own CUDA and doesn't take a shit about version what you have in /usr/local. Everything else should do the same.
- kristjansson 3y agoDon’t install the drivers with CUDA? Don’t rely on distro-packaged CUDA?
- dheera 3y agoNVIDIA's official CUDA is packaged with their damn driver.
- cburdick13 3y agoHi, you can choose not to install the driver with a CUDA install, and to download the driver separately.
- dheera 3y agoOnly if you use the runfile, but that breaks yet other packages that specify 'cuda' as an apt dependency. Your 'cuda' packages should use '>=' not '==' e.g. 'cuda-12' should depend on nvidia>=525 NOT nvidia==525
- cburdick13 3y agoThat's a fair point. I'll look into it.
- cburdick13 3y agoHi, I looked into this and it seems that it is indeed using >= Depends: cuda-libraries-11-8 (>= 11.8.0), cuda-drivers (>= 520.61.05)
- dheera 3y agoHm no, what you actually have is cuda-libraries-11-8 depends on cuda-drivers and cuda-drivers depends on cuda-drivers-520 = 520.61.05-1 where you have the version number repeated in the name. That is a really bad pattern for things intended to be backward compatible, as multiple installs of cuda will overwrite "cuda-drivers" and each is pegged to a certain "cuda-drivers-X" where X is part of the name. This results in some really weird dependency hells if you try to install more than 1 CUDA version at the same time depending on the order you install them. $ sudo apt show cuda-drivers Package: cuda-drivers Version: 520.61.05-1 Priority: optional Section: multiverse/devel Maintainer: cudatools <cudatools@nvidia.com> Installed-Size: 7,168 B Depends: cuda-drivers-520 (= 520.61.05-1)
- the_svd_doctor 3y agoThe solution is to install the latest CUDA driver (which is always 100% backward compatible) but have multiple version of the libraries in /usr/local. Granted it can get messy with various package manager doing their own things.
- dheera 3y agoUsually the CUDA versions end up uninstalling other CUDA versions and drivers due to package manager issues. I wish nvidia would just release a `sudo apt install cuda-all` that just keeps you updated with ALL possible cuda versions simultaneously. I know it would be 20GB but that's fine, it's a drop in the bucket compared to the datasets I play with.
- cburdick13 3y agoIf you use the run files rather than the package manager files it's always installed in completely separate folders inside /use/local/cuda. What you're describing with an "all" install can somewhat be accomplished with containers right now and none of the dependency problems.
- dheera 3y agoBut containers, X forwarding, visualization, and OpenGL is a complete shitshow. I have apps that work outside of a container but freeze the UI that the mouse pointer won't even work when run from within a container.
- touisteur 3y ago+1 for runfiles but beware of the multi-gigabyte installer + installed footprint. Not complaining per-se, just plan for it.
- dheera 3y agoProblem with runfiles is I've run into other .deb packages that I'd like to use that are marked as dependent on 'cuda'. Using a runfile is ditching the package manager on a package-managed system, instead of using the package manager correctly.
- adev_ 3y ago> PyTorch did it right, it now ships with its own CUDA and doesn't take a shit about version what you have in /usr/local Is that even legal ? Up to my understanding Cuda is covered by an EULA license that explicitly require end user agreement.
- dheera 3y ago"Meh" who cares. It works.
- adev_ 3y ago> now ships with its own CUDA and doesn't take a shit about version Bundling system dependency in python software is always a giant shit show. I would not name that "Everything should do the same" Any other python library (randomly cuPy) that also ship its own Cuda, with its own different version, will segfault happily because you have duplicated symbols in a single process.
- dheera 3y agoSo randomize the symbols with appeneded UUIDs
- adev_ 3y agoWhat you just quote is more or less what the GNU glibc is doing with symbol versioning which is an other example of giant shit show and an ABI nightmare. And: (1) It is a solution only Nvidia can provide: Cuda is a binary. And they will certainly not do that just to please the python community. (2) That just create an other set of problems just because python packaging sucks in the first place. Lets just solve the initial problem, shall we ?