11 ms·
GPU Headaches: Notes on Installing CUDA, CuDNN and Tensorflow on Manjaro
- innocenat 8y agoAdd Bumblebee to the mix and you get even more headache. Argh.
- illumin8 8y agoSo basically, your calculations for deciding whether purchasing physical GPUs or using cloud GPUs is more cost effective should add $5,000 (50 hours at $100 an hour seems reasonable) onto the cost of any physical rig, and assume you have a spare 50 hours of opportunity cost to set it up. All the ML researchers lusting for bare metal need to decide if they want to be "Gentoo Ricers" - https://funroll-loops.teurasporsaat.org/ https://funroll-loops.teurasporsaat.org/ - or if they just want to train models and get real work done.
- _Wintermute 8y ago50 hours is definitely an outlier. I'm pretty clueless when it comes to linux and drivers, and I managed to get an nvidia GPU working on ubuntu in about an hour or two of swearing and a dozen reboots.
- tntn 8y agoOr assume you aren't an outlier and can do it in 0.5-3 hours.
- swebs 8y agoWhy spend 50 hours figuring it out from scratch again if the guy in the article already told you exactly what you need to do?
- danieldk 8y agoExcept that I set up CUDA and Tensorflow in less than an hour. Admittedly, this is on Ubuntu (my colleagues don't like wild distro experiments), which may be easier. I compile every Tensorflow version (to compile with march=native). It is usually uneventful and takes a couple of minutes of my time. By the way, a Tesla card also costs > 5000 Euro and then add the cost of a machine with enough fast cores, memory, and disk space and you are north of 25000.
- tntn 8y agoThe author says they chose Manjaro because they couldn't get Ubuntu to work.
- p1esk 8y agoI've just installed it all from scratch on Ubuntu 16 in about 20 minutes. Just followed the instructions on TF website exactly. Didn't compile TF though, don't see the need.
- tntn 8y agoI know; see my other comments on the thread. I was responding directly to "Ubuntu may be easier" to indicate that the distro difference is not the cause of the authors problems. They've clearly got a messed up system or install procedure.
- perfinion 8y agoThere's definitely a need its waaay faster with -march=native. https://blog.perfinion.com/2018/07/tensorflow-cpu-supports-instructions/ https://blog.perfinion.com/2018/07/tensorflow-cpu-supports-i...
- p1esk 8y agoDid I miss it, or have you not compared GPU TF binary to GPU TF source? I thought that was the goal of your blog post as declared in the first paragraph...
- scottlegrand2 8y agoIn my experience, the same mindset that cannot figure out clear directions on how to set up a GPU machine is similarly incompetent at cleaning up data and understanding machine and deep learning models sufficiently to produce useful results. For bonus points, many of them seem to look down on those that can as "the help" or as "Ops." That said, configuring a GPU machine is (still) ridiculously complicated IMO. It's just that I've been doing it for almost 20 years.
- leblancfg 8y agoAuthor here. Good point, but 50 hours seems like an outlier for sure: I'm no expert whatsoever when it comes to debugging hardware in Linux. Assuming setup time across all first-time builds is Poisson-distributed, feels like the peak of that curve is more like 2-3 hours. Certainly something to keep in mind if considering doing it, though. Thoughts: * If I was setting up multiple identical machines, I'd only need to solve the issue once. * Details matter: Switching from VGA to HDMI connector made a world of difference. Me not noticing that earlier cost me ~2 days of trial-and-error to go to waste.
- tebruno99 8y agoYeah, not sure why you would be in Grub for Nvidia graphics drivers. This whole thing seems pretty odd to me since I've not seen a Nvidia driver problem in several years. Was he trying to use like Ubuntu 12.04 or something. Manjaro is a great choice and it does "just work" the best. All the others work just fine though.
- jamesblonde 8y agoI have had issues with centos 7.2 and nvidia drivers. Nvidia assume linux kernel sources are installed, but centos 7.2 puts them in a directory that the nvidia driver can't find. I ended up having to scp sources from a working machine and add symbolics links to get the drivers working (1080Ti card - i assume the Tesla drivers work better).
- tebruno99 8y agoCentOS is based on a stack that is nearing 5 years old so that makes sense. The driver world (even mesa itself) has matured greatly since then. Mesa and Nvidia even have libraries now that help them coexist instead of replacing/overwriting library files.
- ericd 8y agoI've seen some issues with some of the more recent Nvidia drivers+newer kernels which caused a login loop on Ubuntu 16.04.
- jorgemf 8y agoArchlinux user here (Manjaro is based on Archlinux). TensorFlow with GPU support is a supported package you can install it with all it's dependencies in a simple command with Pacman. But you don't want to do that the same way you don't want to compile TensorFlow. It is better to use an official docker image because they include all the dependencies, even the optionals as nccl 2.0. You forgot to install it, and it is speed up if you use more than one GPU. Just use the official docker image and forget all the burdensome of configuring everything. P.S. well you need to install docker with Nvidia support, but you can use it in the future for other things. This is also a couple of packages in Pacman (maybe Nvidia docker is in air, I don't remember now).
- leblancfg 8y agoThat's a fair point! I tried installing the pre-packaged conda Tensorflow package, but it was expecting CUDA 9.0 and CuDNN 7.0. The ones I had finally managed to install were 9.2 and 7.2... hence compiling it from source. I should add that as an addendum to the article, though, great catch!
- perfinion 8y agoWow that's quite a journey. I had a similar one when I started maintaining TensorFlow on Gentoo Linux. It's working pretty well now, emerge tensorflow will have everything installed and working with no fuss.
- acoye 8y agoDo you want to know pain? Try to use OpenCL with a GCN1 GPU (AMD 7970) with TF. You can more or less do it with SYCL / ComputeCPP, which is great, but it is not trivial either. It does require you to compile TF and may fail to operate some operations. Full disclosure I have not retried it recently, last time was 6 month aago, and a great amount of work is done on the SYCL / ComputeCPP front.
- acoye 8y ago… Which is somewhat sad, given AMD's open drivers are terrific and a pleasure to use on my machine, even for Steam games.
- misterbrian88 8y agoI spent a long time trying to set this up as well. Ultimately, I was successful using nvidia-docker. Here’s how I did it http://briancaffey.github.io/2017/11/19/tensorflow-gpu-setup-with-docker-on-arch-linux.html http://briancaffey.github.io/2017/11/19/tensorflow-gpu-setup...
- mippie_moe 8y agoMy company (Lambda Labs) made it possible to install cuda, cudnn, tensorflow, pytorch, and other deep learning frameworks with one line: https://lambdalabs.com/lambda-stack-deep-learning-software https://lambdalabs.com/lambda-stack-deep-learning-software We wrote debian packages for every framework, including cuda and cudnn. Using our Debian repository, you can install all these frameworks using apt/aptitude. When a new version of a framework comes out, we usually have it available in 1-2 weeks.
- alfalfasprout 8y agoAny chance you can get MxNet (and Keras MxNet) in there? Installing this stuff is a huge nuisance for us and we have some pretty insane Dockerfiles to handle all the different combinations. I might look into using this for our ML images.
- mippie_moe 8y agoI’ll talk to the team!
- vokep 8y ago...And then you take all my passwords right? Or mine some btc quietly in the background? Seriously this is too cool and nice, theres no way you're so cool and nice, such people just don't exist! (really this is awesome, thank you)
- mippie_moe 8y agoThanks for the positive feedback! Our company used to run a cluster of ~1000 gpu servers for inference and training. We used caffe and torch. Provisioning and software maintenance was a huge hassle. We realized how painful it is to get a machine set up for deep learning, so we decided to release our debian packages to the public. We sell computers built for deep learning researchers and figure the more people we can help get started, the better :)
- acd 8y agoIt is usually headaches to install closed source software on Linux. Ffmpeg with Cuda hardware acceleration encoding support is also a pain in the... to install. If graphic card makers would open source their drivers it would come default shipped apt-get,pacman,dnf,yum installable by the Linux distros.
- romaniv 8y agoI wonder when we will get to the point where computers (end-user computers) learn using ML techniques. Will we ever? Will it ever become accessible to non-developers?
- solomatov 8y agoThe best way to setup tensorflow for gpu is to use nvidia-docker and pre-built docker images. They mention it in their documentation.
- leblancfg 8y agoAuthor here. You would still need working NVIDIA drivers and CUDA to use nvidia-docker, which I couldn't manage to do.
- nextos 8y agoThe only two distributions I've tried where this worked with a one liner were NixOS and Arch. I'm surprised this doesn't work in Manjaro given that it's an Arch derivative. But I don't see much point in Arch derivatives as they do away with Arch's selling point. No layers between you and upstream.
- leblancfg 8y agoI see what you mean, and I agree with you in theory. The long learning curve and installation time for Arch was a non-starter for me though. Back to your point, Manjaro runs directly on pacman repos, not a layer between it and upstream. It just comes with IMO sane defaults and an installer GUI.
- foepys 8y agoBut don't expect support from the Arch Linux forums. They really, really don't like Manjaro as it attracts people that shouldn't be running Arch in the first place. Arch Linux is not for beginners.
- nextos 8y agoWhen did you try Arch? With the adoption of systemd it's incredibly easy to configure if you aim for a simple no desktop enviroment setup or something minimal. If not, I think it doesn't make any sense to use it unless you are a seasoned user. Rolling release will make running big frameworks like KDE tricky, as things will change often. And unless you have set everything up yourself, you won't know where to look in order to fix things. I'd recommend you look into NixOS if you want to run a desktop environment. It's radically different (totally functional and declarative, whereas Arch is imperative). All Python and deep learning stuff is neatly packaged and tested. And rollbacks are trivial. There's no state. You can even run Nix in other distros or macOS.
- beojan 8y ago> Rolling release will make running big frameworks like KDE tricky, as things will change often. On the contrary, you get new releases (with bugfixes) much sooner.
- gue5t 8y agoexport LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda/extras/CUPTI/lib64 Reboot and cross your fingers. Rebooting will terminate the process tree in which the environment has been modified, making this line a no-op. Please take the time to understand the commands you run before suggesting them to others.
- fooker 8y agoNow I see why the author had so much trouble.