3 ms·
So many details that people gloss over. I have used tensorflow (TF) and it is true that GPUs suck at interference at it. But it's not always the GPUs fault - T
by alexnewman 9y ago
So many details that people gloss over. I have used tensorflow (TF) and it is true that GPUs suck at interference at it. But it's not always the GPUs fault
- TF can't do anything quantized on GPUs. It just switches back to to the CPU/TPU.
- TF gets relatively poor utilization of the GPU and tends to not be careful with memory use.
- I was able to do certain types of classification hundreds of times faster by seeing what TF was doing it and hand writing it in OCL. Using https://docs.rs/ocl/0.14.1/ocl/ https://docs.rs/ocl/0.14.1/ocl/. It's a super cool library for rust. Also users should checkout tensorRT https://github.com/NVIDIA/gpu-rest-engine/tree/master/tensorrt https://github.com/NVIDIA/gpu-rest-engine/tree/master/tensor.... It's not super well supported and may go away, but it is fast
- gcp 9y agoQuestion about TensorRT: It takes in Caffe trained models. Is this BVLC Caffe or NV-Caffe? In my experience the models aren't compatible between the two.
- alexnewman 9y agoEither but I've never tried
- wcrichton 9y agoDo you know why TF is getting poor GPU util? Is it the pipeline feeding data to the core compute ops, inefficiencies in the core compute ops, or something else?
- flx42_ 9y agoJust wanted to chime in on TensorRT, it's a well supported product and it's different than gpu-rest-engine. This GitHub repo is simply an example of how to use TensorRT in a specific situation.