7 ms·

Hello, sorry for my vague question, I want learn CUDA so I can increase performance of an existing framework, pure performance and throughput time in considerat

Hello, sorry for my vague question, I want learn CUDA so I can increase performance of an existing framework, pure performance and throughput time in consideration. And no I dont have no prior experience on working with GPU's before.

exDM69 10y ago
> increase performance of an existing framework Cautionary tale following... I don't know the background on this but your goal might not be realistic, unless the rest of the framework architected in a GPU friendly way. I was once hired for such a project (some scientific applications in chemistry). They had a clear idea what needed to be improved, their Python framework called into C code for some non-trivial matrix operations. This was at the top of the profile so making it faster would have improved the overall performance. However, each matrix operation was just tens to hundreds of kilobytes in size. Doing them with the GPU was 100x faster, but the latency of getting the data to the GPU and back eliminated any gains. It's clear that doing these on the GPU would have improved overall performance if the operations could be batched. But the framework was tens of thousands of LOC of scientist-written Python code that was mission critical and could not be rewritten in the time and budget we had. GPU programming is high throughput at the expense of high latency. It is very difficult to retrofit into a CPU-only framework or application. You need to consider memory management and synchronization for the GPU from ground up.
- hubatrix 10y ago
  Thanks for the insight, Well I am working on a research project, to redesign a CPU architecture framework into a GPU one, and luckly I am working on real time batch processed data, so as of now have some plans as to what should go on GPU and I want to try it out myself and see if there are ways to cut of those bus tranmsion latencies between CPU and GPU. Not sure how exaclty but I just started kearning CUDA now, one more thing how will I know if I should use OpenCL or CUDA ?
  - exDM69 10y ago
    Latency hiding is the key for high throughput GPU applications. Make sure the GPU never waits for CPU and vice versa. Keep transfers to/from GPU flowing constantly and a queue of jobs scheduled. Each job should preferably be several megabytes in size. Memory bandwidth is the bottleneck for most practical tasks. Bandwidths you should know: CPU 50 GB/s, GPU 300 GB/s, PCI-e DMA 16 (v. 3.x) or 32 (v. 4.x) GB/s.
    - hubatrix 10y ago
      Thanks again, when I reach the point of actually implementing it I am sure I'll be asking you a question or two !! but HN, unfortunately does not have friend/follow functionalities
- scott_s 10y ago
  Yes, agreed completely. I had a similar experience in a streaming context, and wrote a paper about it: http://www.scott-a-s.com/files/debs2010.pdf http://www.scott-a-s.com/files/debs2010.pdf
j_s 10y ago
The following blog series may provide additional insight into the process of applying CUDA to existing code: http://blog.marcgravell.com/2016/05/cudagetting-started-in-net.html http://blog.marcgravell.com/2016/05/cudagetting-started-in-n...
- hubatrix 10y ago
  thanks for the resource !!

exDM69 10y ago

> increase performance of an existing framework Cautionary tale following... I don't know the background on this but your goal might not be realistic, unless the rest of the framework architected in a GPU friendly way. I was once hired for such a project (some scientific applications in chemistry). They had a clear idea what needed to be improved, their Python framework called into C code for some non-trivial matrix operations. This was at the top of the profile so making it faster would have improved the overall performance. However, each matrix operation was just tens to hundreds of kilobytes in size. Doing them with the GPU was 100x faster, but the latency of getting the data to the GPU and back eliminated any gains. It's clear that doing these on the GPU would have improved overall performance if the operations could be batched. But the framework was tens of thousands of LOC of scientist-written Python code that was mission critical and could not be rewritten in the time and budget we had. GPU programming is high throughput at the expense of high latency. It is very difficult to retrofit into a CPU-only framework or application. You need to consider memory management and synchronization for the GPU from ground up.

hubatrix 10y ago
Thanks for the insight, Well I am working on a research project, to redesign a CPU architecture framework into a GPU one, and luckly I am working on real time batch processed data, so as of now have some plans as to what should go on GPU and I want to try it out myself and see if there are ways to cut of those bus tranmsion latencies between CPU and GPU. Not sure how exaclty but I just started kearning CUDA now, one more thing how will I know if I should use OpenCL or CUDA ?
- exDM69 10y ago
  Latency hiding is the key for high throughput GPU applications. Make sure the GPU never waits for CPU and vice versa. Keep transfers to/from GPU flowing constantly and a queue of jobs scheduled. Each job should preferably be several megabytes in size. Memory bandwidth is the bottleneck for most practical tasks. Bandwidths you should know: CPU 50 GB/s, GPU 300 GB/s, PCI-e DMA 16 (v. 3.x) or 32 (v. 4.x) GB/s.
  - hubatrix 10y ago
    Thanks again, when I reach the point of actually implementing it I am sure I'll be asking you a question or two !! but HN, unfortunately does not have friend/follow functionalities
scott_s 10y ago
Yes, agreed completely. I had a similar experience in a streaming context, and wrote a paper about it: http://www.scott-a-s.com/files/debs2010.pdf http://www.scott-a-s.com/files/debs2010.pdf

j_s 10y ago

The following blog series may provide additional insight into the process of applying CUDA to existing code: http://blog.marcgravell.com/2016/05/cudagetting-started-in-net.html http://blog.marcgravell.com/2016/05/cudagetting-started-in-n...

hubatrix 10y ago
thanks for the resource !!