4 ms·

> I'd suggest creating a templated buffer class that manages data uploads/downloads and exposes the appropriate pointer on the appropriate device. This is the

by exDM69 11y ago

> I'd suggest creating a templated buffer class that manages data uploads/downloads and exposes the appropriate pointer on the appropriate device. This is the easy part. The hard part is getting this right in the big picture, to avoid having the CPU wait for the GPU and vice versa. Pipeline stalls will easily ruin your performance, to the point where single threaded CPU code runs faster because the high latency of data transfers. This is where modern GPU memory management really helps. For example, in OpenGL, you can use GL_MAP_BUFFER_PERSISTENT and _COHERENT to map your buffers once right after you create them and keep the pointer to the mapped buffer for the entire lifetime of the buffer (see ARB_buffer_storage). When you do this, you're responsible for the synchronization so that the CPU and GPU never touch the same parts of the buffer concurrently. CUDA probably has something similar. This adds some complexity to the application but now it's done explicitly in the application and there are no surprises. And the CPU/GPU page tables don't have to be modified, TLBs flushed and so on, so you get a bit of extra perf from there. And I totally agree with the point about the mindset of single threaded programmers with dynamic languages. I was once put into a project where a scientific application was written in Python and some "big matrix" operations were done in C. The matrix ops were on the top of the profile so they assumed that it can be made faster by simply "porting" them over to GPGPU. Well the matrices weren't that big (a few kilobytes) and the operations were done sequentially, one by one in the application. To make it work, the operations would have had to be batched but this would have meant a big refactoring to the entire Python codebase. This was way too complex for the chemists and phycisists who had written the code. They were really smart people with the domain knowledge but they didn't possess the mindset to think about making it efficient.

varelse 11y ago
The approach I've taken to this dilemma is to work directly with the scientists to understand the quantities they are trying to compute rather than attempt to port the code itself (yes, I actively choose refactoring over porting). Once the underlying algorithm has been deconstructed, one then reconstructs it parallelized in CUDA/OpenCL/whatever. In my experience, the implementation language and APIs can easily lead one to a suboptimal realization of algorithms. Once done, the original python code can serve as a reference implementation for conformance and unit testing. Of course, if you have 1M+ lines of legacy code, this probably won't work. But in my experience so far, the core algorithms have been less than 50K lines of code, and usually less than 10K. IMO sometimes one has to stand one's engineering ground to persuade them to make the right decision. And bonus, one ends up a ground floor domain expert at the end. Finally, I turned down a very prestigious project 4+ years ago because they wouldn't let me take the above approach. They opted instead to do it entirely with compiler directives. That project still hasn't officially shipped. It has, however, driven a significant amount of improvement in said compiler directives.

by exDM69 11y ago

varelse 11y ago

The approach I've taken to this dilemma is to work directly with the scientists to understand the quantities they are trying to compute rather than attempt to port the code itself (yes, I actively choose refactoring over porting). Once the underlying algorithm has been deconstructed, one then reconstructs it parallelized in CUDA/OpenCL/whatever. In my experience, the implementation language and APIs can easily lead one to a suboptimal realization of algorithms. Once done, the original python code can serve as a reference implementation for conformance and unit testing. Of course, if you have 1M+ lines of legacy code, this probably won't work. But in my experience so far, the core algorithms have been less than 50K lines of code, and usually less than 10K. IMO sometimes one has to stand one's engineering ground to persuade them to make the right decision. And bonus, one ends up a ground floor domain expert at the end. Finally, I turned down a very prestigious project 4+ years ago because they wouldn't let me take the above approach. They opted instead to do it entirely with compiler directives. That project still hasn't officially shipped. It has, however, driven a significant amount of improvement in said compiler directives.