The approach I've taken to this dilemma is to work directly with the scientists to understand the quantities they are trying to compute rather than attempt to p

The approach I've taken to this dilemma is to work directly with the scientists to understand the quantities they are trying to compute rather than attempt to port the code itself (yes, I actively choose refactoring over porting). Once the underlying algorithm has been deconstructed, one then reconstructs it parallelized in CUDA/OpenCL/whatever. In my experience, the implementation language and APIs can easily lead one to a suboptimal realization of algorithms. Once done, the original python code can serve as a reference implementation for conformance and unit testing. Of course, if you have 1M+ lines of legacy code, this probably won't work. But in my experience so far, the core algorithms have been less than 50K lines of code, and usually less than 10K. IMO sometimes one has to stand one's engineering ground to persuade them to make the right decision. And bonus, one ends up a ground floor domain expert at the end. Finally, I turned down a very prestigious project 4+ years ago because they wouldn't let me take the above approach. They opted instead to do it entirely with compiler directives. That project still hasn't officially shipped. It has, however, driven a significant amount of improvement in said compiler directives.