As someone dealing with this right now, the challenges are multilayered. You need someone who's an expert at the underlying math and physics, then they also nee

As someone dealing with this right now, the challenges are multilayered. You need someone who's an expert at the underlying math and physics, then they also need to understand optimal GPU programming patterns and need to be convinced of the value of GPU acceleration in the first place. On top of that, the expert maintaining the HPC code has most likely personally contributed a lot of the things the code is using to run efficiently on CPUs, so proposing large design changes for GPU acceleration also leads to ego clashes because they (somewhat reasonably) don't just want to completely throw out most of their own R&D. For the application I'm working with, it has taken ~2 years just to get through basic issues like convincing the scientists of the value of GPU acceleration, striking a balance between their existing methods and restructuring large parts for GPU, and convincing them that instead they'll end up having a lot of room to explore new things like methods which utilize both CPU and GPU.