Yes, but we have a lot more experience and existing code for solving this on more monolithic systems, whereas the massively parallel architecture of a GPU stops

Yes, but we have a lot more experience and existing code for solving this on more monolithic systems, whereas the massively parallel architecture of a GPU stops having so many advantages when you have heavy dependencies between operations on different "cores" (e.g. needing to compose arbitrary-precision math out of smaller operations), and thereby end up needing to synchronize rather than letting the threads compute in parallel.