That's a nice goal on paper but in the real world a "generic" architecture is never going to approach the performance of a specialized one with bare-metal optim

That's a nice goal on paper but in the real world a "generic" architecture is never going to approach the performance of a specialized one with bare-metal optimizations. If your program is computationally intensive enough that you're abandoning general-purpose processors and moving to a specialized co-processor card, you should really just go whole-hog on it and bare-metal optimize to get some performance out of it. It doesn't make sense to do this half-way - which is why Xeon Phi has always faced such an uphill battle for adoption.