The main benefit of using modern GPUs is that they have large high-bandwidth memory. You need to put a lot of work into optimizations to reach >10% of the peak

The main benefit of using modern GPUs is that they have large high-bandwidth memory. You need to put a lot of work into optimizations to reach >10% of the peak compute capability in practical use. That means an ASIC won't eliminate the performance bottleneck.