I don't mean general purpose hardware, I mean specifically neural hardware, whether that means GPUs (as we know them), GPUs with special hardware like the "tens

I don't mean general purpose hardware, I mean specifically neural hardware, whether that means GPUs (as we know them), GPUs with special hardware like the "tensor cores", TPUs, "neuromorphic chips" whatever that is, FPGAs becoming part of average computers, or something else. There's no end in sight to the improvement of neural hardware, not like the wall x86 CPUs have hit anyway.