I think the tensor core in the 4090 really help, and of course CUDA supporting every hardware they offer (cough cough, rocm) means that researchers are going to

I think the tensor core in the 4090 really help, and of course CUDA supporting every hardware they offer (cough cough, rocm) means that researchers are going to start there. That said, I think Apple will have some interesting stuff in a year or two (M4 or more likely M5) where they can flex their NPU, Accelerate framework, and unified memory GPU and have it work with more modern requirements. Time will tell what their software and hardware story is for local inference for generative AI. Siri (dictation, some assistant stuff, and TTS) runs on device, and I doubt they want to undo that. I doubt they will do much for training, but maybe a NUMA version of a MacPro with several M4 Ultras will prove me wrong?