> It never seemed to me like the custom chip from Tesla could possibly ever keep up with the economics of mass chip production from NVidia > The Tesla custom ch

> It never seemed to me like the custom chip from Tesla could possibly ever keep up with the economics of mass chip production from NVidia > The Tesla custom chip just won't be made at the volume of NVidia or AMD. I don't think Tesla ever aimed to produce a chip for the public, it's purely to be used in their own products, so they don't have to be competitive with NVIDIA in flops/$ or worry about keeping up with NVIDIA production rate. The chip only needs to be better, for the specific task they care about, for a specific metric they care about. Considering that NVIDIA has to make a card that works for everyone, they have to make a huge number of tradeoffs, such as supporting many different number formats going from FP64 to INT4, their cores have to support a ton of different instructions, they need to keep a large amount of space on the chip for large memory buffers to support training of models with billions of parameters, they need to support GPUs interconnect for clustering etc. etc. Given the importance of real estate on a chip, those tradeoff have large implications. Tesla has 10x less constraints: * For the training chip they only support their custom 8 and 16 bits format iic, so they can redistribute cores in consequence. They can ditch many instructions that they'll never use. They can size the memories to be exactly what they need, which is probably less than 80GB needed for large transformers, and use the freed space to add more cores etc. * For the inference chip, they can additionally ditch all the interconnects as they just need one per car and reduce memory drastically freeing up even more space for cores. If NVIDIA didn't have to balance so many different use cases on such a small area, then yes it would probably make no sense to try to do your own thing. But since they have to be general purpose, if you target a very specific use case you can be far less knowledgeable than them in GPU architecture and still come out ahead for your use case. Specificity is the same reason why it still possible to write CUDA kernels for some basic operation that will run 3x faster than the official NVIDIA implementation that was done by some amazing engineers: they have to make their operations efficient for every input size, you don't.