Two things matter at cloud-scale: Compatability - does using a tpu require reworking significant parts of your software stack and application stack? If so, th

Two things matter at cloud-scale: Compatability - does using a tpu require reworking significant parts of your software stack and application stack? If so, that sucks for most companies who's researchers are used to using nvidia libraries, tooling and hardware to get their models running, and reworking the entire bottom end of that to work on an esoteric platform needs to equate to a huge cost savings at scale to be worth it. Cost per tensor flop delivered - likely very low if google has optimized the silicon, memory, voltage, power and temperature envelope, networking, boards and chassis for the server running it, as well as optimizing for optimal process node efficiency/cost. They're probably not on bleeding edge tsmc process, but instead optimizing for total deployed running cost per pflop over 2-3 years. It's also now public (as of November) that microsoft/azure have been working for many years on their own ai chip, dubbed "Maia" [1], and appliance with the obvious goal of taking some of that nvidia margin in-house (and with openai/bing/copilot being a massive consumer of capacity). I think this will become even more commonplace with cloud vendors - even medium size ones - than it already is. The knowledge and complexity barrier to designing a tile processor unit seems pretty low and it looks like most of the hard stuff is in the drivers and software - something cloud providers designing and integrating internally can bypass and control to a great degree. It's also very hard to benchmark side by side on these since I'm sure cuda/nvidia hardware can do compute that a TPU cannot. AMD's machine learning accelerators look good on paper too, as do the tensor processors in the apple silicon, but on real world applications and use cases they don't often measure up save for a few optimal workflows. [1] https://www.geekwire.com/2023/microsoft-unveils-custom-ai-chip-with-help-from-openai-playing-catch-up-with-amazon/ https://www.geekwire.com/2023/microsoft-unveils-custom-ai-ch...