7 ms·
TPU transformation: A look back at 10 years of our AI-specialized chips
- alecco 2y agoHow are they connected? PCIe? Something like NVLink?
- tucnak 2y agoOptical circuit switches https://arxiv.org/abs/2304.01433 https://arxiv.org/abs/2304.01433
- nl 2y agoThey use custom optical "Interchip Interconnect" within each 256-chip "pod" and their custom "Jupiter" networking between pods. See https://cloud.google.com/blog/products/compute/introducing-trillium-6th-gen-tpus https://cloud.google.com/blog/products/compute/introducing-t... and https://cloud.google.com/blog/topics/systems/the-evolution-of-googles-jupiter-data-center-network?e=48754805 https://cloud.google.com/blog/topics/systems/the-evolution-o...
- walterbell 2y agoApple Intelligence uses Google TPUs instead of GPUs.
- j16sdiz 2y agoIf you read all their paper, they use a mix of them
- bigcat12345678 2y agoThat's something not surprising, given JG and Ruoming's Google stint. Google is going to dominate LLM ushered AI era. Google has been AI first since 2016, they just don't have the opening. Sam, as inapt at engineering, just has no idea how to navigate the delicate biz & eng competitions.
- dlewis1788 2y agoFor training, yes, but no indications on inference workloads. Apple has said they would use their own silicon for inference in the cloud.
- walterbell 2y agoPlus the Apple "Neural Engine" which has shipped on millions of iPhones for local inference.
- ec109685 2y agoImpressive: “Overall, more than 60% of funded generative AI startups and nearly 90% of gen AI unicorns use Google Cloud’s AI infrastructure, including Cloud TPUs.”
- lsb 2y agoDoesn’t Google Cloud’s AI infrastructure include Colab? That’s useful for so many things
- htrp 2y agoGoogle will also offer GCP credits for Free Nvidia GPUs with almost no questions asked. AWS and Azure (to a lesser extent) can also make this argument.
- zackangelo 2y agoAny strings attached? Do you know if they’ll do it pre-funding?
- bushbaba 2y agoUse does not mean heavily rely on. If an AI Startup uses google colab or runs 1 POC with TPUs, then they would fall under this stat.
- amelius 2y ago[flagged]
- walterbell 2y agoGoogle Pixel phones. 8X Coral Edge TPU M.2 on PCIe card, https://www.digikey.com/en/products/detail/asus/CRL-G18U-P3DF/15838692 https://www.digikey.com/en/products/detail/asus/CRL-G18U-P3D... & https://www.asus.com/networking-iot-servers/aiot-industrial-solutions/gpu-edge-ai-accelerators/ai-accelerator-pcie-card/ https://www.asus.com/networking-iot-servers/aiot-industrial-...
- hansihe 2y agoEdge TPUs are definitely not comparable to the datacenter TPUs. They only support TFLite for one.
- throwaway48476 2y agoDidn't they abandon edge TPUs?
- walterbell 2y agoAny references on that? For a couple of years, they were fetching 100% price premiums on eBay, due to high demand and low supply.
- throwaway48476 2y agohttps://github.com/blakeblackshear/frigate/issues/10056 https://github.com/blakeblackshear/frigate/issues/10056
- walterbell 2y agoHelpful thread, thanks: Google support team churn after the distribution transition to Asus IoT, Frigate devs were preparing to fork Google repos, then new Google devs appeared. > Google is getting back on top of things aka coral support which is nice.. it seems that the original devs weren't on the project and new devs needed to be given notice. Hopefully this continues and things are kept up to date.. updated libcoral and pycoral libraries are coming as well. It's good that Frigate brought attention to languishing Linux maintenance for Coral. Rockchip 3588 and other Arm SoCs have NPUs, which will likely be supported in time, but each SoC will require validation. Coral Edge TPUs were a convenient single target that worked with any x86 and Arm board, via USB or M.2 slot.
- nl 2y agoIt's crazy that Google doesn't spin-out their TPU work as a separate company. TPUs are the second most widely used environment for training after Nvidia. It's the only environment that people build optimized kernels for outside CUDA. If it was separate to Google then there a bunch of companies who would happily spend some money on a real, working NVidia alternative. It might be profitable from day one, and it surely would gain substantial market capitalization - Alphabet shareholders should be agitating for this!
- qwertox 2y agoThere would probably a huge demand, but would Google be able to satisfy it? Is it currently able to satisfy its own demand?
- credit_guy 2y agoThat would be the point of spinning it out. They could have an IPO, raise as much capital as there is in the observable Universe, and build enough fabs to satisfy all the demand.
- mike_hearn 2y agoThat wouldn't work. Even TPUv4 was on a 7nm node and you don't just build a 7nm fab just like that. If it were that easy NVIDIA would already be building their own fabs, as they have basically raised as much capital as there is in the known universe (bigger market cap than the entire London stock exchange), but they seem to prefer to let the fab experts get on with it rather than compete with them. LLM AI is largely HBM bottlenecked anyway i.e. Samsung, SK Hynix and Micron are where the supply chain limits enter the picture.
- throwaway48476 2y agoFabless companies that are large enough such as apple front the capital for fab companies like TSMC to build fabs dedicated to their use. They do, in effect, build their own fabs. If the Google TPU group had the inclination they could have done the same. The memory industry just got busted from the covid bubble and are not too keen to jump into the AI bubble.
- PedroBatista 2y agoThe real winner here is the marketing department who manage to make this article a "celebration of successes" when in fact we know the TPU is yet one more of those biggest failures of Google to have the lead by a mile and then.. squander it. And no, "it's on our cloud and Pixel phones" doesn't cut it at this level.
- visarga 2y agoI have a strong suspicion that previous generations of TPU were not cost effective for decent AI, explaining Google's reluctance to release complex models. They have had superior translation for years, for example. But scaling it up to the world population? Not possible with TPUs. It was OpenAI that showed you can actually deploy a large model, like GPT-4, to a large audience. Maybe Google didn't reach the cost efficiency with just internal use that NVIDIA does.
- throwaway48476 2y agoI suspect it had much more to do with lacking product market fit. They spent 10 years faking demos and dreaming about what they thought AI could do eventually but since it never worked the products never released and so they never expanded. A well optimized TPU will always beat a well optimized GPU on efficiency.
- cavisne 2y agoOnly because of Nvidia's margins. "Worse but cheaper" is actually great for a company of Googles scale, but it doesn't make for a particularly compelling press release or paper.
- throwawaymaths 2y agothere are several talks out there where Google soft-admits that at least the early gens of TPUs really sucked, e.g.: https://www.youtube.com/watch?v=nR74lBO5M3s https://www.youtube.com/watch?v=nR74lBO5M3s (note the lede on the TPU is buried pretty deep here)
- 2y ago