8 ms·
Ironwood: The first Google TPU for the age of inference
- nharada 2y agoThe first specifically designed for inference? Wasn’t the original TPU inference only?
- theptip 2y agoThe phrasing is very precise here, it’s the first TPU for _the age of inference_, which is a novel marketing term they have defined to refer to CoT and Deep Research.
- shitpostbot 2y agoThey didn't though? > first designed specifically for inference. For more than a decade, TPUs have powered Google’s most demanding AI training and serving workloads... What do they think serving is? I think this marketing copy was written by someone with no idea what they are talking about, and not reviewed by anyone who did. Also funny enough it kinda looks like they've scrubbed all their references to v4i, where the i stands for inference. https://gwern.net/doc/ai/scaling/hardware/2021-jouppi.pdf https://gwern.net/doc/ai/scaling/hardware/2021-jouppi.pdf
- dang 2y agoUgh. We should have caught that. Can anyone suggest a better (i.e. more accurate and neutral) title, devoid of marketing tropes?
- yencabulator 2y agoAs a previous boss liked to say, this car is the cheapest in its price range, the roomiest in it's size category, and the fastest in its speed group.
- jeffbee 2y agoYeah that made me chuckle, too. The original was indeed inference-only.
- dgacmu 2y agoYup. (Source: was at brain at the time.) Also holy cow that was 10 years ago already? Dang. Amusing bit: The first TPU design was based on fully connected networks; the advent of CNNs forced some design rethinking, and then the advent of RNNs (and then transformers) did it yet again. So maybe it's reasonable to say that this is the first TPU designed for inference in the world where you have both a matrix multiply unit and an embedding processor. (Also, the first gen was purely a co-processor, whereas the later generations included their own network fabric, a trait shared by this most recent one. So it's not totally crazy to think of the first one as a very different beast.)
- kleiba 2y ago> the advent of CNNs forced some design rethinking, and then the advent of RNNs (and then transformers) did it yet again. Certainly, RNNs are much older than TPUs?!
- miki123211 2y agoWow, you guys needed a custom ASIC for inference before CNNs were even invented? What were the use cases like back then?
- refulgentis 2y agohttps://research.google/blog/the-google-brain-team-looking-back-on-2016/ https://research.google/blog/the-google-brain-team-looking-b... is a good overview I wasn't on Brain, but got obsessed with Kerminology of ML internally at Google because I wanted to know why leadership was so gung ho on it. The general sense in the early days was these things can learn anything, and they'll replace fundamental units of computing. This thought process is best exhibited externally by ex. https://research.google/pubs/the-case-for-learned-index-structures/ https://research.google/pubs/the-case-for-learned-index-stru... It was also a different Google, the "3 different teams working on 3 different chips" bit reminds me of lore re: how many teams were working on Android wearables until upper management settled it. FWIW it's a very, very, different company now. Back then it was more entrepreneurial. A better version of Wave-era, where things launch themselves. An MBA would find this top-down company in 2025 even better, I find it less - it's perfectly tuned to do what Apple or OpenAI did 6-12 months ago, but not to lead - almost certainly a better investment, but a worse version of an average workplace, because it hasn't developed antibodies against BSing. (disclaimer: worked on Android)
- m00x 2y agoThe first one was designed as a proof of concept that it would work at all, not really to be optimal for inference workloads. It just turns out that inference is easier.
- fancyfredbot 2y agoIt looks amazing but I wish we could stop playing silly games with benchmarks. Why compare fp8 performance in ironwood to architectures which don't support fp8 in hardware? Why leave out TPUv6 in the comparison? Why compare fp64 flops in the El Capitan supercomputer to fp8 flops in the TPU pod when you know full well these are not comparable? [Edit: it turns out that El Capitan is actually faster when compared like for like and the statement below underestimated how much slower fp64 is, my original comment in italics below is not accurate] (The TPU would still be faster even allowing for the fact fp64 is ~8x harder than fp8. Is it worthwhile to misleadingly claim it's 24x faster instead of honestly saying it's 3x faster? Really?) It comes across as a bit cheap. Using misleading statements is a tactic for snake oil salesmen. This isn't snake oil so why lower yourself?
- shihab 2y agoI went through the article and it seems you're right about the comparison with El Capitan. These performance figures are so bafflingly misleading. And so unnecessary too- nobody shopping for AI inference server cares at all about its relative performance vs a fp64 machine. This language seems designed solely to wow tech-illiterate C-Suites.
- cheptsov 2y agoI think it’s not misleading, but rather very clear that there are problems. v7 is compared to v5e. Also, notice that it’s not compared to competitors, and the price isn’t mentioned. Finally, I think the much bigger issue with TPU is the software and developer experience. Without improvements there, there’s close to zero chance that anyone besides a few companies will use TPU. It’s barely viable if the trend continues.
- lawlessone 2y agoCan these be repurposed for other things? Encoding/decoding video? Graphics processing etc? edit: >It’s a move from responsive AI models that provide real-time information for people to interpret, to models that provide the proactive generation of insights and interpretation. This is what we call the “age of inference” where AI agents will proactively retrieve and generate data to collaboratively deliver insights and answers, not just data. maybe i will sound like a luddite but im not sure i want this. I'd rather AI/ML only do what i ask it to.
- vinkelhake 2y agoGoogle already has custom ASICs for video transcoding. YouTube has been running those for many years now. https://streaminglearningcenter.com/encoding/asics-vs-software-based-transcoding-an-analysis-of-youtubes-argos-transcoder.html https://streaminglearningcenter.com/encoding/asics-vs-softwa...
- lawlessone 2y agoThank you :)
- cavisne 2y agoThe JAX docs have a good explanation for how a TPU works https://docs.jax.dev/en/latest/pallas/tpu/details.html#what-is-a-tpu https://docs.jax.dev/en/latest/pallas/tpu/details.html#what-... Its not really useful for other workloads (unless your workload looks like a bunch of matrix multiplications).
- no_wizard 2y agoSome honest competition in the chip space in the machine learning race! Genuinely interested to see how this ends up playing out. Nvidia seemed 'untouchable' for so long in this space that its nice to see things get shaken up. I know they aren't selling the TPU as boxed units, but still, even as hardware that backs GCP services and what not, its interesting to see how it'll shake out!
- epolanski 2y ago> Nvidia seemed 'untouchable' for so long in this space that its nice to see things get shaken up. Did it? Both Mistral's LeChat (running on Cerebras) and Google's Gemini (running on Tensors) have clearly showed ages ago Nvidia had no advantage at all in inference. The hundreds of billions spent in hardware till now focused on training, but inference is in the long run gonna get the lion share of the work.
- wyager 2y ago> but inference is in the long run gonna get the lion share of the work. I'm not sure - might not the equilibrium state be that we are constantly fine-tuning models with the latest data (e.g. social media firehose)?
- NoahZuniga 2y agoHead of groq said that in his experience at google training was less than 10% of compute.
- lostmsu 2y agoIsn't Groq still more expensive than GPU-based providers?
- nehalem 2y agoNot knowing much about special-purpose chips, I would like to understand whether chips like this would give Google a significant cost advantage over the likes of Anthropic or OpenAI when offering LLM services. Is similar technology available to Google's competitors?
- baby_souffle 2y agoThere are other ai/llm ‘specific’ chips out there, yes. But the thing about asics is that you need one for each *specific* task. Eventually we’ll hit an equilibrium but for now, the stuff that Cerebras is best at is not what TPUs are best at is not what GPUs are best at…
- monocasa 2y agoI don't even know if eventually we'll hit an equilibrium. The end of Moore's law pretty much dictates specialization, it's just more apparent in fields without as much ossification first.
- avrionov 2y agoNVIDIA operates at 70% profit right now. Not paying that premium and having alternative to NVIDIA is beneficial. We just don't know how much.
- kccqzy 2y agoI might be misremembering here, but Google's own AI models (Gemini) don't use NVIDIA hardware in any way, training or inference. Google bought a large number of NVIDIA hardware only for Google Cloud customers, not themselves.
- heymijo 2y agoGPUs, very good for pretraining. Inefficient for inference. Why? For each new word a transformer generates it has to move the entire set of model weights from memory to compute units. For a 70 billion parameter model with 16-bit weights that requires moving approximately 140 gigabytes of data to generate just a single word. GPUs have off-chip memory. That means a GPU has to push data across a chip - memory bridge for every single word it creates. This architectural choice, is an advantage for graphics processing where large amounts of data needs to be stored but not necessarily accessed as rapidly for every single computation. It's a liability in inference where quick and frequent data access is critical. Listening to Andrew Feldman of Cerebras [0] is what helped me grok the differences. Caveat, he is a founder/CEO of a company that sells hardware for AI inference, so the guy is talking his book. [0] https://www.youtube.com/watch?v=MW9vwF7TUI8&list=PLnJFlI3aINuUNaznonoFEgMUGLBiLmB-w&index=5 https://www.youtube.com/watch?v=MW9vwF7TUI8&list=PLnJFlI3aIN...
- behnamoh 2y agoThe naming of these chips (GPUs, CPUs) is kinda badass: Ironwood, Blackwell, ThreadRipper, Epyc, etc.
- mikrl 2y agoScroll through wikichip sometime and try to figure out the Intel μarch names. I always confuse Blackwell with Bakewell (tart) and my CPU is on Coffee Lake and great… now I want coffee and cake
- qoez 2y agoPost just to tease us since they barely sell TPUs
- throwaway48476 2y agoIts hard to be excited about hardware that will only exist in the cloud before shredding.
- p_j_w 2y agoI think this article is for Wall Street, not Silicon Valley.
- noitpmeder 2y agoWhat's their use case?
- mycall 2y agoBad timing as I think Wall Street is preoccupied at the moment.
- asdfman123 2y agoOh, believe me, they are very much paying attention to tech stocks right now.
- SquareWheel 2y agoMust be part of that Preoccupy Wall Street movement.
- killerstorm 2y agoIt might also be for people who consider working for Google...
- fluidcruft 2y agoThis isn't anything anyone can purchase, is it? Who's the audience for this announcement?
- badlucklottery 2y ago> Who's the audience for this announcement? Probably whales who can afford to rent one from Google Cloud.
- jeffbee 2y agoPeople with $3 are whales now? TPU prices are similar to other cloud resources.
- dylan604 2y agoDoes anyone do anything useful with a $3 spend, or is it $3 X $manyManyHours?
- scarmig 2y agoNo one does anything useful with a $3 spend. That's not anything particular to TPUs, though.
- dylan604 2y agoThat's my point. The touting of $3 is beyond misleading.
- fancyfredbot 2y agoYou can do real work for a few hundred dollars which is hardly the exclusive domain of "whales"? The programmer who writes code to run on these likely costs at least 15x this amount an hour.
- MasterScrat 2y ago
- _hark 2y agoCan anyone comment on where efficiency gains come from these days at the arch level? I.e. not process-node improvements. Are there a few big things, many small things...? I'm curious what fruit are left hanging for fast SIMD matrix multiplication.
- yeahwhatever10 2y agoSpecialization. Ie specialized for inference.
- vessenes 2y agoOne big area the last two years has been algorithmic improvements feeding hardware improvements. Supercomputer folks use f64 for everything, or did. Most training was done at f32 four years ago. As algo teams have shown fp8 can be used for training and inference, hardware has updated to accommodate, yielding big gains. NB: Hobbyist, take all with a grain of salt
- jmalicki 2y agoUnlike a lot of supercomputer algorithms, where fp error accumulates as you go, gradient descent based algorithms don't need as much precision since any fp errors will still show up at the next loss function calculation to be corrected, which allows you to make do with much lower precision.
- cubefox 2y agoMuch lower indeed. Even Boolean functions (e.g. AND) are differentiable (though not exactly in the Newton/Leibniz sense) which can be used for backpropagation. They allow for an optimizer similar to stochastic gradient descent. There is a paper on it: https://arxiv.org/abs/2405.16339 https://arxiv.org/abs/2405.16339 It seems to me that floating point math (matrix multiplication) will over time mostly disappear from ML chips, as Boolean operations are much faster both in training an inference. But currently they are still optimized for FP rather than Boolean operations.
- muxamilian 2y ago
- deleted 2y ago[deleted]
- vessenes 2y ago7.2 Terabit/s HBM Bandwidth raised my eyebrows. But then I googled, and it looks like GB200 is 16Tb/s. In plebe land, 2Tb is pretty awesome. These continue to be mostly for bragging rights and strategic safety I think. I bet they are not on premium processor nodes; If I worked at GOOG I’d probably think about these as competitive insurance vis-a-vis NVIDIA — total costs of chip team, software, tape outs, and increased data center energy use probably wipe out any savings from not buying NV, but you are 100% not beholden to Jensen.
- gigel82 2y agoI was hoping they're launching a Coral kind of device that can run locally and cheaply, with updated specs. It would be awesome for things like homelabs (to run Frigate NVR, Immich ML tasks or the Home Assistant LLM).
- Calwestjobs 2y ago[flagged]
- GrumpyNl 2y agoWhy doesnt google offer the most advanced voice technology when they offer a playback version, it still sounds like the most basic text to voice.
- lostmsu 2y agoMaybe it's good enough. In the end what matters is how strong the text of response is.
- tuna74 2y agoHow is API story for these devices? Are the drivers mainlined in Linux? Is there a specific API you use to code for them? How does the instance you rent on Google Cloud look and what does that have for software?
- cbarrick 2y agoXLA (Accelerated Linear Algebra) [1] is likely the library that you'll want to use to code for these machines. TensorFlow, PyTorch, and Jax all support XLA on the backend. [1]: https://openxla.org/ https://openxla.org/
- ndesaulniers 2y agoThey have out of tree drivers. If they don't ship the hardware to end users, it's not clear upstream (Linux kernel) would want them.
- g42gregory 2y agoAnd ... where could we get one? If they wouldn't sell it anyone, then is this a self-congratulation story? Why do we even need to know about this? If it propagates to the lower Gemini prices, fantastic. If not, then isn't it kind of irrelevant for the actual user experience?
- lordofgibbons 2y agoYou can rent it on GCP in a few months
- g42gregory 2y agoGood point. At what prices per GB/TOPS? Better be lower than the existing TPUs ... That's what I care about.
- jstummbillig 2y agoWell, with stocks and all, there is more that matters in the world than "actual user experience"
- DeathArrow 2y agoCool. But does it support CUDA?
- nickysielicki 2y agoCUDAs are the secret ingredient in the AI scaling sauce.
- saagarjha 2y agoNo. Google has a JAX stack.
- wg0 2y agoCan anyone buy them?
- ein0p 2y agoGod damn it, Google. Make a desktop version of these things.
- DisjointedHunt 2y agoCloud resources are trending towards consumer technology adoption numbers rather than being reserved mostly for Enterprise. This is the most exciting thing in decades! There is going to be a GPU/Accelerator shortage for the foreseeable future to run the most advanced models, Gemini 2.5 Pro is such a good example. It is probably the first model that many developers i've considered skeptics of extended agent use have started to saturate free token thresholds on. Grok is honestly the same, but the lack of an API is suggestive of the massive demand wall they face.
- AlexCoventry 2y agoWho manufactures their TPUs? Wondering whether they're US-made, and therefore immune to the tariff craziness.
- mupuff1234 2y agoPretty sure it's broadcom.
- nickysielicki 2y agoBroadcom is fabless…
- mupuff1234 2y agoOops.
- nsteel 2y agoYou were right in this context. They have previously partnered with Broadcom to make their TPUs, rather than work directly with TSMC. But I read it's actually Mediatek instead for this generation.
- adgjlsfhk1 2y agoit's tsmc
- AlexCoventry 2y agoThanks.
- attentive 2y agoanyone knows how this compares to AWS Inferentia chips?
- aranw 2y agoI wonder if these chips might contribute towards advancements for the Coral TPU chips?
- mianos 2y agoThere were pretty much abandoned years ago. The software stack support from Google only really lasted a few months and even then the stack they ran on was already years old versions of operating systems and python versions. The only support is via a few enthusiastic third party developers.
- QuadrupleA 2y ago"Age of inference", "age of Gemini", bleh. Google's doing some amazing work but I hate how they communicate - vague, self-serving, low signal-to-noise slop. It's like they take some interesting wood carving of communication, then sand it down to a featureless nub.
- sait007 2y agoAny idea on the system topology for their 9216 TPU chip pods ? Curious to know how things are arranged at the board, rack and pod level, CPU Vs TPU ratio, TPU interconnect topology - switch Vs mesh etc.
- budmichstelk 2y agoHad coffee with some of these people. Great to know they've come such a long way.
- MegaAgenticAI 2y agoMore details on Ironwood, Cloud TPUs and insights from Jeff Dean: https://www.youtube.com/watch?v=fNjH5izFeyw https://www.youtube.com/watch?v=fNjH5izFeyw
- plotmaster512 2y agoDumb question, but will these chips be available for personal computing? Or are they datacenter chips.