11 ms·

Our eighth generation TPUs: two chips for the agentic era

454 points by xnx 6mo ago

https://cloud.google.com/blog/products/compute/tpu-8t-and-tpu-8i-technical-deep-dive https://cloud.google.com/blog/products/compute/tpu-8t-and-tp...

TheMrZZ 6mo ago
> A single TPU 8t superpod now scales to 9,600 chips and two petabytes of shared high bandwidth memory, with double the interchip bandwidth of the previous generation. This architecture delivers 121 ExaFlops of compute and allows the most complex models to leverage a single, massive pool of memory. This seems impressive. I don't know much about the space, so maybe it's not actually that great, but from my POV it looks like a competitive advantage for Google.
- cyanydeez 6mo ago
  it is. itll still not create AGI without some breakthrough in instruction vs data separation of concerns
  - lugu 6mo ago
    You can park a lot there. No offence but I love how AGI doesn't mean anything. It used to be that AI was a goal post. Now it is AGI. We could use characters from sci-fi culture to describe milestones. In order to achieve robocop level, we must solve the instruction vs data problem.
    - lokar 6mo ago
      Thus it always was. I’m old enough to remember when “if AI could beat a grandmaster at chess” was considered the finish line.
      - sanex 6mo ago
        Over fitting to the benchmarks since 1996
        deleted 6mo ago
        [deleted]
      - emp17344 6mo ago
        Well, yeah… turns out that goal wasn’t a good indicator for AGI, so we re-evaluated. That’s changing your hypothesis in the face of evidence, not “moving the goalposts” in the fallacious sense.
        Mistletoe 6mo ago
        What’s the indicator for AGI now? We are so far past the Turing Test it isn’t funny. In fact the models now are too intelligent, you would never think a human would have that much knowledge quickly about a subject you chose at random.
    - lowbloodsugar 6mo ago
      Robocop was a human brain in a suit! Don't give them any ideas!
  - scottyah 6mo ago
    In what way do we not already have AGI?
    - airstrike 6mo ago
      in all of them
      - Der_Einzige 6mo ago
        Peter norvig is smarter than you are: https://www.noemamag.com/artificial-general-intelligence-is-already-here/ https://www.noemamag.com/artificial-general-intelligence-is-...
  - omcnoe 6mo ago
    Does non-artificial intelligence have clean instruction/data separation?
NoiseBert69 6mo ago
That cooling system looks crazy. What an unbelievable density.
Keyframe 6mo ago
As others have been capturing news cycle eyes, seems to me Google has been going from strength to strength quietly in the background capturing consumer market share and without much (any?) infrastructure problems considering they're so vertically integrated in AI since day one? At one point they even seemed like a lost cause, but they're like a tide.. just growing all around.
- deleted 6mo ago
  [deleted]
- baq 6mo ago
  you've never tried to use gemini 3 I guess - that thing was so unreliable it might as well not be offered; there's also a reason why everybody here is excited for claude and codex, but not really for antigravity. that said, I actually agree: google IMHO silently dominates the 'normie business' chatbot area. gemini is low key great for day to day stuff.
  - sunaookami 6mo ago
    Yeah Gemini is not usable outside of the gemini.google.com homepage and AI Studio. Gemini CLI does not work at all and all the models are constantly overloaded or time out. Got a trial month and there was not a single day I could actually work with Gemini in a coding harness, 3.1 Pro was never reachable. Awful service, imagine if I'd have paid for this. Additionally since roughly two weeks Gemini 2.5 Flash (a stable GA model) is constantly throwing backend errors on Google's side when Grounding is on.
- deleted 6mo ago
  [deleted]
- youniverse 6mo ago
  Yeah I think there will be a time in a few years (1-2?) when both Google and Apple will get to eat their cake. They aren't playing the same game of speed running unpolished product releases every month to double their valuation. They have time to think and observe and put out something really polished. At least that's the hope! :)
amazingamazing 6mo ago
If ai ends up having a winner I struggle to see how it doesn’t end with Google winning because they own the entire stack, or Apple because they will have deployed the most potentially AI capable edge sites.
- solarkraft 6mo ago
  I see a significant chance that they’ll continue to blunder the product side. It might still not matter because of their massive distribution, but leaves them open to disruption by a better product (think IE vs. Chrome).
- davebren 6mo ago
  I think the winner will be local model wrappers that are designed to do specific tasks well like search without being anthropomorphized sycophants.
aliljet 6mo ago
The real problem is that scientists doing this sort of early work more often than not want to burn hardware under their desks. Renting infrastructure in Google cloud isn't the only way...
- tjwebbnorfolk 6mo ago
  You need to have an awfully big desk to stick compute under in order to run a training workload of any interest...
nickandbro 6mo ago
I am curious what workloads Citadel Securities is running on these TPUs? Are you telling me they need the latest TPUs for market insights?
- vibe42 6mo ago
  Training their own, closed, internal models on their own data sets? Probably a good way to squeeze out some market trading signals.
  - nickandbro 6mo ago
    Reminds me of when hedge funds started laying increasingly shorter fiber-optic cable lines to achieve the lowest possible latency for high-frequency trading.
  - written-beyond 6mo ago
    I thought these TPUs were primarily used for inference?
    - vlovich123 6mo ago
      TPU8t is for training. But even still, once you’ve trained, you need to run the model too. And these kinds of models already have a huge latency hit so there’s not much hurting running it away from the trading switches.
    - knowaveragejoe 6mo ago
      As the article states, there's both training and inference dedicated chips.
- dist-epoch 6mo ago
  Not Citadel, but Jane Street is training LLMs for trading: https://www.janestreet.com/join-jane-street/machine-learning/ https://www.janestreet.com/join-jane-street/machine-learning... > We build on the latest papers in LLMs, computer vision, RL, training libraries, cuda kernels, or whatever else we need to train good models. > We invent our own set of architectures and optimizations that work for trading.
pmb 6mo ago
At this point, when you are doing big AI you basically have to buy it from NVidia or rent it from Google. And Google can design their chips and engine and systems in a whole-datacenter context, centralizing some aspects that are impossible for chip vendors to centralize, so I suspect that when things get really big, Google's systems will always be more cost-efficient. (disclosure: I am long GOOG, for this and a few other reasons)
- sigmoid10 6mo ago
  I'd bet that too if their management wasn't so incredibly uninspiring. Like, Apple under Cook was also pretty mild and a huge step down from Jobs, but Google feels like it fell off a cliff. If it wasn't for OpenAI releasing ChatGPT, they might still be sitting on that tech while only testing it internally. Now it drives their entire chip R&D.
  - WarmWash 6mo ago
    To be fair, I don't think any of the AI players wanted what OAI did. Sam grabbed first mover at the cost of this insane race everyone else got forced into.
  - hkpack 6mo ago
    I am not fan of the era when CEO is expected to be a cult leader type person. Cook did very well in all areas as well as in not trying to create a cult.
  - whattheheckheck 6mo ago
    What would an inspiring leader do differently for you?
    - someguyiguess 6mo ago
      Inspire
      - lokar 6mo ago
paulmist 6mo ago
At $15/GB of HBM4 the 331.8TB of HBM4 per pod is 5 million...
- nsteel 6mo ago
  It's HBM3e
- zozbot234 6mo ago
  $15/GB is retail price for DIMM sticks. Is HBM4 really that cheap?
  - selectodude 6mo ago
    HBM is just DRAM stacked directly next to the die. The expensive part is gluing it on there. The chips themselves are pretty much the same.
    - akelly 6mo ago
      HBM uses about twice as much DRAM silicon per GB due to all the space for interconnect
vibe42 6mo ago
The pics of the cooling system is pretty good sci-fi / cyberpunk / steampunk inspo. If the whole AI bubble spectularly collapes, at least we got a lot of cool pics of custom hardware!
- NitpickLawyer 6mo ago
  > If the whole AI bubble spectularly collapes Every other news for the past month has been about lacking capacity. Everyone is having scaling issues with more demand than they can cover. Anthropic has been struggling for a few months, especially visible when EU tz is still up and US east coast comes online. Everything grinds to a halt. MS has been pausing new subscriptions for gh Copilot, also because a lack of capacity. And yet people are still on bubble this, collapse that? I don't get it. Is it becoming a meme? Are people seriously seeing something I don't? For the past 3 years models have kept on improving, capabilities have gone from toy to actually working, and there's no sign of stopping. It's weird.
  - redsocksfan45 6mo ago
    [dead]
  - vibe42 6mo ago
    Both are possible; increasing demand and bubble collapse. The way this could happen is if model commoditization increases - e.g. some AI labs keep publishing large open models that increasingly close the gap to the closed frontier models. Also, if consumer hardware keep getting better and models get so good that most people can get most of their usage satisfied by smaller models running on their laptop, they won't pay a ton for large frontier models.
    - otterley 6mo ago
      I’m going to stick my neck out a bit and predict that model commoditization will never happen as long as humans keep producing new content and innovation for models to train on. Sure, some open models will be good enough to write software against, but that’s but a fraction of the overall market for this technology.
nsteel 6mo ago
This link has more on the architecture: https://cloud.google.com/blog/products/compute/tpu-8t-and-tpu-8i-technical-deep-dive https://cloud.google.com/blog/products/compute/tpu-8t-and-tp...
fulafel 6mo ago
"TPU 8t and TPU 8i deliver up to two times better performance-per-watt over the previous generation" sounds impressive especially as the previous generation is so recent (2025). Interesting that there's separate inference and training focused hardware. Do companies using NV hardware also use different hardware for each task or is their compute more fungible?
- dataking 6mo ago
  Vera Rubin will have Groq chips focused on fast inference so it points toward a trend. Also, with energy needs so high, why not reach for every feasible optimization?
- xnx 6mo ago
  Nvidia said in March that they're working on specialized inference hardware, but they don't have any right now. You can do inference from Nvidia's current hardware offerings, but it's not as efficient.
  - FuriouslyAdrift 6mo ago
    AMD has been doing inference chips for many years and are the leader for HPC. https://www.amd.com/en/products/accelerators/instinct.html https://www.amd.com/en/products/accelerators/instinct.html
- zozbot234 6mo ago
  The "training" chips will probably be quite usable for slower, higher-throughput inference at scale. I expect that to be quite popular eventually for non-time-sensitive uses.
- electroly 6mo ago
  I can't answer for NVIDIA but AWS has its own training and inference chips, and word on the street is the inference chips are too weak, so some companies are running inference on the training chips.
  - 6mo ago
cmptrnerd6 6mo ago
Which company is building the silicon for Google? Is it tsmc? What node size? I didn't see it with a quick search, sorry if it was in the post.
- wina 6mo ago
  tsmc through broadcom
  - nsteel 6mo ago
    Apparently it's Broadcom for 8t and Mediatek for 8i. https://wccftech.com/google-splits-tpuv8-strategy-two-chips-broadcom-training-mediatek-inference-duties/ https://wccftech.com/google-splits-tpuv8-strategy-two-chips-...
varispeed 6mo ago
I can't help but think we will be "laughing" at this in 10 years time like we laugh at steam engines or abacus.
iandanforth 6mo ago
Anyone know if these are already powering all of Gemini services, some of them, or none yet? It's hard to tell if this will result in improvements in speed, lower costs, etc, or if those will be invisible, or have already happened.
kamranjon 6mo ago
It's interesting that, of the large inference providers, Google has one of the most inconvenient policies around model deprecation. They deprecate models exactly 1 year after releasing them and force you to move onto their next generation of models. I had assumed, because they are using their own silicon, that they would actually be able to offer better stability, but the opposite seems to be true. Their rate limiting is also much stricter than OpenAI for example. I wonder how much of this is related to these TPU's, vs just strange policy decisions.
- gordonhart 6mo ago
  It's frustrating how cavalier they are about killing old Gemini releases. My read is that once a new model is serving >90% of volume, which happens pretty quickly as most tools will just run the latest+greatest model, the standard Google cost/benefit analysis is applied and the old thing is unceremoniously switched off. It's actually surprising that they recently extended the EOL date for Gemini 2.5. Google has never been a particularly customer-obsessed company...
  - surajrmal 6mo ago
    What benefit is there to sticking on older models? If the API is the same, what are the switching costs?
    - gordonhart 6mo ago
      If you're trying to run repeatable workflows, stability from not changing the model can outweigh the benefits of a smarter new model. The cost can also change dramatically: on top of the higher token costs for Gemini Pro ($1.25/mtok input for 2.5 versus $2/mtok input for 3.1), the newer release also tokenizes images and PDF pages less efficiently by default (>2x token usage per image/page) so you end up paying much much more per request on the newer model. These are somewhat niche concerns that don't apply to most chat or agentic coding use cases, but they're very real and account for some portion of the traffic that still flows to older Gemini releases.
    - 6mo ago
WarmWash 6mo ago
Whats interesting to note, as someone who uses Gemini, ChatGPT, and Claude, is that Gemini consistently uses drastically fewer tokens than the other two. It seems like gemini is where it is because it has a much smaller thinking budget. It's hard to reconcile this because Google likely has the most compute and at the lowest cost, so why aren't they gassing the hell out of inference compute like the other two? Maybe all the other services they provide are too heavy? Maybe they are trying to be more training heavy? I don't know, but it's interesting to see.
- someguyiguess 6mo ago
  They have to have SOME competitive advantage. What reason is there to use Gemini over Claude or ChatGPT? It's not producing nearly the quality of output.
  - magicalhippo 6mo ago
    Well comparing Gemini 3.1 Pro vs ChatGPT 5.4 Pro, it's much faster at replying. Of course, if it actually thinks less then that helps a lot towards that. For most of my personal and work use-cases, I prefer waiting a bit longer for a better answer.
  - WarmWash 6mo ago
    I recently did my taxes using all three models (My return is ~50 pages, much more than a standard 1040). GPT (codex) was accurate on the first run and took 12 minutes Gemini (antigravity) missed 1 value because it didn't load the full 1099 pdf (the laziness), but corrected it when prompted. However it only spent 2 minutes on the task. Claude (CC) made all manner of mistakes after waiting overnight for it to finish because it hit my limit before doing so. However claude did the best on the next step of actually filing out the pdf forms, but it ended up not mattering. Ultimately I used gemini in chrome to fill out the forms (freefillableforms.com), but frankly it would have been faster to manually do it copying from the spreadsheets GPT and Gemini output. I also use anti-gravity a lot for small greenfield projects(<5k LOC). I don't notice a difference between gemini and claude, outside usage limits. Besides that I mostly use gemini for it's math and engineering capabilities.
zshn25 6mo ago
[dead]
zshn25 6mo ago
It would be interesting to benchmark a short training / inference run on the latest of TPU vs. NVIDIA GPU per cost basis
jmyeet 6mo ago
In recent discussions about Tim Apple [sic] moving on there was a discussion about whether Apple flopped on AI, which is my opinion. Of course you had the false dichotomy of doing nothing or burning money faster than the US military like OpenAI does. IMHO that happy medium is Google. Not having to pay the NVidia tax will likely be a huge competitive advantage. And nobody builds data centers as cost-effectively as Google. It's kind of crazy to be talking ExaFLOPS and Tb/s here. From some quick Googling: - The first MegaFLOPS CPU was in 1964 - A Cray supercomputer hit GigaFLOPS in 1988 with workstations hitting it in the 1990s. Consumer CPUs I think hit this around 1999 with the Pentium 3 at 1GHz+; - It was the 2010s before we saw off-the-shelf TFLOPS; - It was only last year where a single chip hit PetaFLOPS. I see the IBM Roadrunner hit this in 2008 but that was ~13,000 CPUs so... Obviously this is near 10,000 TPUs to get to ~121 EFLOPS (FP4 admittedly) but that's still an astounding number. IT means each one is doing ~12 PFLOPS (FP4). I saw a claim that Claude Mythos cost ~$10B to train. I personally believe Google can (or soon will be able to) do this for an order of magnitude less at least. I would love to know the true cost/token of Claude, ChatGPT and Gemini. I think you'll find Google has a massive cost advantage here.
- knowaveragejoe 6mo ago
  > I saw a claim that Claude Mythos cost ~$10B to train. Can you cite this? That seems absurd.
  - jmyeet 6mo ago
    I've seen various claims to this (eg [1][2][3]) but nobody reall knows. These may all come from one uunsubstantiated claim. It is I think widely accepted that Mythos is ~10T parameters. I've seen figures that suggest GPT-4 was 1.8T parameters and cost upwards of $100 million to train (also unsubstantiated), in which case the Mythos figure might be inflated and also include development costs. So who really knows? [1]: https://www.softwarereviews.com/research/claude-mythos-preview-and-project-glasswing-what-it-and-security-leaders-need-to-know-now https://www.softwarereviews.com/research/claude-mythos-previ... [2]: https://x.com/duttasomrattwt/status/2041903600516133016 https://x.com/duttasomrattwt/status/2041903600516133016 [3]: https://www.forrester.com/blogs/project-glasswing-the-10-consequences-nobodys-writing-about-yet/ https://www.forrester.com/blogs/project-glasswing-the-10-con...
himata4113 6mo ago
I already felt that gemini 3 proved what is possible if you train a model for efficiency. If I had to guess the pro and flash variants are 5x to 10x smaller than opus and gpt-5 class models. They produce drastically lower amount of tokens to solve a problem, but they haven't seem to have put enough effort into refinining their reasoning and execution as they produce broken toolcalls and generally struggle with 'agentic' tasks, but for raw problem solving without tools or search they match opus and gpt while presumably being a fraction of the size. I feel like google will surprise everyone with a model that will be an entire generation beyond SOTA at some point in time once they go from prototyping to making a model that's not a preview model anymore. All models up till now feel like they're just prototypes that were pushed to GA just so they have something to show to investors and to integrate into their suite as a proof of concept.
- ALLTaken 6mo ago
  [flagged]
  - _boffin_ 6mo ago
    Is your friend on the JAX team?
  - neonstatic 6mo ago
    I'm really struggling with terrible bloating today, but I deemed it too dangerous to release.
    - tclancy 6mo ago
      Thank you for your sacrifice. Could you speak to my dog please? You may wish to yell from a distance, actually.
- onlyrealcuzzo 6mo ago
  > They produce drastically lower amount of tokens to solve a problem, but they haven't seem to have put enough effort into refinining their reasoning and execution as they produce broken toolcalls and generally struggle with 'agentic' tasks, but for raw problem solving without tools or search they match opus and gpt while presumably being a fraction of the size. Agreed, Gemini-cli is terrible compared to CC and even Codex. But Google is clearly prioritizing to have the best AI to augment and/or replace traditional search. That's their bread and butter. They'll be in a far better place to monetize that than anyone else. They've got a 1B+ user lead on anyone - and even adding in all LLMs together, they still probably have more query volume than everyone else put together. I hope they start prioritizing Gemini-cli, as I think they'd force a lot more competition into the space.
SecretDreams 6mo ago
They are missing a header to show the transition in discussion from TPU8t to 8i! Thanks for posting otherwise. Edit: actually, looks like the header got captured as a figure caption on accident.
nicman23 6mo ago
yeah but can you release the sdk for the pixel 10? it was one of then only reasons which i bought this mid phone
bozdemir 6mo ago
[dead]
jjice 6mo ago
I've been saying it, and I'll keep saying it (as someone who has an opinion backed by very little) - I think Google is incredibly well placed for the future with LLMs. Owning your hardware and your entire stack is huge, especially these days with so much demand. Long term, I think they end up doing very well. People clowned so hard on Google for the first two years (until Gemini 2.5 or 3) because it wasn't as good as OpenAI or Anthropic's models, but Google just looked so good for the long game. Another benefit for them: If LLMs end up being a huge bubble that end up not paying the absurd returns the industry expects, they're not kaput. They already own so many markets that this is just an additional thing for them, where as the big AI only labs are probably fucked. All that said: what the hell do I know? Who knows how all of this will play out. I just think Google has a great foundation underneath them that'll help them build and not topple over.
Aissen 6mo ago
Interesting that t8i is both for post-training and inference.
ks2048 6mo ago
For how many times does this article mentions "agentic" and "agents"... Am I correct assume the hardware has nothing to do with "agents"? I assume it's just about a new generation of more efficient transformers / deep-learning layers.
- tomassre 6mo ago
  There’s issues specific to workflows “agents”. For example many requests in an agent are all on top of the same previous results so context (kv cache) needs to be longer, and they use these massive connected nodes with direct nvme to cache the part of the prompt that’s repeatable. It is about agents in that the design is for long context, many requests where the initial “chunk” is cached but spread across many requests. They don’t call this out specifically but in the technical details like about the sram, how it’s all interconnected nodes in a pod it’s “designed” for it.
geremiiah 6mo ago
TPUs are systolic arrays right? So does that mean that Google is using a hetreogenous cluster compromising both GPUs and TPUs, for workloads that don't map well or at all on TPUs?
- thatguysaguy 6mo ago
  What sort of workloads are you thinking of?
- eklitzke 6mo ago
  I can't speak to what every team at Google does, but there are machines with Nvidia GPUs in Borg. However Google charges orgs internally for cpu/memory/gpu/tpu usage and TPUs are *way* more efficient in terms of FLOPS/$ than Nvidia GPUs, so there is a *huge* incentive for teams to use TPUs if they can, especially for teams operating large products.
mlmonkey 6mo ago
FTA: > One pod of TPU 8t is 121 ExaFlops; or 121,000 PetaFlops. Meanwhile, the compute capacity of the top 10 supercomputers in the entire world is 11,487 Petaflops.[1] I know, I know, not the same flops, yada yada, but still. Just 1 pod alone is quite a beast. Edit: [1] https://top500.org/lists/top500/2025/11/ https://top500.org/lists/top500/2025/11/
- dist-epoch 6mo ago
  Not only not the same size, 4-bit flops versus 64-bit flops, but not the same programmability either. the TPUs can do just matrix-multiplications and some supporting math. Otherwise bitcoin mining rigs dwarf everything, if you just want to count raw operations per second.
- stabbles 6mo ago
  Other than TPUs they're also planning for 960,000 Rubin GPUs [1] which can do 33 teraflops fp64 each, so over 30 classical exaflops, and with emulation it could be more than 100 exaflops. [1] https://blogs.nvidia.com/blog/google-cloud-agentic-physical-ai-factories/ https://blogs.nvidia.com/blog/google-cloud-agentic-physical-...
delbronski 6mo ago
I’ve been using Gemini with Junie (jetbrains attempt at Claude code). While Junie is nowhere near as good as Claude Code, it is way ahead of the current Google tooling. I get quite good consistent results for pretty cheap with this combo.
- rtaylorgarlock 6mo ago
  Do you think that Junie specifically inside the context of jetbrains IDEs and tooling matches the competitors?
  - delbronski 6mo ago
    I think so. From my experience Claude/codex tooling really excels at vibe coding the whole thing. You give it a folder and just say: now make it do this. And you don’t really care for the code. Junie tooling excels when you are more involved. Like, look in these two files, add this specific functionality, in this specific way. Junie is usually a lot faster and to the point. Very simple tooling , it just works for this workflow. But it breaks for the “code the whole thing for me” workflow.
    - chaoz_ 6mo ago
      It surprises me how JetBrains managed to lose such a great market opportunity. I don't think they ever going to be able to re-claim large chunk of developers who are now fine with thin VSCode-like + Terminal for non-JVM languages. Perfect example of how large corp with research capacity failed to navigate their product changes.
      - foobazgt 6mo ago
        My most enjoyable and productive experiences with AI so far have looked more like pair-programming than agent-based vibe coding. That is to say, I care about the details, and I want to read, understand, edit, and curate the codebase. I find that if I'm not limiting AI to relatively small enhancements per request-review cycle (100 or so LOC), then when things inevitably go off the rails, I'm in a deep hole that takes a long time to climb out of. I haven't tried out Junie yet, but the concept seems pretty compelling to me. I want a good IDE for the language I'm using, and I'd like an AI that's well integrated and trained on delegating to it for algorithmic/deterministic transforms (e.g. IDE-driven refactorings).
jauntywundrkind 6mo ago
I'm surprised the interconnect per system is so slow? 6x 200Gb feels barely competitive. Same as last year. Trainium3 and Maia 200 are 2.5 and 2.8Tb/s vs this 1.2Tb/s. Maia is 6 stacks of HBMe3, so ratio of mem:interconnect bandwidth is really falling behind here. Notably Maia is also, like TPU, high radix.
- nsteel 6mo ago
  Isn't it 6x 200Gb octals? An octal being 8x 200Gb lanes. So 9.6Tbps?
  - jauntywundrkind 6mo ago
    Thanks. Yeah uhhh the table here says 19.2Tb/s scale up per chip??? Uhhhhhhhh. This math is still not mathing for me. But that makes much much more sense. Google just wildly ahead of literally everyone here.
    - nsteel 6mo ago
      19.2Tb is 2 x 9.6Tbps. Some (most) companies count Tx and Rx separately despite it making no sense in the context of serdes lanes. Stupid marketing in my opinion. I don't think they are meaningfully ahead, it's more to do with what's available at the time. 200/224G is only just coming available this year. The others will have the same in their next product announcements.
      - jauntywundrkind 6mo ago
        Thanks again. Very PS, rad to see you putting commits into mopidy still. I haven't touched mpd like systems in a decade but they have a fond place in my heart!!
deleted 6mo ago
[deleted]
dist-epoch 6mo ago
If you think RAM prices are going to come down, think again. New pods use 10x as much RAM as previous generation.
smlacy 6mo ago
> TPU 8i pairs 288 GB of high-bandwidth memory with 384 MB of on-chip SRAM Wow. Just Wow. I presume that's for each chip, and there are 1152 chips in a pod so that's 331TB HBM and 442TB SRAM per pod. Just wow.
- manquer 6mo ago
  That would be 442GB not TB of SRAM. Aggregating on-chip SRAM is not a useful metric. AFAIK there are no interconnect between two chips at SRAM level so they cannot be shared .
grizzles 6mo ago
Are they refreshing the coral project with these? The coral project for edge ai apps seems like it needs a refresh.
- pnw 6mo ago
  It doesn't seem like it given the form factor. From what I understand, Google let their own hardware efforts die and handed off new Coral hardware to third parties. They announced this Synaptics coral board last month, but you can't buy it anywhere AFAIK. I'm guessing it's going to be a lot more expensive than the original hw. https://www.synaptics.com/products/embedded-processors/sl2610-product-line https://www.synaptics.com/products/embedded-processors/sl261...
- momojo 6mo ago
  I was wondering the same thing. Maybe with the way gemma4 and intelligence density is going, they don't predict the need for NPU's?
yamajun93 6mo ago
I wonder how the focus of agentic-ai differs from that of "normal" LLMs calls in terms of hardware. Does this just provide faster TPU, or does it support it in other ways?
deleted 6mo ago
[deleted]

454 points by xnx 6mo ago

https://cloud.google.com/blog/products/compute/tpu-8t-and-tpu-8i-technical-deep-dive https://cloud.google.com/blog/products/compute/tpu-8t-and-tp...

TheMrZZ 6mo ago

> A single TPU 8t superpod now scales to 9,600 chips and two petabytes of shared high bandwidth memory, with double the interchip bandwidth of the previous generation. This architecture delivers 121 ExaFlops of compute and allows the most complex models to leverage a single, massive pool of memory. This seems impressive. I don't know much about the space, so maybe it's not actually that great, but from my POV it looks like a competitive advantage for Google.

cyanydeez 6mo ago
it is. itll still not create AGI without some breakthrough in instruction vs data separation of concerns
- lugu 6mo ago
  You can park a lot there. No offence but I love how AGI doesn't mean anything. It used to be that AI was a goal post. Now it is AGI. We could use characters from sci-fi culture to describe milestones. In order to achieve robocop level, we must solve the instruction vs data problem.
  - lokar 6mo ago
    Thus it always was. I’m old enough to remember when “if AI could beat a grandmaster at chess” was considered the finish line.
    - sanex 6mo ago
      Over fitting to the benchmarks since 1996
      - deleted 6mo ago
        [deleted]
    - emp17344 6mo ago
      Well, yeah… turns out that goal wasn’t a good indicator for AGI, so we re-evaluated. That’s changing your hypothesis in the face of evidence, not “moving the goalposts” in the fallacious sense.
      - Mistletoe 6mo ago
        What’s the indicator for AGI now? We are so far past the Turing Test it isn’t funny. In fact the models now are too intelligent, you would never think a human would have that much knowledge quickly about a subject you chose at random.
  - lowbloodsugar 6mo ago
    Robocop was a human brain in a suit! Don't give them any ideas!
- scottyah 6mo ago
  In what way do we not already have AGI?
  - airstrike 6mo ago
    in all of them
    - Der_Einzige 6mo ago
      Peter norvig is smarter than you are: https://www.noemamag.com/artificial-general-intelligence-is-already-here/ https://www.noemamag.com/artificial-general-intelligence-is-...
- omcnoe 6mo ago
  Does non-artificial intelligence have clean instruction/data separation?

NoiseBert69 6mo ago

That cooling system looks crazy. What an unbelievable density.

Keyframe 6mo ago

As others have been capturing news cycle eyes, seems to me Google has been going from strength to strength quietly in the background capturing consumer market share and without much (any?) infrastructure problems considering they're so vertically integrated in AI since day one? At one point they even seemed like a lost cause, but they're like a tide.. just growing all around.

deleted 6mo ago
[deleted]
baq 6mo ago
you've never tried to use gemini 3 I guess - that thing was so unreliable it might as well not be offered; there's also a reason why everybody here is excited for claude and codex, but not really for antigravity. that said, I actually agree: google IMHO silently dominates the 'normie business' chatbot area. gemini is low key great for day to day stuff.
- sunaookami 6mo ago
  Yeah Gemini is not usable outside of the gemini.google.com homepage and AI Studio. Gemini CLI does not work at all and all the models are constantly overloaded or time out. Got a trial month and there was not a single day I could actually work with Gemini in a coding harness, 3.1 Pro was never reachable. Awful service, imagine if I'd have paid for this. Additionally since roughly two weeks Gemini 2.5 Flash (a stable GA model) is constantly throwing backend errors on Google's side when Grounding is on.
deleted 6mo ago
[deleted]
youniverse 6mo ago
Yeah I think there will be a time in a few years (1-2?) when both Google and Apple will get to eat their cake. They aren't playing the same game of speed running unpolished product releases every month to double their valuation. They have time to think and observe and put out something really polished. At least that's the hope! :)

amazingamazing 6mo ago

If ai ends up having a winner I struggle to see how it doesn’t end with Google winning because they own the entire stack, or Apple because they will have deployed the most potentially AI capable edge sites.

solarkraft 6mo ago
I see a significant chance that they’ll continue to blunder the product side. It might still not matter because of their massive distribution, but leaves them open to disruption by a better product (think IE vs. Chrome).
davebren 6mo ago
I think the winner will be local model wrappers that are designed to do specific tasks well like search without being anthropomorphized sycophants.

aliljet 6mo ago

The real problem is that scientists doing this sort of early work more often than not want to burn hardware under their desks. Renting infrastructure in Google cloud isn't the only way...

tjwebbnorfolk 6mo ago
You need to have an awfully big desk to stick compute under in order to run a training workload of any interest...

nickandbro 6mo ago

I am curious what workloads Citadel Securities is running on these TPUs? Are you telling me they need the latest TPUs for market insights?

vibe42 6mo ago
Training their own, closed, internal models on their own data sets? Probably a good way to squeeze out some market trading signals.
- nickandbro 6mo ago
  Reminds me of when hedge funds started laying increasingly shorter fiber-optic cable lines to achieve the lowest possible latency for high-frequency trading.
- written-beyond 6mo ago
  I thought these TPUs were primarily used for inference?
  - vlovich123 6mo ago
    TPU8t is for training. But even still, once you’ve trained, you need to run the model too. And these kinds of models already have a huge latency hit so there’s not much hurting running it away from the trading switches.
  - knowaveragejoe 6mo ago
    As the article states, there's both training and inference dedicated chips.
dist-epoch 6mo ago
Not Citadel, but Jane Street is training LLMs for trading: https://www.janestreet.com/join-jane-street/machine-learning/ https://www.janestreet.com/join-jane-street/machine-learning... > We build on the latest papers in LLMs, computer vision, RL, training libraries, cuda kernels, or whatever else we need to train good models. > We invent our own set of architectures and optimizations that work for trading.

pmb 6mo ago

At this point, when you are doing big AI you basically have to buy it from NVidia or rent it from Google. And Google can design their chips and engine and systems in a whole-datacenter context, centralizing some aspects that are impossible for chip vendors to centralize, so I suspect that when things get really big, Google's systems will always be more cost-efficient. (disclosure: I am long GOOG, for this and a few other reasons)

sigmoid10 6mo ago
I'd bet that too if their management wasn't so incredibly uninspiring. Like, Apple under Cook was also pretty mild and a huge step down from Jobs, but Google feels like it fell off a cliff. If it wasn't for OpenAI releasing ChatGPT, they might still be sitting on that tech while only testing it internally. Now it drives their entire chip R&D.
- WarmWash 6mo ago
  To be fair, I don't think any of the AI players wanted what OAI did. Sam grabbed first mover at the cost of this insane race everyone else got forced into.
- hkpack 6mo ago
  I am not fan of the era when CEO is expected to be a cult leader type person. Cook did very well in all areas as well as in not trying to create a cult.
- whattheheckheck 6mo ago
  What would an inspiring leader do differently for you?
  - someguyiguess 6mo ago
    Inspire
    - lokar 6mo ago

paulmist 6mo ago

At $15/GB of HBM4 the 331.8TB of HBM4 per pod is 5 million...

nsteel 6mo ago
It's HBM3e
zozbot234 6mo ago
$15/GB is retail price for DIMM sticks. Is HBM4 really that cheap?
- selectodude 6mo ago
  HBM is just DRAM stacked directly next to the die. The expensive part is gluing it on there. The chips themselves are pretty much the same.
  - akelly 6mo ago
    HBM uses about twice as much DRAM silicon per GB due to all the space for interconnect

vibe42 6mo ago

The pics of the cooling system is pretty good sci-fi / cyberpunk / steampunk inspo. If the whole AI bubble spectularly collapes, at least we got a lot of cool pics of custom hardware!

NitpickLawyer 6mo ago
> If the whole AI bubble spectularly collapes Every other news for the past month has been about lacking capacity. Everyone is having scaling issues with more demand than they can cover. Anthropic has been struggling for a few months, especially visible when EU tz is still up and US east coast comes online. Everything grinds to a halt. MS has been pausing new subscriptions for gh Copilot, also because a lack of capacity. And yet people are still on bubble this, collapse that? I don't get it. Is it becoming a meme? Are people seriously seeing something I don't? For the past 3 years models have kept on improving, capabilities have gone from toy to actually working, and there's no sign of stopping. It's weird.
- redsocksfan45 6mo ago
  [dead]
- vibe42 6mo ago
  Both are possible; increasing demand and bubble collapse. The way this could happen is if model commoditization increases - e.g. some AI labs keep publishing large open models that increasingly close the gap to the closed frontier models. Also, if consumer hardware keep getting better and models get so good that most people can get most of their usage satisfied by smaller models running on their laptop, they won't pay a ton for large frontier models.
  - otterley 6mo ago
    I’m going to stick my neck out a bit and predict that model commoditization will never happen as long as humans keep producing new content and innovation for models to train on. Sure, some open models will be good enough to write software against, but that’s but a fraction of the overall market for this technology.

nsteel 6mo ago

This link has more on the architecture: https://cloud.google.com/blog/products/compute/tpu-8t-and-tpu-8i-technical-deep-dive https://cloud.google.com/blog/products/compute/tpu-8t-and-tp...

fulafel 6mo ago

"TPU 8t and TPU 8i deliver up to two times better performance-per-watt over the previous generation" sounds impressive especially as the previous generation is so recent (2025). Interesting that there's separate inference and training focused hardware. Do companies using NV hardware also use different hardware for each task or is their compute more fungible?

dataking 6mo ago
Vera Rubin will have Groq chips focused on fast inference so it points toward a trend. Also, with energy needs so high, why not reach for every feasible optimization?
xnx 6mo ago
Nvidia said in March that they're working on specialized inference hardware, but they don't have any right now. You can do inference from Nvidia's current hardware offerings, but it's not as efficient.
- FuriouslyAdrift 6mo ago
  AMD has been doing inference chips for many years and are the leader for HPC. https://www.amd.com/en/products/accelerators/instinct.html https://www.amd.com/en/products/accelerators/instinct.html
zozbot234 6mo ago
The "training" chips will probably be quite usable for slower, higher-throughput inference at scale. I expect that to be quite popular eventually for non-time-sensitive uses.
electroly 6mo ago
I can't answer for NVIDIA but AWS has its own training and inference chips, and word on the street is the inference chips are too weak, so some companies are running inference on the training chips.
- 6mo ago

cmptrnerd6 6mo ago

Which company is building the silicon for Google? Is it tsmc? What node size? I didn't see it with a quick search, sorry if it was in the post.

wina 6mo ago
tsmc through broadcom
- nsteel 6mo ago
  Apparently it's Broadcom for 8t and Mediatek for 8i. https://wccftech.com/google-splits-tpuv8-strategy-two-chips-broadcom-training-mediatek-inference-duties/ https://wccftech.com/google-splits-tpuv8-strategy-two-chips-...

varispeed 6mo ago

I can't help but think we will be "laughing" at this in 10 years time like we laugh at steam engines or abacus.

iandanforth 6mo ago

Anyone know if these are already powering all of Gemini services, some of them, or none yet? It's hard to tell if this will result in improvements in speed, lower costs, etc, or if those will be invisible, or have already happened.

kamranjon 6mo ago

It's interesting that, of the large inference providers, Google has one of the most inconvenient policies around model deprecation. They deprecate models exactly 1 year after releasing them and force you to move onto their next generation of models. I had assumed, because they are using their own silicon, that they would actually be able to offer better stability, but the opposite seems to be true. Their rate limiting is also much stricter than OpenAI for example. I wonder how much of this is related to these TPU's, vs just strange policy decisions.

gordonhart 6mo ago
It's frustrating how cavalier they are about killing old Gemini releases. My read is that once a new model is serving >90% of volume, which happens pretty quickly as most tools will just run the latest+greatest model, the standard Google cost/benefit analysis is applied and the old thing is unceremoniously switched off. It's actually surprising that they recently extended the EOL date for Gemini 2.5. Google has never been a particularly customer-obsessed company...
- surajrmal 6mo ago
  What benefit is there to sticking on older models? If the API is the same, what are the switching costs?
  - gordonhart 6mo ago
    If you're trying to run repeatable workflows, stability from not changing the model can outweigh the benefits of a smarter new model. The cost can also change dramatically: on top of the higher token costs for Gemini Pro ($1.25/mtok input for 2.5 versus $2/mtok input for 3.1), the newer release also tokenizes images and PDF pages less efficiently by default (>2x token usage per image/page) so you end up paying much much more per request on the newer model. These are somewhat niche concerns that don't apply to most chat or agentic coding use cases, but they're very real and account for some portion of the traffic that still flows to older Gemini releases.
  - 6mo ago

WarmWash 6mo ago

Whats interesting to note, as someone who uses Gemini, ChatGPT, and Claude, is that Gemini consistently uses drastically fewer tokens than the other two. It seems like gemini is where it is because it has a much smaller thinking budget. It's hard to reconcile this because Google likely has the most compute and at the lowest cost, so why aren't they gassing the hell out of inference compute like the other two? Maybe all the other services they provide are too heavy? Maybe they are trying to be more training heavy? I don't know, but it's interesting to see.

someguyiguess 6mo ago
They have to have SOME competitive advantage. What reason is there to use Gemini over Claude or ChatGPT? It's not producing nearly the quality of output.
- magicalhippo 6mo ago
  Well comparing Gemini 3.1 Pro vs ChatGPT 5.4 Pro, it's much faster at replying. Of course, if it actually thinks less then that helps a lot towards that. For most of my personal and work use-cases, I prefer waiting a bit longer for a better answer.
- WarmWash 6mo ago
  I recently did my taxes using all three models (My return is ~50 pages, much more than a standard 1040). GPT (codex) was accurate on the first run and took 12 minutes Gemini (antigravity) missed 1 value because it didn't load the full 1099 pdf (the laziness), but corrected it when prompted. However it only spent 2 minutes on the task. Claude (CC) made all manner of mistakes after waiting overnight for it to finish because it hit my limit before doing so. However claude did the best on the next step of actually filing out the pdf forms, but it ended up not mattering. Ultimately I used gemini in chrome to fill out the forms (freefillableforms.com), but frankly it would have been faster to manually do it copying from the spreadsheets GPT and Gemini output. I also use anti-gravity a lot for small greenfield projects(<5k LOC). I don't notice a difference between gemini and claude, outside usage limits. Besides that I mostly use gemini for it's math and engineering capabilities.

zshn25 6mo ago

[dead]

zshn25 6mo ago

It would be interesting to benchmark a short training / inference run on the latest of TPU vs. NVIDIA GPU per cost basis

jmyeet 6mo ago

In recent discussions about Tim Apple [sic] moving on there was a discussion about whether Apple flopped on AI, which is my opinion. Of course you had the false dichotomy of doing nothing or burning money faster than the US military like OpenAI does. IMHO that happy medium is Google. Not having to pay the NVidia tax will likely be a huge competitive advantage. And nobody builds data centers as cost-effectively as Google. It's kind of crazy to be talking ExaFLOPS and Tb/s here. From some quick Googling: - The first MegaFLOPS CPU was in 1964 - A Cray supercomputer hit GigaFLOPS in 1988 with workstations hitting it in the 1990s. Consumer CPUs I think hit this around 1999 with the Pentium 3 at 1GHz+; - It was the 2010s before we saw off-the-shelf TFLOPS; - It was only last year where a single chip hit PetaFLOPS. I see the IBM Roadrunner hit this in 2008 but that was ~13,000 CPUs so... Obviously this is near 10,000 TPUs to get to ~121 EFLOPS (FP4 admittedly) but that's still an astounding number. IT means each one is doing ~12 PFLOPS (FP4). I saw a claim that Claude Mythos cost ~$10B to train. I personally believe Google can (or soon will be able to) do this for an order of magnitude less at least. I would love to know the true cost/token of Claude, ChatGPT and Gemini. I think you'll find Google has a massive cost advantage here.

knowaveragejoe 6mo ago
> I saw a claim that Claude Mythos cost ~$10B to train. Can you cite this? That seems absurd.
- jmyeet 6mo ago
  I've seen various claims to this (eg [1][2][3]) but nobody reall knows. These may all come from one uunsubstantiated claim. It is I think widely accepted that Mythos is ~10T parameters. I've seen figures that suggest GPT-4 was 1.8T parameters and cost upwards of $100 million to train (also unsubstantiated), in which case the Mythos figure might be inflated and also include development costs. So who really knows? [1]: https://www.softwarereviews.com/research/claude-mythos-preview-and-project-glasswing-what-it-and-security-leaders-need-to-know-now https://www.softwarereviews.com/research/claude-mythos-previ... [2]: https://x.com/duttasomrattwt/status/2041903600516133016 https://x.com/duttasomrattwt/status/2041903600516133016 [3]: https://www.forrester.com/blogs/project-glasswing-the-10-consequences-nobodys-writing-about-yet/ https://www.forrester.com/blogs/project-glasswing-the-10-con...

himata4113 6mo ago

I already felt that gemini 3 proved what is possible if you train a model for efficiency. If I had to guess the pro and flash variants are 5x to 10x smaller than opus and gpt-5 class models. They produce drastically lower amount of tokens to solve a problem, but they haven't seem to have put enough effort into refinining their reasoning and execution as they produce broken toolcalls and generally struggle with 'agentic' tasks, but for raw problem solving without tools or search they match opus and gpt while presumably being a fraction of the size. I feel like google will surprise everyone with a model that will be an entire generation beyond SOTA at some point in time once they go from prototyping to making a model that's not a preview model anymore. All models up till now feel like they're just prototypes that were pushed to GA just so they have something to show to investors and to integrate into their suite as a proof of concept.

ALLTaken 6mo ago
[flagged]
- _boffin_ 6mo ago
  Is your friend on the JAX team?
- neonstatic 6mo ago
  I'm really struggling with terrible bloating today, but I deemed it too dangerous to release.
  - tclancy 6mo ago
    Thank you for your sacrifice. Could you speak to my dog please? You may wish to yell from a distance, actually.
onlyrealcuzzo 6mo ago
> They produce drastically lower amount of tokens to solve a problem, but they haven't seem to have put enough effort into refinining their reasoning and execution as they produce broken toolcalls and generally struggle with 'agentic' tasks, but for raw problem solving without tools or search they match opus and gpt while presumably being a fraction of the size. Agreed, Gemini-cli is terrible compared to CC and even Codex. But Google is clearly prioritizing to have the best AI to augment and/or replace traditional search. That's their bread and butter. They'll be in a far better place to monetize that than anyone else. They've got a 1B+ user lead on anyone - and even adding in all LLMs together, they still probably have more query volume than everyone else put together. I hope they start prioritizing Gemini-cli, as I think they'd force a lot more competition into the space.

SecretDreams 6mo ago

They are missing a header to show the transition in discussion from TPU8t to 8i! Thanks for posting otherwise. Edit: actually, looks like the header got captured as a figure caption on accident.

nicman23 6mo ago

yeah but can you release the sdk for the pixel 10? it was one of then only reasons which i bought this mid phone

bozdemir 6mo ago

[dead]

jjice 6mo ago

I've been saying it, and I'll keep saying it (as someone who has an opinion backed by very little) - I think Google is incredibly well placed for the future with LLMs. Owning your hardware and your entire stack is huge, especially these days with so much demand. Long term, I think they end up doing very well. People clowned so hard on Google for the first two years (until Gemini 2.5 or 3) because it wasn't as good as OpenAI or Anthropic's models, but Google just looked so good for the long game. Another benefit for them: If LLMs end up being a huge bubble that end up not paying the absurd returns the industry expects, they're not kaput. They already own so many markets that this is just an additional thing for them, where as the big AI only labs are probably fucked. All that said: what the hell do I know? Who knows how all of this will play out. I just think Google has a great foundation underneath them that'll help them build and not topple over.

Aissen 6mo ago

Interesting that t8i is both for post-training and inference.

ks2048 6mo ago

For how many times does this article mentions "agentic" and "agents"... Am I correct assume the hardware has nothing to do with "agents"? I assume it's just about a new generation of more efficient transformers / deep-learning layers.

tomassre 6mo ago
There’s issues specific to workflows “agents”. For example many requests in an agent are all on top of the same previous results so context (kv cache) needs to be longer, and they use these massive connected nodes with direct nvme to cache the part of the prompt that’s repeatable. It is about agents in that the design is for long context, many requests where the initial “chunk” is cached but spread across many requests. They don’t call this out specifically but in the technical details like about the sram, how it’s all interconnected nodes in a pod it’s “designed” for it.

geremiiah 6mo ago

TPUs are systolic arrays right? So does that mean that Google is using a hetreogenous cluster compromising both GPUs and TPUs, for workloads that don't map well or at all on TPUs?

thatguysaguy 6mo ago
What sort of workloads are you thinking of?
eklitzke 6mo ago
I can't speak to what every team at Google does, but there are machines with Nvidia GPUs in Borg. However Google charges orgs internally for cpu/memory/gpu/tpu usage and TPUs are *way* more efficient in terms of FLOPS/$ than Nvidia GPUs, so there is a *huge* incentive for teams to use TPUs if they can, especially for teams operating large products.

mlmonkey 6mo ago

FTA: > One pod of TPU 8t is 121 ExaFlops; or 121,000 PetaFlops. Meanwhile, the compute capacity of the top 10 supercomputers in the entire world is 11,487 Petaflops.[1] I know, I know, not the same flops, yada yada, but still. Just 1 pod alone is quite a beast. Edit: [1] https://top500.org/lists/top500/2025/11/ https://top500.org/lists/top500/2025/11/

dist-epoch 6mo ago
Not only not the same size, 4-bit flops versus 64-bit flops, but not the same programmability either. the TPUs can do just matrix-multiplications and some supporting math. Otherwise bitcoin mining rigs dwarf everything, if you just want to count raw operations per second.
stabbles 6mo ago
Other than TPUs they're also planning for 960,000 Rubin GPUs [1] which can do 33 teraflops fp64 each, so over 30 classical exaflops, and with emulation it could be more than 100 exaflops. [1] https://blogs.nvidia.com/blog/google-cloud-agentic-physical-ai-factories/ https://blogs.nvidia.com/blog/google-cloud-agentic-physical-...

delbronski 6mo ago

I’ve been using Gemini with Junie (jetbrains attempt at Claude code). While Junie is nowhere near as good as Claude Code, it is way ahead of the current Google tooling. I get quite good consistent results for pretty cheap with this combo.

rtaylorgarlock 6mo ago
Do you think that Junie specifically inside the context of jetbrains IDEs and tooling matches the competitors?
- delbronski 6mo ago
  I think so. From my experience Claude/codex tooling really excels at vibe coding the whole thing. You give it a folder and just say: now make it do this. And you don’t really care for the code. Junie tooling excels when you are more involved. Like, look in these two files, add this specific functionality, in this specific way. Junie is usually a lot faster and to the point. Very simple tooling , it just works for this workflow. But it breaks for the “code the whole thing for me” workflow.
  - chaoz_ 6mo ago
    It surprises me how JetBrains managed to lose such a great market opportunity. I don't think they ever going to be able to re-claim large chunk of developers who are now fine with thin VSCode-like + Terminal for non-JVM languages. Perfect example of how large corp with research capacity failed to navigate their product changes.
    - foobazgt 6mo ago
      My most enjoyable and productive experiences with AI so far have looked more like pair-programming than agent-based vibe coding. That is to say, I care about the details, and I want to read, understand, edit, and curate the codebase. I find that if I'm not limiting AI to relatively small enhancements per request-review cycle (100 or so LOC), then when things inevitably go off the rails, I'm in a deep hole that takes a long time to climb out of. I haven't tried out Junie yet, but the concept seems pretty compelling to me. I want a good IDE for the language I'm using, and I'd like an AI that's well integrated and trained on delegating to it for algorithmic/deterministic transforms (e.g. IDE-driven refactorings).

jauntywundrkind 6mo ago

I'm surprised the interconnect per system is so slow? 6x 200Gb feels barely competitive. Same as last year. Trainium3 and Maia 200 are 2.5 and 2.8Tb/s vs this 1.2Tb/s. Maia is 6 stacks of HBMe3, so ratio of mem:interconnect bandwidth is really falling behind here. Notably Maia is also, like TPU, high radix.

nsteel 6mo ago
Isn't it 6x 200Gb octals? An octal being 8x 200Gb lanes. So 9.6Tbps?
- jauntywundrkind 6mo ago
  Thanks. Yeah uhhh the table here says 19.2Tb/s scale up per chip??? Uhhhhhhhh. This math is still not mathing for me. But that makes much much more sense. Google just wildly ahead of literally everyone here.
  - nsteel 6mo ago
    19.2Tb is 2 x 9.6Tbps. Some (most) companies count Tx and Rx separately despite it making no sense in the context of serdes lanes. Stupid marketing in my opinion. I don't think they are meaningfully ahead, it's more to do with what's available at the time. 200/224G is only just coming available this year. The others will have the same in their next product announcements.
    - jauntywundrkind 6mo ago
      Thanks again. Very PS, rad to see you putting commits into mopidy still. I haven't touched mpd like systems in a decade but they have a fond place in my heart!!

deleted 6mo ago

[deleted]

dist-epoch 6mo ago

If you think RAM prices are going to come down, think again. New pods use 10x as much RAM as previous generation.

smlacy 6mo ago

> TPU 8i pairs 288 GB of high-bandwidth memory with 384 MB of on-chip SRAM Wow. Just Wow. I presume that's for each chip, and there are 1152 chips in a pod so that's 331TB HBM and 442TB SRAM per pod. Just wow.

manquer 6mo ago
That would be 442GB not TB of SRAM. Aggregating on-chip SRAM is not a useful metric. AFAIK there are no interconnect between two chips at SRAM level so they cannot be shared .

grizzles 6mo ago

Are they refreshing the coral project with these? The coral project for edge ai apps seems like it needs a refresh.

pnw 6mo ago
It doesn't seem like it given the form factor. From what I understand, Google let their own hardware efforts die and handed off new Coral hardware to third parties. They announced this Synaptics coral board last month, but you can't buy it anywhere AFAIK. I'm guessing it's going to be a lot more expensive than the original hw. https://www.synaptics.com/products/embedded-processors/sl2610-product-line https://www.synaptics.com/products/embedded-processors/sl261...
momojo 6mo ago
I was wondering the same thing. Maybe with the way gemma4 and intelligence density is going, they don't predict the need for NPU's?

yamajun93 6mo ago

I wonder how the focus of agentic-ai differs from that of "normal" LLMs calls in terms of hardware. Does this just provide faster TPU, or does it support it in other ways?

deleted 6mo ago

[deleted]