7 ms·

TPU Deep Dive

451 points by transpute 1y ago

jan_Sate 1y ago
[flagged]
almostgotcaught 1y ago
> In essence, caches allow hardware to be flexible and adapt to a wide range of applications. This is a large reason why GPUs are very flexible hardware (note: compared to TPUs). this is correct but mis-stated - it's not the caches themselves that cost energy but MMUs that automatically load/fetch/store to cache on "page faults". TPUs don't have MMUs and furthermore are a push architecture (as opposed to pull).
RossBencina 1y ago
Can you suggest a good reference for understanding which algorithms map well onto the regular grid systolic arrays used by TPUs? The fine article says dese matmul and convolution are good, but is there anything else? Eigendecomposition? SVD? matrix exponential? Solving Ax = b or AX = B? Cholesky?
- WithinReason 1y ago
  Anything that you can express as 128x128 (but ideally much larger) dense matrix multiplication and nothing else
- musebox35 1y ago
  I think https://jax-ml.github.io/scaling-book/ https://jax-ml.github.io/scaling-book/ is one of the best references to go through. It details how single device and distributed computations map to TPU hardware features. The emphasis is on mapping the transformer computations, both forwards and backwards, so requires some familiarity with how transformer networks are structured.
- cdavid 1y ago
  SVD/eigendecomposition will often boil down to making many matmul (e.g. when using Krylov-based methods, e.g. Arnoldi, Krylov-schur, etc.), so I would expect TPU to work well there. GMRES, one method to solve Ax = b is also based on Arnoldi decomp.
- Straw 1y ago
  You can do all of these in terms of matmul to some extent: Solving AX=B can be done with Newton's method to invert A, which boils down to matmuls. Matrix exponential is normally done with matmuls- the scale down, Taylor/Pade and square approach. Why do you need Cholesky? It's typically a means to an end, and when matmul is your primitive, you reach for it much less often. Eigendecomposition is hard. If we limit ourselves to symmetric, we could use a blocked Jacobi algorithm where we run a non-matmul Jacobi to do 128x128 off-diagonal blocks and then use the matmul unit to apply to the whole matrix- for large enough matrices, still bottlenecked on matmul. SVD we can get from Polar decomposition, which has purely-matmul iterations, and symmetric eigendecomposition. One does have to watch out for numerical stability and precision very carefully when doing all these!
  - RossBencina 1y ago
    Cholesky for generating so-called sigma points in the Unscented Transformation.
serf 1y ago
does that cooling channel have a NEMA stepper on it as a pump or metering valve?[0] If so, wild. That seems like overkill. [0]: https://henryhmko.github.io/posts/tpu/images/tpu_tray.png https://henryhmko.github.io/posts/tpu/images/tpu_tray.png
- fellowmartian 1y ago
  definitely closed-loop, might even be a servo
frays 1y ago
How can someone have this level of knowledge about TPUs without working at Google?
- musebox35 1y ago
  From the acknowledgment at the end, I guess the author has access to TPUs through https://sites.research.google/trc/about/ https://sites.research.google/trc/about/ This is not the only way though. TPUs are available to companies operating on GCP as an alternative to GPUs with a different price/performance point. That is another way to get hands-on experience with TPUs.
  - erwincoumans 1y ago
    A quick free way to access TPUs is through https://colab.research.google.com https://colab.research.google.com, Runtime / Change Runtime Type / v2-8 TPU
- ipsum2 1y ago
  Everything thats in the blog post is basically well known already. Google publishes papers and gives talks about their TPUs. Many details are lacking though, and require some assumptions/best guesses. Jax and XLA are (partially) open source and give clues about how TPUs work under the hood as well. https://arxiv.org/abs/2304.01433 https://arxiv.org/abs/2304.01433 https://jax-ml.github.io/scaling-book/ https://jax-ml.github.io/scaling-book/
- antognini 1y ago
  There's a pretty detailed description of TPUs in the latest edition of Hennessy's Computer Architecture. (Hennessy was also involved with the design of the early TPU architectures if I recall right.)
ariwilson 1y ago
Cool article!
sgt101 1y ago
ELI5: how (specifically) do GPU and TPU optimisations effect determinism in LLMs? Or is this all a myth?
- barrkel 1y ago
  LLMs are generally deterministic. The token sampling step is usually randomized to some degree because it gets better results (creativity) and helps avoid loops, but you can turn that off (temp zero for simple samplers).
  - perching_aix 1y ago
    + can also just pin the seed instead, right?
  - sgeisenh 1y ago
    This is an oversimplification. When distributed, the nondeterministic order of additions during reductions can produce nondeterministic results due to floating point error. It’s nitpicking for sure, but it causes real challenges for reproducibility, especially during model training.
  - Der_Einzige 1y ago
    This belief (LLMs are deterministic except for samplers) is very wrong and will get you into hilariously large amounts of trouble for assuming it's true. Also greedy sampling considered harmful: https://arxiv.org/abs/2506.09501 https://arxiv.org/abs/2506.09501 From the abstract: "For instance, under bfloat16 precision with greedy decoding, a reasoning model like DeepSeek-R1-Distill-Qwen-7B can exhibit up to 9% variation in accuracy and 9,000 tokens difference in response length due to differences in GPU count, type, and evaluation batch size. We trace the root cause of this variability to the non-associative nature of floating-point arithmetic under limited numerical precision. This work presents the first systematic investigation into how numerical precision affects reproducibility in LLM inference. Through carefully controlled experiments across various hardware, software, and precision settings, we quantify when and how model outputs diverge. Our analysis reveals that floating-point precision—while critical for reproducibility—is often neglected in evaluation practices."
lanthissa 1y ago
can someone help me understand how the following can be true: 1. TPU's are a serious competitor to nvidia chips. 2. Chip makers with the best chips are valued at 1-3.5T. 3. Google's market cap is 2T. 4. It is correct for google to not sell TPU's. i have heard the whole, its better to rent them thing, but if they're actually good selling them is almost as good a business as every other part of the company.
- mft_ 1y ago
  If they think they’ve got a competitive advantage vs. GPUs which benefits one of their core products, it would make sense to retain that competitive advantage for the long term, no?
  - Uehreka 1y ago
    No. If they sell the TPUs for “what they’re worth”, they get to reap a portion of the benefit their competitors would get from them. There’s money they could be making that they aren’t. Or rather, there would be if TPUs were that good in practice. From the other comments it sounds like TPUs are difficult to use for a lot of workloads, which probably leads to the real explanation: No one wants to use them as much as Google does, so selling them for a premium price as I mentioned above won’t get them many buyers.
- smokel 1y ago
  Selling them and supporting that in the field requires quite some infrastructure you'd have to build. Why go through all that trouble if you already make higher margins renting them out? Also, if they are so good, it's best to not level the playing field by sharing that with your competitors. Also "chip makers with the best chips" == Nvidia, there aren't many others. And Alphabet does more than just produce TPUs.
  - matt-p 1y ago
    Does Google cloud offer them on a "aws outpost" style model? I think that plus cloud access is probably the easiest and ' best ' way to offer them. Last thing you need to be dealing with is super micro, gigabyte etc building a box for them and so on - I can definitely understand not selling the raw chip.
cdg007 1y ago
What will competitors say?
b0a04gl 1y ago
tpu's predictable latency under scale. when you control the compiler, the runtime, the interconnect and the chip, you eliminate so much variance that you can actually schedule jobs efficiently at data center scale. so the obvious question why haven't we seen anyone outside Google replicate this full vertical stack yet? is it because the hardware's hard or because no one has nailed the compiler/runtime contract at that scale?
- kevindamm 1y ago
  you mean other than NVidia and AMD?
- transpute 1y ago
  Groq, https://en.wikipedia.org/wiki/Groq https://en.wikipedia.org/wiki/Groq & https://news.ycombinator.com/item?id=44345738 https://news.ycombinator.com/item?id=44345738
Neywiny 1y ago
What's not mentioned is a comparison vs FPGAs. You can have a systolic, fully pipelined system for any data processing not just vectorized SIMD. Every primitive is able to work independently of everything else. For example, if you have 240 DSP slices (which is far from outrageous on low scale), a perfect design could use those as 240 cores at 100% throughput. No memory, caching, decoding, etc overhead.
- adrian_b 1y ago
  True, but FPGAs are suitable only for things that will not be produced in great numbers, because their cost and power consumption are many times higher than those of an ASIC. For a company of the size of Google, the development costs for a custom TPU are quickly recovered. Comparing a Google TPU with an FPGA is like comparing an injection-moulded part with a 3D-printed part. Unfortunately, the difference in performance between FPGAs and ASICs has greatly increased in recent years, because the FPGAs have remain stuck on relatively ancient CMOS manufacturing processes, which are much less efficient than the state-of-the-art CMOS manufacturing processes.
  - Neywiny 1y ago
    When you can ASIC, yes, do an ASIC. But my point was that there was a lot of GPU comparison. GPUs are also not ASICs relative to AI.
    - QuadmasterXLII 1y ago
      They’re close, they’re basically matmul asics
      - Neywiny 1y ago
        Arguably so are the DSP heavy FPGAs. And the unused logic will have a minimal static power draw relative to the unused but clocked G-only parts of the GPU.
        1y ago
cheptsov 1y ago
It’s so ridiculous to see TPUs being compared to NVIDIA GPUs. IMO proprietary chips such as TPU had no future sure to the monopoly on the cloud services. There is no competition across the cloud services providers. The only way to access TPUs is through GCP. As the result nobody wants to use them regardless of the technology. This is the biggest fault of GCP. Further the road, the gap between NVIDIA GPUs and Google TPUs (call it „moat“ or CUDA) is going to grow. The opposite situation is with AMD which are avoiding the mistakes of Google. My hope though is that AMD doesn’t start to compete with cloud service providers, e.g. by introducing their own cloud.
- hiddencost 1y ago
  TPUs will thrive regardless of public adoption; Google's internal demand for TPU is such that they could buy every TPU ever produced.
  - roughly 1y ago
    One thing worth note here - TPUs are optimized for a fairly constrained set of operations. Google’s had good success with them, but, like many of the other Google architectural choices, this will constrain Google’s technical choice space in the future - if they’ve gone all in on TPUs, future Google machine learning projects will be using the sets of operations the TPUs excel at because that’s what Google has a lot of, not necessarily because that’s the optimal choice. This will have knock-on effects across the industry due to Google’s significant influence on industry practice and technical direction.
- hustwindmaple1 1y ago
  Every major Cloud vendor is trying to develop their custom AI ASIC. Putting Google aside, Amazon has trainium/inferentia, which Anthropic uses quite extensively. Microsoft is doing sth. similar, although they are quite behind. OpenAI is doing it. Meta is doing it. That's why the stock price of Broadcom/Marvell soared.
trostaft 1y ago
Excellent write up, thank you. The benefits section was illustrative
mdaniel 1y ago
Related: OpenTPU: Open-Source Reimplementation of Google Tensor Processing Unit (TPU) - https://news.ycombinator.com/item?id=44111452 https://news.ycombinator.com/item?id=44111452 - May, 2025 (23 comments)
wkat4242 1y ago
Haha I thought this was about 3D printing with thermostatic poly urethane. It's one of the harder materials to print and it also took me some time to get my head around it.
curtisszmania 1y ago
[dead]

451 points by transpute 1y ago

[flagged]

> In essence, caches allow hardware to be flexible and adapt to a wide range of applications. This is a large reason why GPUs are very flexible hardware (note: compared to TPUs). this is correct but mis-stated - it's not the caches themselves that cost energy but MMUs that automatically load/fetch/store to cache on "page faults". TPUs don't have MMUs and furthermore are a push architecture (as opposed to pull).

RossBencina 1y ago

Can you suggest a good reference for understanding which algorithms map well onto the regular grid systolic arrays used by TPUs? The fine article says dese matmul and convolution are good, but is there anything else? Eigendecomposition? SVD? matrix exponential? Solving Ax = b or AX = B? Cholesky?

WithinReason 1y ago
Anything that you can express as 128x128 (but ideally much larger) dense matrix multiplication and nothing else
musebox35 1y ago
I think https://jax-ml.github.io/scaling-book/ https://jax-ml.github.io/scaling-book/ is one of the best references to go through. It details how single device and distributed computations map to TPU hardware features. The emphasis is on mapping the transformer computations, both forwards and backwards, so requires some familiarity with how transformer networks are structured.
cdavid 1y ago
SVD/eigendecomposition will often boil down to making many matmul (e.g. when using Krylov-based methods, e.g. Arnoldi, Krylov-schur, etc.), so I would expect TPU to work well there. GMRES, one method to solve Ax = b is also based on Arnoldi decomp.
Straw 1y ago
You can do all of these in terms of matmul to some extent: Solving AX=B can be done with Newton's method to invert A, which boils down to matmuls. Matrix exponential is normally done with matmuls- the scale down, Taylor/Pade and square approach. Why do you need Cholesky? It's typically a means to an end, and when matmul is your primitive, you reach for it much less often. Eigendecomposition is hard. If we limit ourselves to symmetric, we could use a blocked Jacobi algorithm where we run a non-matmul Jacobi to do 128x128 off-diagonal blocks and then use the matmul unit to apply to the whole matrix- for large enough matrices, still bottlenecked on matmul. SVD we can get from Polar decomposition, which has purely-matmul iterations, and symmetric eigendecomposition. One does have to watch out for numerical stability and precision very carefully when doing all these!
- RossBencina 1y ago
  Cholesky for generating so-called sigma points in the Unscented Transformation.

serf 1y ago

does that cooling channel have a NEMA stepper on it as a pump or metering valve?[0] If so, wild. That seems like overkill. [0]: https://henryhmko.github.io/posts/tpu/images/tpu_tray.png https://henryhmko.github.io/posts/tpu/images/tpu_tray.png

fellowmartian 1y ago
definitely closed-loop, might even be a servo

frays 1y ago

How can someone have this level of knowledge about TPUs without working at Google?

musebox35 1y ago
From the acknowledgment at the end, I guess the author has access to TPUs through https://sites.research.google/trc/about/ https://sites.research.google/trc/about/ This is not the only way though. TPUs are available to companies operating on GCP as an alternative to GPUs with a different price/performance point. That is another way to get hands-on experience with TPUs.
- erwincoumans 1y ago
  A quick free way to access TPUs is through https://colab.research.google.com https://colab.research.google.com, Runtime / Change Runtime Type / v2-8 TPU
ipsum2 1y ago
Everything thats in the blog post is basically well known already. Google publishes papers and gives talks about their TPUs. Many details are lacking though, and require some assumptions/best guesses. Jax and XLA are (partially) open source and give clues about how TPUs work under the hood as well. https://arxiv.org/abs/2304.01433 https://arxiv.org/abs/2304.01433 https://jax-ml.github.io/scaling-book/ https://jax-ml.github.io/scaling-book/
antognini 1y ago
There's a pretty detailed description of TPUs in the latest edition of Hennessy's Computer Architecture. (Hennessy was also involved with the design of the early TPU architectures if I recall right.)

ariwilson 1y ago

Cool article!

sgt101 1y ago

ELI5: how (specifically) do GPU and TPU optimisations effect determinism in LLMs? Or is this all a myth?

barrkel 1y ago
LLMs are generally deterministic. The token sampling step is usually randomized to some degree because it gets better results (creativity) and helps avoid loops, but you can turn that off (temp zero for simple samplers).
- perching_aix 1y ago
  + can also just pin the seed instead, right?
- sgeisenh 1y ago
  This is an oversimplification. When distributed, the nondeterministic order of additions during reductions can produce nondeterministic results due to floating point error. It’s nitpicking for sure, but it causes real challenges for reproducibility, especially during model training.
- Der_Einzige 1y ago
  This belief (LLMs are deterministic except for samplers) is very wrong and will get you into hilariously large amounts of trouble for assuming it's true. Also greedy sampling considered harmful: https://arxiv.org/abs/2506.09501 https://arxiv.org/abs/2506.09501 From the abstract: "For instance, under bfloat16 precision with greedy decoding, a reasoning model like DeepSeek-R1-Distill-Qwen-7B can exhibit up to 9% variation in accuracy and 9,000 tokens difference in response length due to differences in GPU count, type, and evaluation batch size. We trace the root cause of this variability to the non-associative nature of floating-point arithmetic under limited numerical precision. This work presents the first systematic investigation into how numerical precision affects reproducibility in LLM inference. Through carefully controlled experiments across various hardware, software, and precision settings, we quantify when and how model outputs diverge. Our analysis reveals that floating-point precision—while critical for reproducibility—is often neglected in evaluation practices."

lanthissa 1y ago

can someone help me understand how the following can be true: 1. TPU's are a serious competitor to nvidia chips. 2. Chip makers with the best chips are valued at 1-3.5T. 3. Google's market cap is 2T. 4. It is correct for google to not sell TPU's. i have heard the whole, its better to rent them thing, but if they're actually good selling them is almost as good a business as every other part of the company.

mft_ 1y ago
If they think they’ve got a competitive advantage vs. GPUs which benefits one of their core products, it would make sense to retain that competitive advantage for the long term, no?
- Uehreka 1y ago
  No. If they sell the TPUs for “what they’re worth”, they get to reap a portion of the benefit their competitors would get from them. There’s money they could be making that they aren’t. Or rather, there would be if TPUs were that good in practice. From the other comments it sounds like TPUs are difficult to use for a lot of workloads, which probably leads to the real explanation: No one wants to use them as much as Google does, so selling them for a premium price as I mentioned above won’t get them many buyers.
smokel 1y ago
Selling them and supporting that in the field requires quite some infrastructure you'd have to build. Why go through all that trouble if you already make higher margins renting them out? Also, if they are so good, it's best to not level the playing field by sharing that with your competitors. Also "chip makers with the best chips" == Nvidia, there aren't many others. And Alphabet does more than just produce TPUs.
- matt-p 1y ago
  Does Google cloud offer them on a "aws outpost" style model? I think that plus cloud access is probably the easiest and ' best ' way to offer them. Last thing you need to be dealing with is super micro, gigabyte etc building a box for them and so on - I can definitely understand not selling the raw chip.

cdg007 1y ago

What will competitors say?

b0a04gl 1y ago

tpu's predictable latency under scale. when you control the compiler, the runtime, the interconnect and the chip, you eliminate so much variance that you can actually schedule jobs efficiently at data center scale. so the obvious question why haven't we seen anyone outside Google replicate this full vertical stack yet? is it because the hardware's hard or because no one has nailed the compiler/runtime contract at that scale?

kevindamm 1y ago
you mean other than NVidia and AMD?
transpute 1y ago
Groq, https://en.wikipedia.org/wiki/Groq https://en.wikipedia.org/wiki/Groq & https://news.ycombinator.com/item?id=44345738 https://news.ycombinator.com/item?id=44345738

Neywiny 1y ago

What's not mentioned is a comparison vs FPGAs. You can have a systolic, fully pipelined system for any data processing not just vectorized SIMD. Every primitive is able to work independently of everything else. For example, if you have 240 DSP slices (which is far from outrageous on low scale), a perfect design could use those as 240 cores at 100% throughput. No memory, caching, decoding, etc overhead.

adrian_b 1y ago
True, but FPGAs are suitable only for things that will not be produced in great numbers, because their cost and power consumption are many times higher than those of an ASIC. For a company of the size of Google, the development costs for a custom TPU are quickly recovered. Comparing a Google TPU with an FPGA is like comparing an injection-moulded part with a 3D-printed part. Unfortunately, the difference in performance between FPGAs and ASICs has greatly increased in recent years, because the FPGAs have remain stuck on relatively ancient CMOS manufacturing processes, which are much less efficient than the state-of-the-art CMOS manufacturing processes.
- Neywiny 1y ago
  When you can ASIC, yes, do an ASIC. But my point was that there was a lot of GPU comparison. GPUs are also not ASICs relative to AI.
  - QuadmasterXLII 1y ago
    They’re close, they’re basically matmul asics
    - Neywiny 1y ago
      Arguably so are the DSP heavy FPGAs. And the unused logic will have a minimal static power draw relative to the unused but clocked G-only parts of the GPU.
      - 1y ago

cheptsov 1y ago

It’s so ridiculous to see TPUs being compared to NVIDIA GPUs. IMO proprietary chips such as TPU had no future sure to the monopoly on the cloud services. There is no competition across the cloud services providers. The only way to access TPUs is through GCP. As the result nobody wants to use them regardless of the technology. This is the biggest fault of GCP. Further the road, the gap between NVIDIA GPUs and Google TPUs (call it „moat“ or CUDA) is going to grow. The opposite situation is with AMD which are avoiding the mistakes of Google. My hope though is that AMD doesn’t start to compete with cloud service providers, e.g. by introducing their own cloud.

hiddencost 1y ago
TPUs will thrive regardless of public adoption; Google's internal demand for TPU is such that they could buy every TPU ever produced.
- roughly 1y ago
  One thing worth note here - TPUs are optimized for a fairly constrained set of operations. Google’s had good success with them, but, like many of the other Google architectural choices, this will constrain Google’s technical choice space in the future - if they’ve gone all in on TPUs, future Google machine learning projects will be using the sets of operations the TPUs excel at because that’s what Google has a lot of, not necessarily because that’s the optimal choice. This will have knock-on effects across the industry due to Google’s significant influence on industry practice and technical direction.
hustwindmaple1 1y ago
Every major Cloud vendor is trying to develop their custom AI ASIC. Putting Google aside, Amazon has trainium/inferentia, which Anthropic uses quite extensively. Microsoft is doing sth. similar, although they are quite behind. OpenAI is doing it. Meta is doing it. That's why the stock price of Broadcom/Marvell soared.

trostaft 1y ago

Excellent write up, thank you. The benefits section was illustrative

mdaniel 1y ago

Related: OpenTPU: Open-Source Reimplementation of Google Tensor Processing Unit (TPU) - https://news.ycombinator.com/item?id=44111452 https://news.ycombinator.com/item?id=44111452 - May, 2025 (23 comments)

wkat4242 1y ago

Haha I thought this was about 3D printing with thermostatic poly urethane. It's one of the harder materials to print and it also took me some time to get my head around it.

curtisszmania 1y ago

[dead]