3 ms·

I was thinking about that (I definitely agree with you on the software and data angle). But when you think about it it's actually a bit more complex. Right now

I was thinking about that (I definitely agree with you on the software and data angle). But when you think about it it's actually a bit more complex. Right now (eg) OpenAI buys GPUs from (eg) NVidia, who buys HBM from Samsung and fabs the card on TSMC. Google instead designs the chip, with I assume a significant amount of assistance of Broadcom - at least in terms of manufacturing, who then buys the HBM from the same supplier(s) and fabs the card with TSMC. So I'm not entirely sure if the margin savings are that huge. I assume Broadcom charges a fair bit to manage the manufacturing process on behalf of Google. Almost certainly a lot less than NVidia would charge in terms of gross profit margins, but Google also has to pay for a lot of engineers to do the work that would be done in NVidia. No doubt it is a saving overall - otherwise they wouldn't do it. But I wonder how dramatic it is. Obviously Google has significant upside in the ability to customise their chips exactly how they want them, but NVidia (and to a lesser extent) AMD probably can source more customer workflows/issues from their broader set of clients. I think "Google makes its own TPUs" makes a lot of people think that the entire operation in house, but in reality they're just doing more design work than the other players. There's still a lot of margin "leaking" through Broadcom, memory suppliers and TSMC so I wonder how dramatic it is really is

coredog64 8mo ago
My take is it's the inference efficiency. It's one thing to have a huge GPU cluster for training, but come inference time you don't need nearly so much. Having the TPU (and models purpose built for TPU) allows for best cost in serving at hyperscale.
- martinald 8mo ago
  Yes potentially - but the OG TPUs were actually very poorly suited for LLM usage - designed for far smaller models with more parallelism in execution. They've obviously adapted the design but it's a risk optimising in hardware like that - if there is another model architecture jump the risk of having a narrow specialised set of hardware means you can't generalise enough.
  - zozbot234 8mo ago
    Prefill has a lot of parallelism, and so does decode with a larger context (very common with agentic tasks). People like to say "old inference chips are no good for LLM use" but that's not really true.
flyinglizard 8mo ago
NVidia is operating with what, 70% gross margin? That’s what Google saves. Plus, Broadcom may be in for the design but I’m not sure they’re involved in the manufacturing of TPUs.
- lizknope 8mo ago
  Broadcom does the physical design and sources a huge amount of the IP like serdes blocks. TSMC manufactures the chips.
dyauspitr 8mo ago
What a wild situation to have a significant part of Earth’s major economies be directly reliant, not on one country, but on one building in the world.
- collingreen 8mo ago
  Yeah this is a bummer. If it goes south everyone in power will also have perfect hindsight and say they saw it coming because obviously you shouldn't have this much built on such a small footprint. And yet...
  - palmotea 8mo ago
    > Yeah this is a bummer. If it goes south everyone in power will also have perfect hindsight and say they saw it coming because obviously you shouldn't have this much built on such a small footprint. And yet... It'll be true, everyone does see it coming (just like with rare earth minerals). But the market-infected Western society doesn't have the maturity to do anything about it. Businesses won't because they're expected to optimize for short-term financial returns, government won't because it's hobbled because biases against it (e.g. any failure becomes a political embarrassment, and there's a lot of pressure to stay out of areas where businesses operate and not interfere with businesses). America needs a lot more strategic government control of the economy, to kick businesses out of their short-term shareholder-focused thinking. If it can't manage that, it will decline into irrelevance.
    - nebula8804 8mo ago
      When USSR fell there was a lot of talk about how it was meant to be since the US system is the best of all the terrible systems. It deserved to win and the USSR system deserved to die. >America needs a lot more strategic government control of the economy, to kick businesses out of their short-term shareholder-focused thinking. If it can't manage that, it will decline into irrelevance. If it is meant to be then its meant to be. If the US decides to cling to its old system and it fails well then we would know that it wasn't the best system after all. Humankind will keep moving forward even if it means that another continent controls the show.
HDThoreaun 8mo ago
Just by removing nvidia's profits Google gets their TPUs for like 25-30% of what they cost to get them from nvidia, assuming similar cost structures. Google's cost structures are probably higher than nvidia's so realistically theyre probably paying around 50% of nvidia charges, but thats still billions of dollars a year and allows them to create a product that is tailor made for their needs.

coredog64 8mo ago

My take is it's the inference efficiency. It's one thing to have a huge GPU cluster for training, but come inference time you don't need nearly so much. Having the TPU (and models purpose built for TPU) allows for best cost in serving at hyperscale.

martinald 8mo ago
Yes potentially - but the OG TPUs were actually very poorly suited for LLM usage - designed for far smaller models with more parallelism in execution. They've obviously adapted the design but it's a risk optimising in hardware like that - if there is another model architecture jump the risk of having a narrow specialised set of hardware means you can't generalise enough.
- zozbot234 8mo ago
  Prefill has a lot of parallelism, and so does decode with a larger context (very common with agentic tasks). People like to say "old inference chips are no good for LLM use" but that's not really true.

flyinglizard 8mo ago

NVidia is operating with what, 70% gross margin? That’s what Google saves. Plus, Broadcom may be in for the design but I’m not sure they’re involved in the manufacturing of TPUs.

lizknope 8mo ago
Broadcom does the physical design and sources a huge amount of the IP like serdes blocks. TSMC manufactures the chips.

dyauspitr 8mo ago

What a wild situation to have a significant part of Earth’s major economies be directly reliant, not on one country, but on one building in the world.

collingreen 8mo ago
Yeah this is a bummer. If it goes south everyone in power will also have perfect hindsight and say they saw it coming because obviously you shouldn't have this much built on such a small footprint. And yet...
- palmotea 8mo ago
  > Yeah this is a bummer. If it goes south everyone in power will also have perfect hindsight and say they saw it coming because obviously you shouldn't have this much built on such a small footprint. And yet... It'll be true, everyone does see it coming (just like with rare earth minerals). But the market-infected Western society doesn't have the maturity to do anything about it. Businesses won't because they're expected to optimize for short-term financial returns, government won't because it's hobbled because biases against it (e.g. any failure becomes a political embarrassment, and there's a lot of pressure to stay out of areas where businesses operate and not interfere with businesses). America needs a lot more strategic government control of the economy, to kick businesses out of their short-term shareholder-focused thinking. If it can't manage that, it will decline into irrelevance.
  - nebula8804 8mo ago
    When USSR fell there was a lot of talk about how it was meant to be since the US system is the best of all the terrible systems. It deserved to win and the USSR system deserved to die. >America needs a lot more strategic government control of the economy, to kick businesses out of their short-term shareholder-focused thinking. If it can't manage that, it will decline into irrelevance. If it is meant to be then its meant to be. If the US decides to cling to its old system and it fails well then we would know that it wasn't the best system after all. Humankind will keep moving forward even if it means that another continent controls the show.

HDThoreaun 8mo ago

Just by removing nvidia's profits Google gets their TPUs for like 25-30% of what they cost to get them from nvidia, assuming similar cost structures. Google's cost structures are probably higher than nvidia's so realistically theyre probably paying around 50% of nvidia charges, but thats still billions of dollars a year and allows them to create a product that is tailor made for their needs.