8 ms·

On the podcast interview now Groq CEO Jonathon Ross did[1] he talked about the creation of the original TPUs (which he built at Google). Apparently originally i

by nl 3y ago

On the podcast interview now Groq CEO Jonathon Ross did[1] he talked about the creation of the original TPUs (which he built at Google). Apparently originally it was a FPGA he did in his 20% time because he sat near the team who was having inference speed issues. They got it working, then Jeff Dean did the math and the decided to do an ASIC. Now of course Google should spin off the TPU team as a separate company. It's the only credible competition NVidia has, and the software support is second only to NVidia. [1] https://open.spotify.com/episode/0V9kRgNS7Ds6zh3GjdXUAQ?si=qE3MLNuFQLCFmSnvhG87jQ&nd=1&dlsi=3752b373bdfd4f2c https://open.spotify.com/episode/0V9kRgNS7Ds6zh3GjdXUAQ?si=q...

summerlight 3y ago
> Now of course Google should spin off the TPU team as a separate company. Given the size of the market and its near-monopoly situation, I strongly think this has the potential to (almost immediately) surpass the Pixel hardware business. But the problem here is that TPU is a relatively scarce computing resource even inside Google and it's very likely that Google has a hard time to meet its internal demands...
- fnbr 3y ago
  I’m surprised they sell any to external customers, to be honest.
  - KeplerBoy 3y ago
    They don't sell any TPUs, do they? Besides the, now ancient, coral toy-TPUs.
    - fnbr 3y ago
      I meant rent on GCP. You’re right.
    - Kelteseth 3y ago
      Has there been any development? The last update is from 2021 [0], but it is not officially killed by google(.com) [0] https://coral.ai/news/updates-07-2021 https://coral.ai/news/updates-07-2021
      - ddalex 3y ago
        My guess is that the "AI" accelerators in Google Tensor phone chips are based on Coral....
- nl 3y ago
  > I strongly think this has the potential to (almost immediately) surpass the Pixel hardware business. But the problem here is that TPU is a relatively scarce computing resource even inside Google and it's very likely that Google has a hard time to meet its internal demands... Yes. But imagine how the company would do: they have a guaranteed market at Google say for 3 years, and while yes maybe Google takes 100% of the production in the first 12 months it's not a bad position to start from. Plus there are other products which they could ship that might not always need to be built on the latest process. I imagine there would be demand for inference only earlier generation TPUs that can run LLMs fast if the power usage is low enough.
bfeynman 3y ago
Amazon acquired Annapurna labs doing the same thing and have their own train,/inferentia silicon, and they definitely have more support than Google.
Laremere 3y ago
The way I see, NVidia only has a few advantages ordered from most important to least: 1. Reserved fab space. 2. Highly integrated software. 3. Hardware architecture that exists today. 4. Customer relationships. but all of these aspects are weak in one way or another: For #1, fab space is tight, and NVidia can strangle its consumer GPU market if it means selling more AI chips at a higher price. This advantage is gone if a competitor makes big bets years in advance, or another company that has a lot of fab space (intel?) is willing to change priorities. 2. Life is good when your proprietary software is the industry standard. Whether this actually matters will depend on the use case heavily. 3. A benefit now, but not for long. It's my estimation that the hardware design for TPUs is fundamentally much simpler than for GPUs. No need for raytracing, texture samplers, or rasterization. Mostly just needs lots of matrix multiplication and memory. Others moving into the space will be able to catch up quickly. 4. Useful to stay in the conversation, but in a field hungry for any advantage, the hardware vendor with the highest FLOPS (or equivalent) per dollar is going to win enough customers to saturate their manufacturing ability. So overall, I give them a few years, and then the competition is going to be real quite fast.
- 7e 3y ago
  These have always been NVIDIA's "few" advantages and yet they've still dominated for years. It's their relentless pace of innovation that is their advantage. They resemble Intel of old, and despite Intel's same "few" advantages, Intel is still dominant in the PC space (even with recent missteps).
  - weweersdfsd 3y ago
    They've dominated for years, but now all big tech companies are using their products in scale not seen before, and all have vested interest in cutting their margins by introducing some real competition. Nvidia will do good in the future, but perhaps not good enough to justify their stock price.
- logicchains 3y ago
  >2. Highly integrated software. NVidia's biggest advantage is that AMD is unwilling to pay for top notch software engineers (and unwilling to pay the corresponding increase in hardware engineer salaries this would entail). If you check online you'll see NVidia pays both hardware and software engineers significantly more than AMD does. This is a cultural/management problem, which AMD's unlikely to overcome in the near-term future. Apple so far seems like the only other hardware company that doesn't underpay its engineers, but Apple's unlikely to release a discrete/stand-alone GPU any time soon.
ipsum2 3y ago
> It's the only credible competition NVidia has This is wrong, both AMD and Intel (through Habana) have GPUs comparable to H100s in performance.
- nl 3y ago
  Yes, but they don't have the custom kernels that CUDA has. TPUs do have some!
  - rurban 3y ago
    They have Vulcan, which is cross-compatible. And AMD has ROCm. pytorch is standard and pytorch has ROCm support. And the Google TPU v5 also has pytorch support. We do have a couple of H100's, but I'd love to replace them with AMD's
    - nl 3y ago
      Vulcan is a driver-level API. It competes with DirectX and OpenGL. CUDA is a language you write kernels. It competes with OpenAI's Triton language. Here's what CUDA looks like: https://github.com/tspeterkim/flash-attention-minimal/blob/main/flash.cu https://github.com/tspeterkim/flash-attention-minimal/blob/m... This is what Triton looks like: https://triton-lang.org/main/getting-started/tutorials/06-fused-attention.html#sphx-glr-download-getting-started-tutorials-06-fused-attention-py https://triton-lang.org/main/getting-started/tutorials/06-fu... By contrast Vulcan looks like this: https://github.com/KhronosGroup/Vulkan-Samples/blob/main/samples/api/hello_triangle/hello_triangle.cpp https://github.com/KhronosGroup/Vulkan-Samples/blob/main/sam... (It's true to some extent that maybe you could use Vulcan shaders to write deep learning kernels, maybe? I'm not aware of anyone doing it though)
    - Kelteseth 3y ago
      If AMD fixes or open sources their proprietary firmware blob[0]. Geohot streamed all weekend on Twitch, reverse engineering the AMD firmware. It was quite entertaining learning about how that low level hardware firmware works[1] and his rants about AMD of course. [0] https://www.phoronix.com/news/Tinybox-Radeon-Again-UMR https://www.phoronix.com/news/Tinybox-Radeon-Again-UMR [1] https://www.twitch.tv/georgehotz https://www.twitch.tv/georgehotz
pyb 3y ago
There seem to be conflicting reports as to who came up with the TPU https://mastodon.social/@danluu/109641269333636407 https://mastodon.social/@danluu/109641269333636407
bionhoward 3y ago
Speaking of which, mega props to Groq, they really are awesome, so many startups launch with bullshit and promises, but Groq came to the scene with something awesome already working, which is reason enough to love them. I really respect this company and I say that extremely never-often.
- elorant 3y ago
  I wouldn't call it awesome. It's just a big chip with lots of cache. You need hundreds of them to sufficiently load any decent model. At which point the cost has skyrocketed.

by nl 3y ago

summerlight 3y ago

> Now of course Google should spin off the TPU team as a separate company. Given the size of the market and its near-monopoly situation, I strongly think this has the potential to (almost immediately) surpass the Pixel hardware business. But the problem here is that TPU is a relatively scarce computing resource even inside Google and it's very likely that Google has a hard time to meet its internal demands...

fnbr 3y ago
I’m surprised they sell any to external customers, to be honest.
- KeplerBoy 3y ago
  They don't sell any TPUs, do they? Besides the, now ancient, coral toy-TPUs.
  - fnbr 3y ago
    I meant rent on GCP. You’re right.
  - Kelteseth 3y ago
    Has there been any development? The last update is from 2021 [0], but it is not officially killed by google(.com) [0] https://coral.ai/news/updates-07-2021 https://coral.ai/news/updates-07-2021
    - ddalex 3y ago
      My guess is that the "AI" accelerators in Google Tensor phone chips are based on Coral....
nl 3y ago
> I strongly think this has the potential to (almost immediately) surpass the Pixel hardware business. But the problem here is that TPU is a relatively scarce computing resource even inside Google and it's very likely that Google has a hard time to meet its internal demands... Yes. But imagine how the company would do: they have a guaranteed market at Google say for 3 years, and while yes maybe Google takes 100% of the production in the first 12 months it's not a bad position to start from. Plus there are other products which they could ship that might not always need to be built on the latest process. I imagine there would be demand for inference only earlier generation TPUs that can run LLMs fast if the power usage is low enough.

bfeynman 3y ago

Amazon acquired Annapurna labs doing the same thing and have their own train,/inferentia silicon, and they definitely have more support than Google.

Laremere 3y ago

The way I see, NVidia only has a few advantages ordered from most important to least: 1. Reserved fab space. 2. Highly integrated software. 3. Hardware architecture that exists today. 4. Customer relationships. but all of these aspects are weak in one way or another: For #1, fab space is tight, and NVidia can strangle its consumer GPU market if it means selling more AI chips at a higher price. This advantage is gone if a competitor makes big bets years in advance, or another company that has a lot of fab space (intel?) is willing to change priorities. 2. Life is good when your proprietary software is the industry standard. Whether this actually matters will depend on the use case heavily. 3. A benefit now, but not for long. It's my estimation that the hardware design for TPUs is fundamentally much simpler than for GPUs. No need for raytracing, texture samplers, or rasterization. Mostly just needs lots of matrix multiplication and memory. Others moving into the space will be able to catch up quickly. 4. Useful to stay in the conversation, but in a field hungry for any advantage, the hardware vendor with the highest FLOPS (or equivalent) per dollar is going to win enough customers to saturate their manufacturing ability. So overall, I give them a few years, and then the competition is going to be real quite fast.

7e 3y ago
These have always been NVIDIA's "few" advantages and yet they've still dominated for years. It's their relentless pace of innovation that is their advantage. They resemble Intel of old, and despite Intel's same "few" advantages, Intel is still dominant in the PC space (even with recent missteps).
- weweersdfsd 3y ago
  They've dominated for years, but now all big tech companies are using their products in scale not seen before, and all have vested interest in cutting their margins by introducing some real competition. Nvidia will do good in the future, but perhaps not good enough to justify their stock price.
logicchains 3y ago
>2. Highly integrated software. NVidia's biggest advantage is that AMD is unwilling to pay for top notch software engineers (and unwilling to pay the corresponding increase in hardware engineer salaries this would entail). If you check online you'll see NVidia pays both hardware and software engineers significantly more than AMD does. This is a cultural/management problem, which AMD's unlikely to overcome in the near-term future. Apple so far seems like the only other hardware company that doesn't underpay its engineers, but Apple's unlikely to release a discrete/stand-alone GPU any time soon.

ipsum2 3y ago

> It's the only credible competition NVidia has This is wrong, both AMD and Intel (through Habana) have GPUs comparable to H100s in performance.

nl 3y ago
Yes, but they don't have the custom kernels that CUDA has. TPUs do have some!
- rurban 3y ago
  They have Vulcan, which is cross-compatible. And AMD has ROCm. pytorch is standard and pytorch has ROCm support. And the Google TPU v5 also has pytorch support. We do have a couple of H100's, but I'd love to replace them with AMD's
  - nl 3y ago
    Vulcan is a driver-level API. It competes with DirectX and OpenGL. CUDA is a language you write kernels. It competes with OpenAI's Triton language. Here's what CUDA looks like: https://github.com/tspeterkim/flash-attention-minimal/blob/main/flash.cu https://github.com/tspeterkim/flash-attention-minimal/blob/m... This is what Triton looks like: https://triton-lang.org/main/getting-started/tutorials/06-fused-attention.html#sphx-glr-download-getting-started-tutorials-06-fused-attention-py https://triton-lang.org/main/getting-started/tutorials/06-fu... By contrast Vulcan looks like this: https://github.com/KhronosGroup/Vulkan-Samples/blob/main/samples/api/hello_triangle/hello_triangle.cpp https://github.com/KhronosGroup/Vulkan-Samples/blob/main/sam... (It's true to some extent that maybe you could use Vulcan shaders to write deep learning kernels, maybe? I'm not aware of anyone doing it though)
  - Kelteseth 3y ago
    If AMD fixes or open sources their proprietary firmware blob[0]. Geohot streamed all weekend on Twitch, reverse engineering the AMD firmware. It was quite entertaining learning about how that low level hardware firmware works[1] and his rants about AMD of course. [0] https://www.phoronix.com/news/Tinybox-Radeon-Again-UMR https://www.phoronix.com/news/Tinybox-Radeon-Again-UMR [1] https://www.twitch.tv/georgehotz https://www.twitch.tv/georgehotz

pyb 3y ago

There seem to be conflicting reports as to who came up with the TPU https://mastodon.social/@danluu/109641269333636407 https://mastodon.social/@danluu/109641269333636407

bionhoward 3y ago

Speaking of which, mega props to Groq, they really are awesome, so many startups launch with bullshit and promises, but Groq came to the scene with something awesome already working, which is reason enough to love them. I really respect this company and I say that extremely never-often.

elorant 3y ago
I wouldn't call it awesome. It's just a big chip with lots of cache. You need hundreds of them to sufficiently load any decent model. At which point the cost has skyrocketed.