4 ms·

The first specifically designed for inference? Wasn’t the original TPU inference only?

theptip 2y ago
The phrasing is very precise here, it’s the first TPU for _the age of inference_, which is a novel marketing term they have defined to refer to CoT and Deep Research.
- shitpostbot 2y ago
  They didn't though? > first designed specifically for inference. For more than a decade, TPUs have powered Google’s most demanding AI training and serving workloads... What do they think serving is? I think this marketing copy was written by someone with no idea what they are talking about, and not reviewed by anyone who did. Also funny enough it kinda looks like they've scrubbed all their references to v4i, where the i stands for inference. https://gwern.net/doc/ai/scaling/hardware/2021-jouppi.pdf https://gwern.net/doc/ai/scaling/hardware/2021-jouppi.pdf
- dang 2y ago
  Ugh. We should have caught that. Can anyone suggest a better (i.e. more accurate and neutral) title, devoid of marketing tropes?
- yencabulator 2y ago
  As a previous boss liked to say, this car is the cheapest in its price range, the roomiest in it's size category, and the fastest in its speed group.
jeffbee 2y ago
Yeah that made me chuckle, too. The original was indeed inference-only.
dgacmu 2y ago
Yup. (Source: was at brain at the time.) Also holy cow that was 10 years ago already? Dang. Amusing bit: The first TPU design was based on fully connected networks; the advent of CNNs forced some design rethinking, and then the advent of RNNs (and then transformers) did it yet again. So maybe it's reasonable to say that this is the first TPU designed for inference in the world where you have both a matrix multiply unit and an embedding processor. (Also, the first gen was purely a co-processor, whereas the later generations included their own network fabric, a trait shared by this most recent one. So it's not totally crazy to think of the first one as a very different beast.)
- kleiba 2y ago
  > the advent of CNNs forced some design rethinking, and then the advent of RNNs (and then transformers) did it yet again. Certainly, RNNs are much older than TPUs?!
  - woodson 2y ago
    So are CNNs, but I guess their popularity heavily increased at that time, to the point where it made sense to optimize the hardware for them.
  - hyhjtgh 2y ago
    RNN was of course well known at the at time, but they werent putting out state of the art numbers at that time.
- miki123211 2y ago
  Wow, you guys needed a custom ASIC for inference before CNNs were even invented? What were the use cases like back then?
  - refulgentis 2y ago
    https://research.google/blog/the-google-brain-team-looking-back-on-2016/ https://research.google/blog/the-google-brain-team-looking-b... is a good overview I wasn't on Brain, but got obsessed with Kerminology of ML internally at Google because I wanted to know why leadership was so gung ho on it. The general sense in the early days was these things can learn anything, and they'll replace fundamental units of computing. This thought process is best exhibited externally by ex. https://research.google/pubs/the-case-for-learned-index-structures/ https://research.google/pubs/the-case-for-learned-index-stru... It was also a different Google, the "3 different teams working on 3 different chips" bit reminds me of lore re: how many teams were working on Android wearables until upper management settled it. FWIW it's a very, very, different company now. Back then it was more entrepreneurial. A better version of Wave-era, where things launch themselves. An MBA would find this top-down company in 2025 even better, I find it less - it's perfectly tuned to do what Apple or OpenAI did 6-12 months ago, but not to lead - almost certainly a better investment, but a worse version of an average workplace, because it hasn't developed antibodies against BSing. (disclaimer: worked on Android)
m00x 2y ago
The first one was designed as a proof of concept that it would work at all, not really to be optimal for inference workloads. It just turns out that inference is easier.

The phrasing is very precise here, it’s the first TPU for _the age of inference_, which is a novel marketing term they have defined to refer to CoT and Deep Research.

shitpostbot 2y ago
They didn't though? > first designed specifically for inference. For more than a decade, TPUs have powered Google’s most demanding AI training and serving workloads... What do they think serving is? I think this marketing copy was written by someone with no idea what they are talking about, and not reviewed by anyone who did. Also funny enough it kinda looks like they've scrubbed all their references to v4i, where the i stands for inference. https://gwern.net/doc/ai/scaling/hardware/2021-jouppi.pdf https://gwern.net/doc/ai/scaling/hardware/2021-jouppi.pdf
dang 2y ago
Ugh. We should have caught that. Can anyone suggest a better (i.e. more accurate and neutral) title, devoid of marketing tropes?
yencabulator 2y ago
As a previous boss liked to say, this car is the cheapest in its price range, the roomiest in it's size category, and the fastest in its speed group.

jeffbee 2y ago

Yeah that made me chuckle, too. The original was indeed inference-only.

dgacmu 2y ago

Yup. (Source: was at brain at the time.) Also holy cow that was 10 years ago already? Dang. Amusing bit: The first TPU design was based on fully connected networks; the advent of CNNs forced some design rethinking, and then the advent of RNNs (and then transformers) did it yet again. So maybe it's reasonable to say that this is the first TPU designed for inference in the world where you have both a matrix multiply unit and an embedding processor. (Also, the first gen was purely a co-processor, whereas the later generations included their own network fabric, a trait shared by this most recent one. So it's not totally crazy to think of the first one as a very different beast.)

kleiba 2y ago
> the advent of CNNs forced some design rethinking, and then the advent of RNNs (and then transformers) did it yet again. Certainly, RNNs are much older than TPUs?!
- woodson 2y ago
  So are CNNs, but I guess their popularity heavily increased at that time, to the point where it made sense to optimize the hardware for them.
- hyhjtgh 2y ago
  RNN was of course well known at the at time, but they werent putting out state of the art numbers at that time.
miki123211 2y ago
Wow, you guys needed a custom ASIC for inference before CNNs were even invented? What were the use cases like back then?
- refulgentis 2y ago
  https://research.google/blog/the-google-brain-team-looking-back-on-2016/ https://research.google/blog/the-google-brain-team-looking-b... is a good overview I wasn't on Brain, but got obsessed with Kerminology of ML internally at Google because I wanted to know why leadership was so gung ho on it. The general sense in the early days was these things can learn anything, and they'll replace fundamental units of computing. This thought process is best exhibited externally by ex. https://research.google/pubs/the-case-for-learned-index-structures/ https://research.google/pubs/the-case-for-learned-index-stru... It was also a different Google, the "3 different teams working on 3 different chips" bit reminds me of lore re: how many teams were working on Android wearables until upper management settled it. FWIW it's a very, very, different company now. Back then it was more entrepreneurial. A better version of Wave-era, where things launch themselves. An MBA would find this top-down company in 2025 even better, I find it less - it's perfectly tuned to do what Apple or OpenAI did 6-12 months ago, but not to lead - almost certainly a better investment, but a worse version of an average workplace, because it hasn't developed antibodies against BSing. (disclaimer: worked on Android)

m00x 2y ago

The first one was designed as a proof of concept that it would work at all, not really to be optimal for inference workloads. It just turns out that inference is easier.