4 ms·

Yup. (Source: was at brain at the time.) Also holy cow that was 10 years ago already? Dang. Amusing bit: The first TPU design was based on fully connected net

by dgacmu 2y ago

Yup. (Source: was at brain at the time.) Also holy cow that was 10 years ago already? Dang. Amusing bit: The first TPU design was based on fully connected networks; the advent of CNNs forced some design rethinking, and then the advent of RNNs (and then transformers) did it yet again. So maybe it's reasonable to say that this is the first TPU designed for inference in the world where you have both a matrix multiply unit and an embedding processor. (Also, the first gen was purely a co-processor, whereas the later generations included their own network fabric, a trait shared by this most recent one. So it's not totally crazy to think of the first one as a very different beast.)

kleiba 2y ago
> the advent of CNNs forced some design rethinking, and then the advent of RNNs (and then transformers) did it yet again. Certainly, RNNs are much older than TPUs?!
- woodson 2y ago
  So are CNNs, but I guess their popularity heavily increased at that time, to the point where it made sense to optimize the hardware for them.
- hyhjtgh 2y ago
  RNN was of course well known at the at time, but they werent putting out state of the art numbers at that time.
miki123211 2y ago
Wow, you guys needed a custom ASIC for inference before CNNs were even invented? What were the use cases like back then?
- refulgentis 2y ago
  https://research.google/blog/the-google-brain-team-looking-back-on-2016/ https://research.google/blog/the-google-brain-team-looking-b... is a good overview I wasn't on Brain, but got obsessed with Kerminology of ML internally at Google because I wanted to know why leadership was so gung ho on it. The general sense in the early days was these things can learn anything, and they'll replace fundamental units of computing. This thought process is best exhibited externally by ex. https://research.google/pubs/the-case-for-learned-index-structures/ https://research.google/pubs/the-case-for-learned-index-stru... It was also a different Google, the "3 different teams working on 3 different chips" bit reminds me of lore re: how many teams were working on Android wearables until upper management settled it. FWIW it's a very, very, different company now. Back then it was more entrepreneurial. A better version of Wave-era, where things launch themselves. An MBA would find this top-down company in 2025 even better, I find it less - it's perfectly tuned to do what Apple or OpenAI did 6-12 months ago, but not to lead - almost certainly a better investment, but a worse version of an average workplace, because it hasn't developed antibodies against BSing. (disclaimer: worked on Android)
  - marsten 2y ago
    Google was changed by two things, neither of which were much fun. But very understandable. One was the transition to a mature product line. In the early days it was about how do we do cool new things that will delight users: Gmail, Google Maps (Where 2), YouTube. The focus was on user growth and adoption. Then growth saturated and the focus turned to profitability: Getting more value out of existing users and defending the business. That shift causes you to think very differently, and it's not as fun. The second was changing market conditions. The web grew up, tech grew up, and the investment needed to make a competitive product skyrocketed. Google needed more wood behind fewer arrows and that meant reining in all the small teams running around doing kooky things. Again not fun, but understandable.
- huijzer 2y ago
  According to a Google blog post from 2016 [1], use-cases were RankBrain to improve the relevancy of search results and Street View. Also they used it for AlphaGo. And from what I remember from my MSc thesis, they also probably were starting to use it for Translate. I can't find any TPU reference in the Attention is All You Need or BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, but I have been fine-tuning BERT in a TPU at the time in okt 2018 [2]. If I remember correctly, the BERT example repository showed how to fit a model with a TPU inside a Colab. So I would guess that the natural language research was mostly not on TPU's around 2016-2018, but then moved over to TPU in production. I could be wrong though and dgacmu probably knows more. [1]: https://cloud.google.com/blog/products/ai-machine-learning/google-supercharges-machine-learning-tasks-with-custom-chip https://cloud.google.com/blog/products/ai-machine-learning/g... [2]: https://github.com/rikhuijzer/improv/blob/master/runs/2018-11-28%20askubuntu%20bert-base%2030%20epochs%20f1/GPU_Google_BERT_Sentence_Classification.ipynb https://github.com/rikhuijzer/improv/blob/master/runs/2018-1...
  - mmx1 2y ago
    Yes, IIRC (please correct me if I'm wrong), translate did utilize Seastar (TPU v1) which was integer only, so not easily useful for training.
- dekhn 2y ago
  As an aside, Google used CPU-based machine learning (using enormous numbers of CPUs) for a long time before custom ASICS or tensorflow even existed. The big ones were SmartASS (ads serving) and Sibyl (everything else serving). There was an internal debate over the value of GPUs with a prominent engineer writing an influential doc that caused Google continue with fat CPU nodes when it was clear that accelerators were a good alternative. This was around the time ImageNet blew up, and some eng were stuffing multiple GPUs in their dev boxes to demonstrate training speeds on tasks like voice recognition. Sibyl was a heavy user of embeddings before there was any real custom ASIC support for that and there was an add-on for TPUs called barnacore to give limited embedding support (embeddings are very useful for maximizing profit through ranking).
  - mmx1 2y ago
    What I've heard is that the extrapolation of compute needed so many additional CPU servers to keep running the existing workload types that it obviously justified dedicated hardware. Same for video encoding accelerators[1]. [1]: https://research.google/pubs/warehouse-scale-video-acceleration-co-design-and-deployment-in-the-wild/ https://research.google/pubs/warehouse-scale-video-accelerat...
    - dekhn 2y ago
      yes, that was one of the externally shared narratives. The other part is that google didn't want to be beholden to nvidia GPUs, since they have an associated profit margin that is higher than TPUs, as well as resource constraints (total amount of GPUs shipping at any given time). Another part that was left out was that Google did not make truly high speed (low-latency) networking and so many of their CPU jobs had to be engineered around slow networks to maintain high utilization and training speed. Google basically ended up internally relearning the lessons that HPC and supercomputing communities had already established over decades.
  - miki123211 2y ago
    > The big ones were SmartASS (ads serving) and Sibyl (everything else serving). Ah, the days when you, as a tech company employee, could call a service "SmartASS" and get away with it...

by dgacmu 2y ago

kleiba 2y ago

> the advent of CNNs forced some design rethinking, and then the advent of RNNs (and then transformers) did it yet again. Certainly, RNNs are much older than TPUs?!

woodson 2y ago
So are CNNs, but I guess their popularity heavily increased at that time, to the point where it made sense to optimize the hardware for them.
hyhjtgh 2y ago
RNN was of course well known at the at time, but they werent putting out state of the art numbers at that time.

miki123211 2y ago

Wow, you guys needed a custom ASIC for inference before CNNs were even invented? What were the use cases like back then?

refulgentis 2y ago
https://research.google/blog/the-google-brain-team-looking-back-on-2016/ https://research.google/blog/the-google-brain-team-looking-b... is a good overview I wasn't on Brain, but got obsessed with Kerminology of ML internally at Google because I wanted to know why leadership was so gung ho on it. The general sense in the early days was these things can learn anything, and they'll replace fundamental units of computing. This thought process is best exhibited externally by ex. https://research.google/pubs/the-case-for-learned-index-structures/ https://research.google/pubs/the-case-for-learned-index-stru... It was also a different Google, the "3 different teams working on 3 different chips" bit reminds me of lore re: how many teams were working on Android wearables until upper management settled it. FWIW it's a very, very, different company now. Back then it was more entrepreneurial. A better version of Wave-era, where things launch themselves. An MBA would find this top-down company in 2025 even better, I find it less - it's perfectly tuned to do what Apple or OpenAI did 6-12 months ago, but not to lead - almost certainly a better investment, but a worse version of an average workplace, because it hasn't developed antibodies against BSing. (disclaimer: worked on Android)
- marsten 2y ago
  Google was changed by two things, neither of which were much fun. But very understandable. One was the transition to a mature product line. In the early days it was about how do we do cool new things that will delight users: Gmail, Google Maps (Where 2), YouTube. The focus was on user growth and adoption. Then growth saturated and the focus turned to profitability: Getting more value out of existing users and defending the business. That shift causes you to think very differently, and it's not as fun. The second was changing market conditions. The web grew up, tech grew up, and the investment needed to make a competitive product skyrocketed. Google needed more wood behind fewer arrows and that meant reining in all the small teams running around doing kooky things. Again not fun, but understandable.
huijzer 2y ago
According to a Google blog post from 2016 [1], use-cases were RankBrain to improve the relevancy of search results and Street View. Also they used it for AlphaGo. And from what I remember from my MSc thesis, they also probably were starting to use it for Translate. I can't find any TPU reference in the Attention is All You Need or BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, but I have been fine-tuning BERT in a TPU at the time in okt 2018 [2]. If I remember correctly, the BERT example repository showed how to fit a model with a TPU inside a Colab. So I would guess that the natural language research was mostly not on TPU's around 2016-2018, but then moved over to TPU in production. I could be wrong though and dgacmu probably knows more. [1]: https://cloud.google.com/blog/products/ai-machine-learning/google-supercharges-machine-learning-tasks-with-custom-chip https://cloud.google.com/blog/products/ai-machine-learning/g... [2]: https://github.com/rikhuijzer/improv/blob/master/runs/2018-11-28%20askubuntu%20bert-base%2030%20epochs%20f1/GPU_Google_BERT_Sentence_Classification.ipynb https://github.com/rikhuijzer/improv/blob/master/runs/2018-1...
- mmx1 2y ago
  Yes, IIRC (please correct me if I'm wrong), translate did utilize Seastar (TPU v1) which was integer only, so not easily useful for training.
dekhn 2y ago
As an aside, Google used CPU-based machine learning (using enormous numbers of CPUs) for a long time before custom ASICS or tensorflow even existed. The big ones were SmartASS (ads serving) and Sibyl (everything else serving). There was an internal debate over the value of GPUs with a prominent engineer writing an influential doc that caused Google continue with fat CPU nodes when it was clear that accelerators were a good alternative. This was around the time ImageNet blew up, and some eng were stuffing multiple GPUs in their dev boxes to demonstrate training speeds on tasks like voice recognition. Sibyl was a heavy user of embeddings before there was any real custom ASIC support for that and there was an add-on for TPUs called barnacore to give limited embedding support (embeddings are very useful for maximizing profit through ranking).
- mmx1 2y ago
  What I've heard is that the extrapolation of compute needed so many additional CPU servers to keep running the existing workload types that it obviously justified dedicated hardware. Same for video encoding accelerators[1]. [1]: https://research.google/pubs/warehouse-scale-video-acceleration-co-design-and-deployment-in-the-wild/ https://research.google/pubs/warehouse-scale-video-accelerat...
  - dekhn 2y ago
    yes, that was one of the externally shared narratives. The other part is that google didn't want to be beholden to nvidia GPUs, since they have an associated profit margin that is higher than TPUs, as well as resource constraints (total amount of GPUs shipping at any given time). Another part that was left out was that Google did not make truly high speed (low-latency) networking and so many of their CPU jobs had to be engineered around slow networks to maintain high utilization and training speed. Google basically ended up internally relearning the lessons that HPC and supercomputing communities had already established over decades.
- miki123211 2y ago
  > The big ones were SmartASS (ads serving) and Sibyl (everything else serving). Ah, the days when you, as a tech company employee, could call a service "SmartASS" and get away with it...