3 ms·
> We used Trillium TPUs to train the new Gemini 2.0, Wow. I knew custom Google silicon was used for inference, but I didn't realize it was used for training to
by xnx 2y ago
> We used Trillium TPUs to train the new Gemini 2.0,
Wow. I knew custom Google silicon was used for inference, but I didn't realize it was used for training too. Does this mean Google is free of dependence on Nvidia GPUs? That would be a huge advantage over AI competitors.
- m3kw9 2y agoMaybe only for their own models
- walterbell 2y agoNow any Google customer can use Trillium for training any model?
- richards 2y ago[Google employee] Yes, you can use TPUs in Compute Engine and GKE, among other places, for whatever you'd like. I just checked and the v6 are available.
- KaoruAoiShiho 2y agoIs there not goin to be a v6p?
- richards 2y agoCan't speculate on futures, but here's the current version log ... https://cloud.google.com/tpu/docs/system-architecture-tpu-vm#versions https://cloud.google.com/tpu/docs/system-architecture-tpu-vm...
- xnx 2y agoGoogle trained Llama-2-70B on Trillium chips
- monocasa 2y agoI thought llama was trained by meta.
- DrBenCarson 2y ago> Google trained Llama Source? This would make quite the splash in the market
- xnx 2y agoIt's in the article: "When training the Llama-2-70B model, our tests demonstrate that Trillium achieves near-linear scaling from a 4-slice Trillium-256 chip pod to a 36-slice Trillium-256 chip pod at a 99% scaling efficiency."
- llm_nerd 2y agoI'm pretty sure they're doing fine-tune training, using Llama because it is a widely known and available sample. They used SDXL elsewhere for the same reason. Llama 2 was released well over a year ago and was training between Meta and Microsoft.
- hhh 2y agoThey can just train another one.
- llm_nerd 2y agoLlama 2 end weights are public. The data used to train it, or even the process used to train it, are not. Google can't just train another Llama 2 from scratch. They could train something similar, but it'd be super weird if they called it Llama 2. They could call it something like "Gemini", or if it's open weights, "Gemma".
- lern_too_spel 2y agoThe article says they used maxtext to load the weights and pretrain on additional data. It looks like the instructions for doing that are here: https://github.com/AI-Hypercomputer/maxtext/blob/main/getting_started%2FRun_Llama2.md https://github.com/AI-Hypercomputer/maxtext/blob/main/gettin...
- Permit 2y agoMy understanding is that the Trillium TPU was primarily targeted at inference (so it’s surprising to see it was used to train Gemini 2.0) but other generations of TPUs have targeted training. For example the chip prior to this one is called TPU v5p and was targeted toward training.
- dekhn 2y agoGoogle silicon TPUs have been used for training for at least 5 years, probably more (I think it's 10 years). They do not depend on nvidia gpus for the majority of their projects. Took TPUs a while to catch up on some details, like sparsity.
- summerlight 2y agoThis aligns with my knowledge. I don't know much about LLM, but TPU has been used for training deep prediction models in ads at least from 2018, though there were some gap filled by CPU/GPU for a while. Nowaday, TPU capacity is probably more than the combination of CPU/GPU.
- felarof 2y ago+1, almost all (if not all) Google training runs on TPU. They don't use NVIDIA GPUs at all.
- dekhn 2y agoat some point some researchers were begging for GPUs... mainly for sparse work. I think that's why sparsecore was added to TPU (https://cloud.google.com/tpu/docs/system-architecture-tpu-vm#sparsecore https://cloud.google.com/tpu/docs/system-architecture-tpu-vm...) in v4. I think at this point with their tech turnaround time they can catch up as competitors add new features and researchers want to use them.
- felarof 2y agodumb question: wdym by sparse work? Is it embedding lookups? (TPUs have had BarnaCore for efficient embedding lookups since TPU v3)
- dekhn 2y agoMostly embedding, but IIRC DeepMind RL made use of sparsity- basically, huge matrices with only a few non-zero elements. BarnaCore existed and was used, but was tailored mostly for embeddings. BTW, IIRC they were called that because they were added "like a barnacle hanging off the side". The evolution of TPU has been interesting to watch; I came from the HPC and supercomputing space, and seeing Google as mostly-CPU for the longest time, and then finally learning how to build "supercomputers" over a decade+ (gradually adding many features that classical supercomputers have long had), was a very interesting process. Some very expensive mistakes along the way. But now they've paid down almost all the expensive up-front costs and can now ride on the margins, adding new bits and pieces while increasing the clocks and capacities on a cadence.
- lern_too_spel 2y agoSince TPUv2, announced in 2017: https://arstechnica.com/information-technology/2017/05/google-brings-45-teraflops-tensor-flow-processors-to-its-compute-cloud/ https://arstechnica.com/information-technology/2017/05/googl... The superscalers are all working on this. https://aws.amazon.com/ai/machine-learning/trainium/ https://aws.amazon.com/ai/machine-learning/trainium/
- drusepth 2y agoWhy is that a huge advantage over AI competitors? Just not having to fight for limited Nvidia supply?
- aseipp 2y agoThat is one factor, but another is total cost of ownership. At large scales something that's 1/2 the overall speed but 1/3rd the total cost is still a net win by a large margin. This is one of the reasons why every major hyperscaler is, to some extent, developing their own hardware e.g. Meta, who famously have an insane amount of Nvidia GPUs. Of course this does not mean their models will necessarily be proportionally better, nor does it mean Google won't buy GPUs for other reasons (like providing them to customers on Google Cloud.)
- xnx 2y agoYes. And cheaper operating cost per TFLOP.
- badlucklottery 2y agoVertical integration. Nvidia is making big bucks "selling shovels in a gold rush". Google has made their own shovel factory and they can avoid paying Nvidia's margins.
- bufferoverflow 2y agoTPUs are cheaper and faster than GPUs. But it's custom silicon. Which means barrier to entry is very very high.
- felarof 2y ago> Which means barrier to entry is very very high. +1 on this. The tooling to use TPUs still needs more work. But we are betting on building this tooling and unlocking these ASIC chips (https://github.com/felafax/felafax https://github.com/felafax/felafax).
- ein0p 2y agoThey are cheaper, yes. But GPUs are faster and easier to program.
- felarof 2y agoTPUs have been used for training since a long time. (PS: we are startup trying to make TPUs more accessible, if you wanna fine-tune Llama3 on TPU check out https://github.com/felafax/felafax https://github.com/felafax/felafax)
- ec109685 2y agoApple trained their LLM on Google TPU’s: https://www.theregister.com/2024/07/30/apple_google_tpu_ai/ https://www.theregister.com/2024/07/30/apple_google_tpu_ai/