7 ms·
Trillium TPU Is GA
- xnx 2y ago> We used Trillium TPUs to train the new Gemini 2.0, Wow. I knew custom Google silicon was used for inference, but I didn't realize it was used for training too. Does this mean Google is free of dependence on Nvidia GPUs? That would be a huge advantage over AI competitors.
- m3kw9 2y agoMaybe only for their own models
- walterbell 2y agoNow any Google customer can use Trillium for training any model?
- richards 2y ago[Google employee] Yes, you can use TPUs in Compute Engine and GKE, among other places, for whatever you'd like. I just checked and the v6 are available.
- KaoruAoiShiho 2y agoIs there not goin to be a v6p?
- richards 2y agoCan't speculate on futures, but here's the current version log ... https://cloud.google.com/tpu/docs/system-architecture-tpu-vm#versions https://cloud.google.com/tpu/docs/system-architecture-tpu-vm...
- xnx 2y agoGoogle trained Llama-2-70B on Trillium chips
- monocasa 2y agoI thought llama was trained by meta.
- DrBenCarson 2y ago> Google trained Llama Source? This would make quite the splash in the market
- xnx 2y agoIt's in the article: "When training the Llama-2-70B model, our tests demonstrate that Trillium achieves near-linear scaling from a 4-slice Trillium-256 chip pod to a 36-slice Trillium-256 chip pod at a 99% scaling efficiency."
- llm_nerd 2y agoI'm pretty sure they're doing fine-tune training, using Llama because it is a widely known and available sample. They used SDXL elsewhere for the same reason. Llama 2 was released well over a year ago and was training between Meta and Microsoft.
- hhh 2y agoThey can just train another one.
- llm_nerd 2y agoLlama 2 end weights are public. The data used to train it, or even the process used to train it, are not. Google can't just train another Llama 2 from scratch. They could train something similar, but it'd be super weird if they called it Llama 2. They could call it something like "Gemini", or if it's open weights, "Gemma".
- Permit 2y agoMy understanding is that the Trillium TPU was primarily targeted at inference (so it’s surprising to see it was used to train Gemini 2.0) but other generations of TPUs have targeted training. For example the chip prior to this one is called TPU v5p and was targeted toward training.
- dekhn 2y agoGoogle silicon TPUs have been used for training for at least 5 years, probably more (I think it's 10 years). They do not depend on nvidia gpus for the majority of their projects. Took TPUs a while to catch up on some details, like sparsity.
- summerlight 2y agoThis aligns with my knowledge. I don't know much about LLM, but TPU has been used for training deep prediction models in ads at least from 2018, though there were some gap filled by CPU/GPU for a while. Nowaday, TPU capacity is probably more than the combination of CPU/GPU.
- felarof 2y ago+1, almost all (if not all) Google training runs on TPU. They don't use NVIDIA GPUs at all.
- dekhn 2y agoat some point some researchers were begging for GPUs... mainly for sparse work. I think that's why sparsecore was added to TPU (https://cloud.google.com/tpu/docs/system-architecture-tpu-vm#sparsecore https://cloud.google.com/tpu/docs/system-architecture-tpu-vm...) in v4. I think at this point with their tech turnaround time they can catch up as competitors add new features and researchers want to use them.
- felarof 2y agodumb question: wdym by sparse work? Is it embedding lookups? (TPUs have had BarnaCore for efficient embedding lookups since TPU v3)
- dekhn 2y agoMostly embedding, but IIRC DeepMind RL made use of sparsity- basically, huge matrices with only a few non-zero elements. BarnaCore existed and was used, but was tailored mostly for embeddings. BTW, IIRC they were called that because they were added "like a barnacle hanging off the side". The evolution of TPU has been interesting to watch; I came from the HPC and supercomputing space, and seeing Google as mostly-CPU for the longest time, and then finally learning how to build "supercomputers" over a decade+ (gradually adding many features that classical supercomputers have long had), was a very interesting process. Some very expensive mistakes along the way. But now they've paid down almost all the expensive up-front costs and can now ride on the margins, adding new bits and pieces while increasing the clocks and capacities on a cadence.
- lern_too_spel 2y agoSince TPUv2, announced in 2017: https://arstechnica.com/information-technology/2017/05/google-brings-45-teraflops-tensor-flow-processors-to-its-compute-cloud/ https://arstechnica.com/information-technology/2017/05/googl... The superscalers are all working on this. https://aws.amazon.com/ai/machine-learning/trainium/ https://aws.amazon.com/ai/machine-learning/trainium/
- drusepth 2y agoWhy is that a huge advantage over AI competitors? Just not having to fight for limited Nvidia supply?
- aseipp 2y agoThat is one factor, but another is total cost of ownership. At large scales something that's 1/2 the overall speed but 1/3rd the total cost is still a net win by a large margin. This is one of the reasons why every major hyperscaler is, to some extent, developing their own hardware e.g. Meta, who famously have an insane amount of Nvidia GPUs. Of course this does not mean their models will necessarily be proportionally better, nor does it mean Google won't buy GPUs for other reasons (like providing them to customers on Google Cloud.)
- xnx 2y agoYes. And cheaper operating cost per TFLOP.
- badlucklottery 2y agoVertical integration. Nvidia is making big bucks "selling shovels in a gold rush". Google has made their own shovel factory and they can avoid paying Nvidia's margins.
- bufferoverflow 2y agoTPUs are cheaper and faster than GPUs. But it's custom silicon. Which means barrier to entry is very very high.
- felarof 2y ago> Which means barrier to entry is very very high. +1 on this. The tooling to use TPUs still needs more work. But we are betting on building this tooling and unlocking these ASIC chips (https://github.com/felafax/felafax https://github.com/felafax/felafax).
- ein0p 2y agoThey are cheaper, yes. But GPUs are faster and easier to program.
- felarof 2y agoTPUs have been used for training since a long time. (PS: we are startup trying to make TPUs more accessible, if you wanna fine-tune Llama3 on TPU check out https://github.com/felafax/felafax https://github.com/felafax/felafax)
- ec109685 2y agoApple trained their LLM on Google TPU’s: https://www.theregister.com/2024/07/30/apple_google_tpu_ai/ https://www.theregister.com/2024/07/30/apple_google_tpu_ai/
- randomcatuser 2y agoHow good is Trillium/TPU compared to Nvidia? It seems the stats are: tpu v6e achieves 900 TFLOPS per chip (fp16) while Nvidia H100 achieves 1800 TFLOPS per gpu? (fp16)? Would be neat if anyone has benchmarks!!
- chessgecko 2y ago1800 on the h100s is with 2/4 sparsity, it’s half of that without. Not sure if the tpu number is doing that too, but I don’t think 2/4 is used that heavily so I probably would compare without it.
- amelius 2y agoAre the Gemini models open?
- blackeyeblitzar 2y agoSo Google has Trillium, Amazon has Trainium, Apple is working on a custom chip with Broadcom, etc. Nvidia’s moat doesn’t seem that big. Plus big tech companies have the data and customers and will probably be the only surviving big AI training companies. I doubt startups can survive this game - they can’t afford the chips, can’t build their own, don’t have existing products to leech data off of, and don’t have control over distribution channels like OS or app stores
- talldayo 2y ago> Nvidia’s moat doesn’t seem that big. Well, look at it this way. Nvidia played their cards so well that their competitors had to invent entirely new product categories to supplant their demand for Nvidia hardware. This new hardware isn't even reprising the role of CUDA, just the subset of tensor operations that are used for training and AI inference. If demand for training and inference wanes, these hardware investments will be almost entirely wasted. Nvidia's core competencies - scaling hardware up and down, providing good software interfaces and selling direct to consumer are not really assailed at all. The big lesson Nvidia is giving to the industry is that you should invest in complex GPU architectures and write the software to support it. Currently the industry is trying it's hardest to reject that philosophy, and only time will tell if they're correct.
- felarof 2y ago> CUDA, just the subset of tensor operations that are used for training and AI inference. If demand for training and inference wanes Interesting take, but why would demand for training and inference wade? This seems like a very contrarian take.
- talldayo 2y agoMaybe it won't - I say "time will tell" because we really do not know how much LLMs will be demanded in 10 years. Nvidia's stock skyrocketed because they were incidentally prepared for an enormous increase in demand the moment it happened. Now that expectations are cooling down and Sam Altman is signalling that AGI is a long ways off, the math that justified designing NPU/TPU hardware in-house might not add up anymore. Even if you believe in the tech itself, the hype is cooling and the do-or-die moment is rapidly approaching. My overall point is that I think Nvidia played smartly from the start. They could derive profit from any sufficiently large niche their competitors were too afraid to exploit, and general purpose GPU compute was the perfect investment. With AMD, Apple and the rest of the industry focusing on simpler GPUs, Nvidia was given an empty soapbox to market CUDA with. The big question is whether demand for CUDA can be supplanted with application-specific accelerators.
- Hilift 2y ago"we constantly strive to enhance the performance and efficiency of our Mamba and Jamba language models." ... "The growing importance of multi-step reasoning at inference time necessitates accelerators that can efficiently handle the increased computational demands." Unlike others, my main concern with AI is any savings we got from converting petroleum generating plants to wind/solar, it was blasted away by AI power consumption months or even years ago. Maybe Microsoft is on to something with the TMI revival.
- beepbooptheory 2y agoThis has been a constant thought for me as well. Like, the plan from what I can tell is that we are going to start to spinning all this stuff up every single time someone searches something on google, or perhaps, when someone would otherwise search something on there. Just that alone feels like an absolutely massive load to bear! But its only a drop in the bucket compared to everything else around this stuff. But while I may be thirsty and hungry in the future, at least I will (maybe) be able to know how many rs are in "strawberry".
- r3trohack3r 2y agoEnergy availability at this point appears (to me at least) to be practically infinite. In the sense that it is technically finite but not for any definition of the word that is meaningful for Earth or humans at this stage. I don't see our current energy production scaling to meet the demands of AI. I see a lot of signals that most AI players feel the same. From where I'm sitting, AI is already accelerating energy generation to meet demand. If your goal is to convert the planet to clean energy, AI seems like one of the most effective engines for doing that. It's going to drive the development of new technologies (like small modular nuclear reactors) pushing down the cost of construction and ownership. Strongly suspect that, in 50 years, the new energy tech that AI drives development of will have rendered most of our current energy infrastructure worthless. We will abandon current forms of energy production not because they were "bad" but because they were rendered inefficient.
- deleted 2y ago[deleted]
- peepeepoopoo95 2y ago[dead]
- WanderPanda 2y agoCrazy conglomerate discount on Alphabet if you can see TPUs as the only Nvidia competitor for training. Breaking up Alphabet seems more profitable than ever
- wrsh07 2y agoTPU is available to people outside of Google, but people prefer Nvidia It's not obvious to me that a hardware business manufacturing tpus for the general public is necessarily more valuable than one that benefits from tight integration with Google's internal software stack and datacenter tech It's orthogonal: tpu doesn't do much marketing and doesn't have to. Engineers at Google will use it because they have to and because of the huge cost advantage they get for using it. TPU is probably lacking the libraries that Nvidia has developed to be accessible to a broad array of use cases
- lanthissa 2y agoThis is the thing that makes no sense to me, if TPU's are even close to nvidia its a business worth as much as search. They could spend billions on just dx and get 100x that back on investment. The whole TPU line makes no sense to me, if its as good as it says (which is does seem to be) sell it publicly and add a trillion to your market cap. The only way this makes any sense is if inside google people legitimately think google cloud is going to be bigger than nvidia+ a lot of azure&aws, which seems crazy
- wrsh07 2y agoTPU is extremely high performance for Google because it can be optimized for Google's workloads. Google has absolutely world class data centers (think power efficiency) and TPU performs extremely well in them. (In turn: I expect Gemini is optimized to be trained and run on TPU) Google has vastly different constraints than an average startup or user of machine learning. The flexibility of Nvidia GPU's, the software, these things are really valuable for most people. Only recently with LLMs has the majority of uses started to look very very similar (some slightly modified transformer) Early versions of TPU required you to write your ml logic using a very restricted subset of tensorflow. (It had to be somewhat functional, etc) Normal people don't write code like that, and it's not worth it for them to re write a working model to run on a TPU because software engineers are expensive and Nvidia GPUs are really good and general.
- lanthissa 2y agoOkay I really dont understand this, nvidia has a 3.4T market cap google has a 2.4T post run up, and its PE is like 38 vs 25 so its a higher multiple on the business too. It appears making the best AI chip is a better business than googles entire conglomerate. If TPU's are really that good why on earth would google not sell them. People say its better to rent, but how can that be true when you look at the value of nvidia.
- LittleTimothy 2y agoI don't know why you'd be confused by the valuations, they're just 2 different businesses. Google is an ad sales business with an extremely strong revenue driver (youtube and Search) so it churns out money but it's pretty well understood revenue stream and is facing some possible disruption. Nvidia is a chip sales business and whether through luck or through skill (or both) they've essentially ended up as the monopoly provider of the worlds best chips. It's not unreasonable to expect Nvidia can squeeze a tonne of cash out of their customers for the next few years in an undisruptable way. So it's easy to understand the valuations. On the GPU side - One way to look at it is lets just ignore whether you're selling chips. Let's just look at where compute is done - because Google let's you rent time on the TPUs so if they're really great people will just choose to rent time on them. How much time is being rented on TPUs in the cloud vs. Nvidia GPUs? Google's TPUs are not a market leader, they're probably not even significant market share. Pretend Google decided to sell TPUs tomorrow - are many people who weren't choosing to rent TPUs going to decide to buy them? No. There's an element of interoperability and supply chain etc. etc. but I think the core of it is a full ML product the TPU isn't anywhere close to the Nvidia GPUs.
- lanthissa 2y agoI'd be confused by them choosing to not enter hte more lucrative business if they had the product to do so. The only way the current situation makes sense is if your last sentence is true, which is what i'm getting at. Either TPUs are significantly behind Nvidia chips, or google is choosing not to add like a trillion dollars of market cap to their business its really one or the other.
- 2y ago
- LittleTimothy 2y agoThe question I'd like to know the answer to is "What was the total cost of training Gemini 2.0 and how does it compare to the total cost to train equivalent capability models on Nvidia GPUs?". I'd be fascinated to know, and there must be someone at Google who has the data to actually answer that question. I suspect it's politically savvy for everyone at Google to pretend that question doesn't exist or can't be answered (because it would be an existential threat to the huge TPU project), but it would be absolutely fascinating. In the same way that Amazon eventually had to answer the "Soo.... how much money is this Alexa division actually making" question.
- aaron695 2y ago[dead]
- teleforce 2y agoIt's beyond me why processor with dataflow architecture is not being used for ML/AI workloads, not even in minority [1]. Native dataflow processor will hands down beats Von Neumann based architecture in term of performance and efficiency for ML/AI workloads, and GPU will be left redundant for graphics processing instead of being the default co-processor or accelerator for ML/AI [2]. [1] Dataflow architecture: https://en.wikipedia.org/wiki/Dataflow_architecture https://en.wikipedia.org/wiki/Dataflow_architecture [2] The GPU is not always faster: https://news.ycombinator.com/item?id=42388009 https://news.ycombinator.com/item?id=42388009
- almostgotcaught 2y agolet me ask you a very serious question and please answer honestly: have you ever tried to program a "dataflow processor"? if the answer is no then I invite you to try and then you will understand intimately why they're not being used for anything.
- germanjoey 2y agoSambanova's RDU is a dataflow processor being used for ML/AI workloads! It's amazing and actually works.