9 ms·
Amidst the noise and haste, Google has successfully pulled a SpaceX
- lmm 2y agoOK but vertical integration and low costs is only half of the problem. You also have to make a product that people want, otherwise the whole enterprise is meaningless. SpaceX is selling launches every week. What is Google selling that uses all those TPUs, and to whom?
- lokar 2y agoBehavioral advertising.
- dekhn 2y agoTPUs are used internally for products and research and also sold as a service thru Google Cloud. The list of TPU customers is pretty long. But Google could the product entirely internal and still reap plenty of benefit.
- mmaunder 2y agoGemini 2 was entirely trained on Trillium, Googles 6th generation TPU. About 100,000 of them. And all inference is run on TPUs. https://venturebeat.com/ai/google-new-trillium-ai-chip-delivers-4x-speed-and-powers-gemini-2-0/ https://venturebeat.com/ai/google-new-trillium-ai-chip-deliv...
- lmm 2y agoRight. But are people paying for Gemini? Does all that vertical integration make it cheaper than its competitors? (Which is what works for SpaceX)
- nmfisher 2y agoGemini Flash is my go-to for many client projects, so there's at least a few people paying for it. If it fits your use case, it's easily the best value for money, and often makes something commercially viable that wouldn't otherwise be. Sample size of 1, obviously, so YMMV. I also pay for Claude Pro for personal coding use, so I'm aware that Flash isn't exactly a drop-in replacement for a more powerful model. From my limited testing, though, Pro 2.0 is almost indistinguishable from 3.5 Sonnet.
- Reubend 2y agoEven if the LLM hype dies, and nobody at all wants to rent TPUs from GCP, then Google can use the TPUs to train and run large advertising models and improve the quality of their search. They themselves are a big enough customer to make this worth it. Any business from GCP is just icing on the cake. Keep in mind that TPUs can train non-LLM models too.
- m_mueller 2y agoAll the ad tech in the world won’t matter if people have stopped using the medium those ads are displayed on top of by then. I think the way people use the web is about to change drastically.
- fredski42 2y agoVery much this. I think the reason Google products are more and more infested with ads is to keep up revenue against people moving to better alternatives.
- jeffbee 2y agoGoogle has been charging people actual money to access features running on TPUs for years. I helped launch TPU inference features for Gmail in 2015. There may have been earlier systems running on TPU that I wasn't aware of.
- griomnib 2y agoTargeted ads use a lot of behavioral models. Google sells ads.
- kelvinjps10 2y agoJust killing the competition is enough with their integrated products?, like google search with ai features,(perplexity.ai) Ai on android now gemini replacing google voice (chat gpt and others).
- creato 2y agoMany of those launches are sold to itself (starlink), just like google uses a lot of its TPU capacity for its own purposes.
- dn762 2y agoAnd yet Amazon and Azure have more market share in Cloud. Whatsapp in chat. Microsoft in Office. Zoom in vid conf. The Telcos in fibre, Netflix and Disney more than Youtube etc etc. Google's enemies are internal ad tech managers, who aren't interested in any other team overshadowing them. If Android is better than iOS it wont be used to take out Apple. It will be used to sell more Ads. Their Adtech guys run the show.
- 7e 2y ago[flagged]
- Gabriel_Martin 2y agoWhile I do so love a good use of the word "wanker", this seems overly cynical to me, many systems which are tech islands succeed by growing an ecosystem, the console wars come to mind. Or maybe I'm misunderstanding you, certainly open to hearing more about your perspective Can you expand?
- Dig1t 2y agoThey are optimized for total cost of ownership, not pure speed. The idea is that they can provide excellent performance for an unbeatable price.
- kajecounterhack 2y ago> they're a tech. island because nobody can easily run their models on that stack You can easily train and serve models on TPU through Google's cloud services. Colab for example makes it easy to train models on TPU. > TPUs are not faster than H100s There are several generations of TPU with different characteristics, some oriented toward serving, some toward training, and all toward minimizing TCO. You might give up FLOPs but have much faster memory bandwidth, or be overall similar but have a much more efficient data pipeline thanks to Google's scaled infrastructure, netting out ahead. The OP isn't a wanker for pointing out that long-term infra investment is paying off for Google.
- shikon7 2y agoIf they have 10x cheaper compute, and a lot of cash to spare, why isn't Google dominating AI, like SpaceX is dominating the launch of satellites? Does Google not want to, or is there another reason they cannot?
- mmaunder 2y agoThere seems to be a consensus that we haven't yet seen the killer AI app that produces ROI that justifies the level of investment we're seeing. If that's true, then it's all still up for grabs. My own view is that what AI is going after i.e. the size of the pie, is the $50 trillion per year portion of global GDP that makes up wages. Putting aside the risks - which are enormous - when you see it that way you begin to understand openai's valuation late last year, and that that's just the tip of the iceberg of what's coming. The risk of disrupting a sizable chunk of the world's wages is fuel for a separate post and conversation. IMHO it's the real risk of AI, and the Skynet scenario around ASI/AGI that has been popularized is a distraction.
- janalsncm 2y agoNone of that matters if you are selling shovels. Cash is cash. Google could be selling TPUs and directly competing with Nvidia. You might think Nvidia’s valuation is inflated, but the fact of the matter is they sold $60B in hardware last year.
- mmaunder 2y agoNvidia is selling shovels. Google will be selling gold if successful. And they make their own shovels.
- m_mueller 2y agoWhen I look at current ChatGPT I honestly think that the gold has been found, and a significant amount of it is undermining Google’s home market, the presentation and aggregation of information on the web. Their product has become so much worse over the years while ChatGPT has improved many fold. It’s a better search engine among many other things.
- deleted 2y ago[deleted]
- janalsncm 2y agoThe author is right about the vertical integration. It’s just that none of the stages are great. I don’t use Google cloud and have no interest in learning its idiosyncrasies. And I have PTSD from Tensorflow. (It’s only a bit of an exaggeration to say no one outside of Google uses it anymore.) Using TPU in PyTorch is possible though. I wish Google would compete with Nvidia directly and let me buy a TPU (the big boy ones, not the flash drive sized Corals). Because as it stands now, Google could have Nvidia’s business and OpenAI’s.
- kajecounterhack 2y ago> It’s just that none of the stages are great. Really? Nobody uses TF but plenty of people use XLA, which is what you allude to saying that TPU in Pytorch is possible. That's arguably the most important piece; the sledgehammer that makes good performance and good devex compatible. > I wish Google would compete with Nvidia directly and let me buy a TPU I don't see the utility of buying _any_ GPUs. It's mainly a cost + availability optimization to own them yourself if you're doing a lot of continual training. Outside of the foundation model companies, most of us just use cloud services -- and I want those to be cheap and always use the latest thing. Google has a highly optimized infrastructure to make sure unused resources (CPU, TPU, RAM, disk) are properly allocated. This means preemptible instances, highly co-located compute + data, etc. In theory everyone should win when you use ML hw through hyperscalers, because they collect on their structural advantages and your TCO is lower. > Google could have Nvidia’s business and OpenAI’s They arguably have a better _business_ on their hands but a worse _speculative outlook_ in the eyes of Mr. Market. Those are different things. There's an argument to be made that Google's valuation could be as high if they ran their public relations strategy as well as OpenAI / Nvidia / Tesla, which are all riding monumental hype.
- janalsncm 2y ago> most of us just use cloud services Yes, but because cloud services buy GPUs almost exclusively Nvidia, it would likely drive cloud prices down as well.
- 2y ago
- motohagiography 2y agoif AI is a weapon, google wins, but if AI is a product, apple wins. google can build compute capacity, but solving cringe at their products isn't a compute problem. products are hit songs, who here is going to write them.
- askafriend 2y agoSince it's both, both companies will win big.
- motohagiography 2y agothe "why not both?" worldview is kind of muddying and neutralizing to conversation tho. really, one model will prevail.
- askafriend 2y agoDid one Cloud prevail?
- FabHK 2y agoMinor nitpick: > NVidia’s margin on the H100 is 1000%. That means they’re selling it for 10X what it costs to produce If the margin is 1000%, they're selling it at 11x what it costs to produce. Were they selling it at 10x, their margin would be 900%.
- mmaunder 2y agoDon’t you just love percentages? Thanks.
- chillee 2y agoOne of the big things this article misses is that Google pays Broadcom a significant amount for the actual chip design, also around a 70% margin. Google certainly has infra/cost advantages, but it's nowhere near 10x.
- mmaunder 2y agoMind sharing your source on that? I’ve been trying to find one. Edit: Specifically the nature and current status of the Broadcom/Google relationship as it relates to TPUs.
- chillee 2y agohttps://www.theinformation.com/articles/to-reduce-ai-costs-google-wants-to-ditch-broadcom-as-its-tpu-server-chip-supplier?rc=rgbphv https://www.theinformation.com/articles/to-reduce-ai-costs-g... Which takes it from > Broadcom generates a 70% profit margin from its work on TPUs, said a person with direct knowledge of the internal analysis. SemiAnalysis, a chip research firm, earlier reported that figure. https://semianalysis.com/2023/08/30/broadcoms-google-tpu-revenue-explosion/ https://semianalysis.com/2023/08/30/broadcoms-google-tpu-rev...
- mmaunder 2y agoThanks!
- zoratu 2y agoInteresting allusion to Desiderata, “Go placidly amid the noise and the haste, and remember what peace there may be in silence.”