10 ms·
Posts like this underscore why the smart money is betting on Google as the long term AI winner. Meta, Microsoft, OpenAI, etc. are trying to address problems wit
by mike_d 2y ago
Posts like this underscore why the smart money is betting on Google as the long term AI winner. Meta, Microsoft, OpenAI, etc. are trying to address problems with consumer video cards and spending billions to try and out bid each other to win Nvidia's favor - while Google is on their 6th generation of custom silicon.
Literally the only thing that can stop Google now is the fact they keep bringing Microsoft and Oracle flunkies into leadership positions.
- coralreef 2y agoHow do TPUs perform compared to GPUs on LLMs and image generation?
- smarterclayton 2y agoPretty well. Anthropic runs some of Claude inference on GKE and TPU v5e - this talk at the last GCP conference has some details: https://youtu.be/b87I1plPeMg?si=T4XSFUzXG8BwpphR https://youtu.be/b87I1plPeMg?si=T4XSFUzXG8BwpphR Ecosystem support for GPU is very strong and so smaller teams might find TPUs to have a steeper learning curve. Sophisticated users can definitely get wins today on TPU if they can get capacity. And it’s not as if nvidia is standing still, so how the strengths of each change over future generations isn’t set either. Ie TPU are “simple” matrix multipliers and also optimized for operation at scale, GPU are more general purpose and have strong ecosystem power. Disclaimer - work on GKE on enabling AI workloads.
- zdyn5 2y agoH100s are far from consumer video cards
- stygiansonic 2y agoYeah, ops comment makes it seem like they are building racks of RTX 4090s, when this isn’t remotely true. Tensor Core performance is far different on the data center class devices vs consumer ones.
- mike_d 2y agoThey are building racks of 4090s. Nobody can get H100s in any reasonable volume. Hell, Microsoft is renting GPUs from Oracle Cloud to get enough capacity to run Bing.
- kkielhofner 2y agoWho is "they"? RTX 4090s are terrible for this task. Off the top of my head: - VRAM (obviously). Isn't that where the racks come in? Not really. Nvidia famously removed something as basic as NVLink between two cards from the 3090 to the 4090. When it comes to bandwidth between cards (crucial) even 16 lanes of PCIe 4 isn't fast enough. When you start talking about "racks" unless you're running on server grade CPUs (contributing to cost vs power vs density vs perf) you're not going to have nearly enough PCIe lanes to get very far. Even P2P over PCIe requires a hack geohot developed[0] and needless to say that's umm, less than confidence inspiring for what you would lay out ($$$) in terms of hardware, space, cooling, and power. The lack of ECC is a real issue as well. - Form factor. Remember PCIe lanes, etc? The RTX 4090 is a ~three slot beast when using air cooling and needless to say rigging up something like the dual slot water cooled 4090s I have at scale is another challenge altogether... How are people going to wire this up? What do the enclosures/racks/etc look like? This isn't like crypto mining where cheap 1x PCIe risers can be used without dramatically limiting performance to the point of useless. - Performance. As grandparent comment noted 4090s are not designed for this workload. In typical usage for training I see them as 10-20% faster than an RTX 3090 at much higher cost. Compared to my H100 with SXM it's ridiculously slow. - Market segmentation. Nvidia really knows what they're doing here... There are all kinds of limitations you run into with how the hardware is designed (like Tensor Core performance for inference especially). - Issues at scale. Look at the Meta post - their biggest issues are things that are dramatically worse with consumer cards like the RTX 4090, especially when you're running with some kind of goofy PCIe cabling issue (like risers). - Power. No matter what power limiting you employ an RTX 4090 is pretty bad for power/performance ratio. The card isn't fundamentally designed for these tasks - it's designed to run screaming for a few hours a day so gamers can push as many FPS at high res as possible. Training, inference, etc is a different beast and the performance vs power ratio for these tasks is terrible compared to A/H100. Now lets talk about the physical cabling, PSU, etc issues. Yes miners had hacks for this as well but it's yet another issue. - Fan design. There isn't a single "blower" style RTX 4090 on the market. There was a dual-slot RTX 3090 at one point (I have a bunch of them) but Nvidia made Gigabyte pull them from the market because people were using them for this. Figuring out some kind of air-cooling setup with the fan and cooling design of the available RTX 4090 cards sounds like a complete nightmare... - Licensing issues. Again, laying out the $$$ for this with a deployment that almost certainly violates the Nvidia EULA is a risky investment. Three RTX 4090s (at 9 slots) to get "only" 72GB of VRAM, talking over PCIe, using 48 PCIe lanes, multi-node over sloooow ethernet (hitting CPU - slower and yet more power), using what likely ends up at ~900 watts (power limited) for significantly reduced throughput and less VRAM is ridiculous. Scaling the kind of ethernet you need for this (100 gig) comes at a very high per-port cost and due to all of these issues the performance would still be terrible. I'm all for creativity but deploying "racks" of 4090s for AI tasks is (frankly) flat-out stupid. [0] - https://github.com/tinygrad/open-gpu-kernel-modules https://github.com/tinygrad/open-gpu-kernel-modules
- hooloovoo_zoo 2y agoGoogle has been working on TPUs and Tensorflow for a decade with pretty mixed success; I don't think it's clear that they're going to win.
- ai4ever 2y agohere is an older take on this same topic.. https://www.yitay.net/blog/training-great-llms-entirely-from-ground-zero-in-the-wilderness https://www.yitay.net/blog/training-great-llms-entirely-from... GPU vs TPU, and good software managing large clusters of them across all sorts of failure. the funny bit from the above article is the incident when someone forgot about a training job at google, and month later had the model fully trained without an alert of any kind. "outrageously good infra"
- dinobones 2y agoDo you really think Google’s hardware expertise is better than Nvidia’s? If needed these other companies have the $$$ to buy the best chips money can buy from Nvidia. Better chips than Google could ever produce. If anything, this is why IMO Google will fail.
- candiddevmike 2y agoI thought NVIDIA's moat was mostly software/CUDA?
- rapsey 2y agoThey have by far the fastest chip, the best software with CUDA and feverishly working on next gen chips. They also bought out multiple years worth of global high bandwidth memory manufacturing capacity. No one will beat them at their game. However if there are any major breakthroughs that might render those processing capacities unneeded, or the major players hitting a wall regarding AI spending, then they will take a massive hit. It will come eventually because the chip business is always in boom/bust cycles.
- mike_d 2y agoYes. Google was building custom HPC hardware 5-8 years before Nvidia decided to expand outside the consumer and "workstation" markets.
- rapsey 2y agoTheir work on folding and ai could very well be a business worth in the hundreds of billions and they know it as well as many other bets. Wheres others are playing the llm race to the bottom.
- htrp 2y ago> Their work on folding and ai could very well be a business worth in the hundreds of billions and they know it as well as many other bets. And we'll be writing case studies of how they squandered billions of R&D to help found other companies (kinda like Xerox Parc) Almost every interesting paper after transformers has had it's authors leave to commercialize their own companies.
- deleted 2y ago[deleted]
- throwaway920102 2y agoWhat can stop Google is building the wrong thing, or being so scared to launch anything that they smother their own fledgling products before they are born or before they can mature. Their product and finance and accounting teams should be tossed.
- throwaway_ab 2y agoI think it's likely Nvidia's GPU's, many of which are $50,000+ for a single unit, far surpass Google's custom silicon otherwise why wouldn't Google be selling shovels like Nvidia? If Google had a better chip, or even a chip that was close, they would sell it to anyone and everyone. From a quick search I can see Google's custom chips are 15x to 30x slower to train AI compared to Nvidia's current latest gen AI specific GPU's.
- candiddevmike 2y agoThey do sell shovels, you can get Google TPUs on Google Cloud.
- matt-p 2y agoExactly and they are still about 1/18ths as good at training llms as a H100. Maybe they are less than 1/18ths the cost, so google technically have a marginally better unit cost but i doubt it when you consider the R&D cost. They are less bad at inference, but still much worse than even an A100.
- jeffbee 2y agoI don't see how you can evaluate better and worse for training without doing so on cost basis. If it costs less and eventually finishes then it's better.
- tmostak 2y agoThis assumes that you can linearly scale up the number of TPUs to get equal performance to Nvidia cards for less cost. Like most things distributed, this is unlikely to be the case.
- logicchains 2y agoThis is absolutely the case, TPUs scale very well: https://github.com/google/maxtext https://github.com/google/maxtext .
- loeg 2y agoThe only thing that can stop Google is Google. Somehow every bet that isn't Search doesn't pan out. And inexplicably, they're working hard to kill Search now. As a shareholder, I hope they succeed. But I am more pessimistic about it than you.
- deleted 2y ago[deleted]
- yellow_postit 2y agoAnd they missed multiple waves of effectively building on their own in house research.
- deleted 2y ago[deleted]
- aworks 2y agoReminds me of Xerox Parc vs. Apple. Building successful products is hard.
- deleted 2y ago[deleted]
- tw04 2y agoExcept Microsoft is making their own chips as well? https://www.theverge.com/2023/11/15/23960345/microsoft-cpu-gpu-ai-chips-azure-maia-cobalt-specifications-cloud-infrastructure https://www.theverge.com/2023/11/15/23960345/microsoft-cpu-g...
- jauer 2y agoand so is Meta: https://ai.meta.com/blog/next-generation-meta-training-inference-accelerator-AI-MTIA/ https://ai.meta.com/blog/next-generation-meta-training-infer...
- boringg 2y agoEver since Apple did it everyone has leaped on board. Let's see how things pan out for everyone...
- jacurtis 2y agoAnd that's why $ARM is a good buy. Selling swords and steel to all these armies as they go to war.
- silisili 2y agoARM is just collecting royalties in this space. Their reference designs aren't exactly competitive. I'm not saying it's a bad buy, either, but if and when they turn the screws, there will be a mass exodus. Solid long term play perhaps, but not going to see Nvidia like price action.
- 1024core 2y agoGoogle introduced their first TPU in 2015...? Long before Apple taped out their first silicon.
- fragmede 2y agoif we're talking custom silicon, Google acquired Motorola in 2011, and Apple acquired PA semi in 2008. The idea is obvious to everybody in the industry, it's a question of money and motivation.
- moneywoes 2y agoIs no one else working on custom silicon?
- fnordpiglet 2y agoThe problem isn’t just developing your own processor. Nvidia has a huge stack of pretty cutting edge technology including a huge stack from mellanox, an enormous OSS tool chain around CUDA, etc, that people seeking to make comparable products have to overcome.
- sangnoir 2y agoAre you suggesting Meta or Google - who stand to save billions - won't be able to get top performance from their custom chips because their tooling/hardware won't support CUDA?
- fnordpiglet 2y agoNo. I’m suggesting they won’t because IP like the mellanox treasure chest they acquired is ridiculously difficult to develop and Nvidia has aggressively exploited it, along with their other already advanced IP in the space of their -core business-. I understand, especially amongst googlers, there’s a belief there are no others smarter than a googler. But it’s simply not the case. Nvidia is excellent at their core competencies and business, which is making absurdly parallel compute platforms with absurdly powerful interconnects. I’m saying google or meta won’t beat Nvidia at hardware. I’d also point to the fact Nvidias ability to raise capital is the best on earth now, so even money isn’t a barrier. The advantage CUDA gives is in the tool chains, libraries, research, and all that that tens of thousands of people are contributing to as part of their jobs, research, and hobbies. This is almost -more valuable- than getting top performance. Getting top techniques, top software, top everything by having everyone everywhere working to make the ecosystem of your stuff is invaluable. Google won’t have that. They will just have the hubris of googlers who believe they’re smarter. I would also note that at this phase of a cycle in tech trying to save billions takes your eye off the prize. Cost optimization comes much later after the market has been fully explored and directions are clean and diminishing returns on R&D kick in. Any company that doesn’t recognize that is run by CPA and deserves the ignominy they’ll face.
- HarHarVeryFunny 2y agoThe chips are somewhat irrelevant. It's the overall system architecture, management, and fault recovery that matters.
- matt-p 2y agoCan't agree. This is like saying $popularApp will fail because they buy expensive hosting at AWS. Rubbish they will fail because the product didn't fit the market, if they're successful they'll have money to buy servers and colo then drive down cost. If they succeed it will be in large part due to the fact they spent thier capital and more importantly time on code/engineers rather than servers. Right now companies are searching for a use of AI that will add hundreds of billions to thier market cap. Once they find that they can make TPUs, right now only one thing matters; getting there first.
- deleted 2y ago[deleted]
- mike_d 2y ago> This is like saying $popularApp will fail because they buy expensive hosting at AWS. For any given mobile app startup, AWS is effectively infinite. The more money you throw at it the more doodads you get back. Nvidia's supply chain is not infinite and is the bottle neck for all the non-Google players to fight over.
- KaiserPro 2y agoIF you are training on AWS, its not infinite. worse still you are bidding against other people.
- r_hanz 2y agoMuch the same way you can have all the best gear and still fail - Google’s primary strength seems to be the DeepMind group. I’m not affiliated with Google, but IMHO the reason they will slowly die is because their engineering culture has taken a backseat due to their broken hiring practices. Bad hiring practices aren’t exclusive to them, but from all accounts it seems like their internal focus is on optimizing ad revenue over everything else. I could be wrong or misinformed, but it seems to me like they are playing the finite game in the AI space (DeepMind group aside) while FAIR are playing the infinite game. *meanwhile MSFT are simply trying to buy their way to relevance (e.g. OpenAI investments, etc) and carve out future revenues (Recall) and Jobs-less Apple is building their trademark walled-garden (AppleIntelligence?). Although the use of unified memory in Apple silicon poses some interesting possibilities for enabling the use of sizable models on consumer hardware. Overall it seems like “big-tech” is by-and-large uninspired and asleep at the wheel save specific teams like those led by Lecun, Hassabis, etc. not sure where that leaves OpenAI now that Karpathy is gone.
- VirusNewbie 2y ago>ecause their engineering culture has taken a backseat due to their broken hiring practices. What company do you think has better hiring practices, and subsequently a higher talent pool? Meta is pretty similar to Google's (though with an emphasis on speed over creativity). Microsoft is certainly worse at hiring than the two aforementioned...
- r_hanz 2y agoTo be fair, I don’t have any examples of “good practices” readily in-hand. However, I did try to address why I thought others were less impacted by this problem in the second half of my post.
- Jabrov 2y ago"Consumer video cards"? Meta's not building their clusters out of 3090s. They're using advanced cards meant for data centers and machine learning -- almost effectively "custom silicon"
- blackoil 2y agoI would call it "stupid" money. This isn't a commodity business. Value of the final product is orthogonal to amount invested in compute. If Google is 10% slower or its product is 10% worse, it can lose all the value. This is like valuing a software company higher because its devs are using cheap PC desktops instead of Mac.
- lxgr 2y agoI wish I had your faith in Google’s ability to refrain from kneecapping their own perfectly fine product.
- callalex 2y agoThat’s how all huge tech companies become dinosaurs though. Upper management that is already stupidly wealthy (and therefore unmotivated) have the funding and patience to hire geniuses to build incredible machines and then constantly tie their shoelaces together while asking them to sprint. Examples include Microsoft and Oracle as you said, and before them IBM, AT$T, TIBCO, Marvell, Motorola, I could go on for a while…
- xcv123 2y agoNone of these companies are using consumer video cards. https://www.nvidia.com/en-us/data-center/h200/ https://www.nvidia.com/en-us/data-center/h200/
- zmmmmm 2y agoCustom silicon is fantastic when things have stabilised and you know exactly want you want. But things are still evolving fast in real time and in that environment, whoever can move fastest to be ultra flexible and deploy the latest architecture as soon as possible is the winner. I think in a nutshell, that is the story of Nvidia's success here : they created a GPGPU platform with just the right level of abstraction to capture the market for AI research.
- htrp 2y ago> Literally the only thing that can stop Google now is the fact they keep bringing Microsoft and Oracle flunkies into leadership positions. You mean the thing that's already stopped them? If they had seriously invested into the TPU ecosystem in 2015, they would already have "won" AI.
- fragmede 2y agosmart money has a diversified portfolio and isn't betting on any one winner and has invested in all of them, and then some.
- asynchronous 2y agoLaughable response when you actually look at the quality of the algorithms being produced by Google. They’re so behind it’s embarrassing.
- raincole 2y ago^ How to compact as many mistakes as possible in one single comment. 1. Google's stock didn't siginificantly outperformed Meta, Microsoft, etc, in thet past two years. 2. Meta and Microsoft are trying to make their own chips as well. 3. They're not using "consumer video cards" to train AI. I don't even know if you can call these beasts video cards any more. H100 doesn't have HDMI port.
- deleted 2y ago[deleted]
- hipadev23 2y agoGoogle’s on their 6th generation and still can’t find anyone to use it. Hmm.
- guardiang 2y agoGoogle is old like MetLife, relative to each's respective industry. Both are carrying too much baggage and are top heavy. As a result, I personally don't think Google will be able to keep pace with OpenAI in the long run.
- houseplant 2y agoafter everything I've seen and the litigation coming out of europe, I really can't see AI lasting long after they're obligated to prove rights for the data they're training on. they can't get away with having scraped people's owned work forever. You can't steal things from workers and then undercut them by selling that hard work for pennies, and not expect everything to collapse. I mean, I know that the folks in charge of this aren't really known for their foresight, especially when stock numbers and venture capital are the entire point, but... surely I hope people can recognize that this can't go on unimpeded.
- logicchains 2y agoEventually they're going to put vision LLMs in robotic bodies and they'll be able to learn just by listening and watching, just like humans, at which stage the idea that they're "stealing" just by viewing content will be seen as absurd.
- Der_Einzige 2y agoUntil the lion share of AI projects support that custom silicon, I will continue to bet on anyone buying Nvidia GPUs.
- iamflimflam1 2y agoDon’t forget Apple’s Private Compute Cloud - built on top of Apple Silicon.
- towawy 2y agoModels trained on Google TPUs according to Reuters [0]. Does anyone know the "technical document" the news article references? [0] https://www.reuters.com/technology/artificial-intelligence/how-apple-used-googles-help-train-its-ai-models-2024-06-11/ https://www.reuters.com/technology/artificial-intelligence/h...
- astrange 2y agohttps://machinelearning.apple.com/research/introducing-apple-foundation-models https://machinelearning.apple.com/research/introducing-apple...
- jatins 2y agothis was the same argument that was presented a decade ago on why Google was supposed to win the cloud because their internal infra was miles ahead of Amazon and Microsoft. Yet here we are. Will the consumer video cards get cheaper and better faster or will Google's directors' infighting stop first?
- checkyoursudo 2y agoWhat would any company as "the long term AI winner" look like? What would it mean to be the winner in this context?
- ketchupdebugger 2y agoThe winner is just nvidia, I see this like the battle of gas vs electric cars. Nvidia is basically making the wheels. whichever company wins, you'd still need wheels.
- KaiserPro 2y ago> Meta, Microsoft, OpenAI, etc. are trying to address problems with consumer video cards Yes, but you are buying access to tested, supported units that are proven to work, don't require custom software, and are almost plug an go. When its time to upgrade, its not that costly. Designing, fabricating and deploying your own silicon is Expensive, creating software support for it, also more expense. THen there is the opportunity cost of having to optimise the software stack your self. You're exchanging a large capex, for a similar sized capex plus a fuckton of opex as well.
- baskire 2y ago[dead]
- jejeyyy77 2y agololwat. google is the biggest loser in all of this.
- rldjbpin 2y agobeing first/early doesn't always mean long-term dominance, see tensorflow. at consumer level, npu are becoming a useful accelerator, and here google can choose to become the platform of choice. but nobody comes close for training workflow but nvidia as it stands. imho it is currently possible thanks to community efforts afforded by cuda being the only realistic option. politics and leadership aside, it would be nice to sustian a market for highly efficient matrix multipliers. also a s/w ecosystem that finally makes multiprocessing workloads easy to integrate for dummies like me.