9 ms·
This seems to be competing directly with Google's TPU pods. Looks like TPU v4 has a 300 GB/s interconnect, and 32 GB HBM per chip * 4096 chips = 131 TB (which i
by modeless 3y ago
This seems to be competing directly with Google's TPU pods. Looks like TPU v4 has a 300 GB/s interconnect, and 32 GB HBM per chip * 4096 chips = 131 TB (which is all HBM, so higher bandwidth than the LPDDR in Nvidia's system). So yeah, Nvidia's interconnect seems better. However, TPU v4 was deployed in 2020 (!) and Nvidia's thing won't be ready until next year. I've gotta imagine that TPU v5 has already been deployed internally for a while now, but hasn't been disclosed yet. Who knows, TPU v6 might even be deployed before this Nvidia thing.
- tbruckner 3y agoSome of the specs seem inaccurate here, HBM has been present in NVIDIA datacenter GPUs for awhile now. LPDDR is for their gaming hardware.
- wmf 3y agoThe GH200 uses a combination of HBM3 for the GPU and LPDDR5 for the CPU but it's a unified memory system so the GPU can access all the RAM. Gaming GPUs use GDDR which is a third flavor.
- the_svd_doctor 3y agoThe network topology of TPUv4 is far far inferior though. It's a torus. No switches.
- renonce 3y agoIf TPUv4 pods were so powerful, why are most new models trained with NVIDIA cards rather than TPU?
- llm_nerd 3y agoThere is a mythology to Google's TPU that is not validated by real world numbers. Where we can actually test (I mean -- TPUv4 pods are available right now on their cloud) it is very good, but remains uncompetitive with the h100. I mean, Google disclaims that you shouldn't compare it, doing the classic "the h100 is on a better process node so it's unfair". People will always point at a mythical next generation that is surely way better, despite the fact that Google is currently building big supercomputers with their TPUv4. And in Google's shootout, again comparing with the last generation of nvidia hardware (the A100), Google's biggest advantage was in the connection fabric, which with this DGX GH200 nvidia not only overcame, but bested by a magnitude. More competition would be fantastic. Better pricing at scale would be fantastic. But there is absolutely no doubt that nvidia is far ahead of Google right now. Tesla made some believably pushing claims about their own efforts with their own hardware, so who knows maybe they're the real challenger.
- boyka 3y agoAvailability only on GCP and in particular cost.
- fulafel 3y agoGoogle has advertised that they have better perf/$ than GPUs, is this wrong or do you just mean absolute cost (so not available in small enough slices)? edit: actually now i can't find the claim, maybe i misremember what the papers said.
- mochomocha 3y agoPerf/$ where $ is what it cost _them_ , not $ they're ready to sell to others as a product. Cloud margins in the high two-digit percents are typical, and I'd imagine even higher for very specialized products in high-demand from deep-pocketed customers.
- sanxiyn 3y agohttps://arxiv.org/abs/2304.01433 https://arxiv.org/abs/2304.01433 does claim "1.2x-1.7x faster and uses 1.3x-1.9x less power than the NVIDIA A100".
- ShamelessC 3y agoThese are just my guesses but: Software for TPU is still in its early stages. CUDA is well established. You can test on a gaming GPU that you can find (locally!) in many markets. XLA is meant to solve this, but first impressions matter and my first impression was that it has not yet "solved" this issue. TPU is only available via Google Cloud - as far as I know they don't have NVIDIA's widespread distribution to various HPC/supercomputer systems. This also has implications on scaling up more than a few pods, as they will need to be colocated with speedy interconnect (which is provided by the various existing HPC systems that use NVIDIA's chips). Finally, I think many people are discovering that the supposed benefits of TPU are marginal at best in the face of the types of natural scaling issues that both GPU's and TPU's suffer from when scaling out to e.g. hundreds of pods. I'm certain that someone with more experience than I could give a better answer though - and again, all speculation. I refuse to use TPU because Google Cloud's system for getting access to said TPU's was horrible for me when I tried it. I believe John Carmack has a nice tweet thread specifying the same issues I ran into. In general, Google has a habit of developing tech for other Googlers first, and as such winds up ignoring a lot of real-world scenarios faced by researchers/practitioners. NVIDIA on the other hand has been working directly with a ton of institutions and businesses ever since the inception of CUDA. That their TPU's have seen any adoption at all is mostly due to their research program which granted very cheap access to TPU's to tons of people.
- modeless 3y agoTo add to the other answers, TPUv4 was not released to cloud customers until last year. And I bet availability is not as good as GPUs, even in Google Cloud (obviously TPUs are not available at all in other clouds).
- ioedward 3y agoTPUs are mostly hoarded by Google Research (including Deepmind) and Ads. Very few are being used by external people.
- vintermann 3y agoThe greatest artificial minds of our generation are thinking about how to make us click on ads?
- ktta 3y agoIt's been that way for over a decade now. Welcome
- deleted 3y ago[deleted]
- philjohn 3y agoIt was ads that made the money to develop the artificial minds in the first place.
- lubesGordi 3y agoYeah, and look at how some very simple clustering ML/recommender systems impact social/political dynamics all to keep people engaged on the site and maximize chances to click ads ( see youtube/facebook, etc. ).
- temp0826 3y agoEx Machina vibes
- exikyut 3y agoThis is technically correct to the extent of paperclip maximization and I don't like it.
- deleted 3y ago[deleted]
- imsaw 3y agoFrom a personal use case, the number of instructions available in TPUs are still limited and some workaround is needed when designing custom layers. Even if it's available in platforms like Colab or Kaggle, people still lean to GPUs as it is more versatile.
- ijidak 3y agoIs HBM mostly Samsung?
- ec109685 3y agoWhat evidence is there that Google would be able to out compete nvidia on AI hardware?
- madaxe_again 3y agoNone. Heck, I can’t even search my gmail effectively any more, so if they can’t maintain a core product, I doubt they can build a new one of any quality. alphabet are now just a big, bloated catch-up corporation running on inertia and past glory. I don’t think they will exist in 10 years.
- Tyr42 3y agoGmail dev here (but not search), I don't think anything has changed with search. Operators still work too. What's actually wrong? Do you just have more email now?
- ec109685 3y agoIf you don’t match the terms in your email exactly right with your query, gmail starts returning email that matches one of the terms which is rarely what you want.
- ddalex 3y agoWhat would you expect it to do? How is it supposed to know what you want if you don't provide it with the exact search terms? Shouldn't it do partial matches if there are no full matches ?
- ec109685 3y agoIt should look for things that match the meaning of the words searched. E.g. what’s possible with embedding where the query terms are matched with similar messages that mean the same thing.
- jeodjdodh 3y ago
- dweekly 3y agoJust want to flag a potential unit issue: 900gbps vs 300GB/s? Also worth noting - TPUv4 uses a 6-way 3D torus interconnect vs the 3-way "multi ToR" NVLINK topology; the total bisection bandwidth of the TPUv4 pod is over 1PB/s! Can't wait to see what TPUv5 looks like. As you say, it's probably already chugging away with v6 on track to tape out in a year. That said, I think NVidia has nailed bringing the ecosystem along, and I think making the whole setup look more like "one huge GPU" could simplify a lot of ML programming. I am actually disappointed I haven't seen more of that style in CPU programming. Where's my 20,000 core 100TB RAM VM instance?
- llm_nerd 3y ago>900gbps vs 300GB/s? The nvidia device uses a fabric with 900 GBps switched fabric between any of the 256 nodes in the system. The TPUv4 3d torus network is basically a ring network of 56 GBps connections creating separate rings. From a raw perspective, the nvidia solution is the overwhelming winner. There is absolutely no contest.
- deleted 3y ago[deleted]
- mike_hearn 3y ago> Where's my 20,000 core 100TB RAM VM instance? Machines with terabytes of RAM do exist and get used - working well on such setups is a goal of modern JVM GCs for instance - but making a single machine that large which acts like a single machine isn't easy, nor especially desirable. One machine is a unified failure domains outside of mainframe-land, so if you had a 20k core machine with 100TB of RAM you could never reboot it to apply OS updates and it'd die all the time from failed parts. Even once you get beyond that most software stacks use locking and stop scaling beyond a few hundred cores at best and that's assuming very heavily optimized stacks. AI workloads are easier because they're designed from scratch to be inherently parallel without lots of little locks and custom data structures all over the place like a regular computer has. Disk storage is one of the places where you can parallelize and scale out relatively easily and you do see datacenter sized disks there.
- 3y ago