9 ms·
Advances in semiconductors are feeding the AI boom
- ViktorRay 3y agoThis article has two authors. One author is chairman of TSMC. The other author is Chief Scientist of TSMC. This is important to note because they clearly know some stuff and we should listen.
- bigbillheck 3y agoTwo credited authors.
- tzm 3y agoCerebras WSE-3 contains 4 trillion transistors and 8 exaflops per sec, 20 PB bandwidth. 62 times the cores of an H100.. 900,000. I wonder if the WSE-3 can compete on price / performance though. Interesting times!
- winwang 3y agoHow's the single core-to-core bandwidth?
- jsheard 3y agoIs anyone actually using those WSEs in anger yet? They're on their third generation now, but as far as I can tell the discussion of each generation consists of "Cerebras announces new giant chip" and then radio silence until they announce the next giant chip.
- JonChesterfield 3y agoThey sold some. Not strictly speaking the same as using any but there's a decent chance some code is running on the machines.
- rthnbgrredf 3y agoProblem is Software. You can put out a XYZ trillion monster chip that beats anything hardware wise, but it is going nowhere if you don't have the tooling and massive community (like Nvidia has) to actually do some real A.I. stuff.
- IshKebab 3y agoUnlikely. They cost so much that nobody is going to do research on them - at best it's porting existing models. And they're so different to GPUs that the porting effort is going to be enormous. They also suffer from the global optimisation problem for layout of calculations so compile time is going to be insane. Their WSE technology is also already obsolete - Tesla's chip does it in a much more logical and cost effective way.
- shrubble 3y agoThe Cerebras-2 is at the Pittsburgh Supercomputing Center. Not sure if they ordered a 3.
- monocasa 3y ago> 62 times the cores of an H100.. 900,000. More than that arguably. CUDA cores are more like SIMD lanes than CPU cores like cerebras's usage of 'core'. Since they have 4 wide tensor ops on cerebras, there's arguably 3.6M CUDA equivalent cores.
- AnimalMuppet 3y ago9 trillion flops per core? That's... mind-boggling. Is that real? And, 9 trillion flops per core in 4.4 million transistors per core. That sounds a bit too good to be true.
- xcv123 3y agoNo the real performance is 125 petaflops at FP16. That is 138 gigaflops per core. You would need 64 of these to get 8 exaflops. https://www.tomshardware.com/tech-industry/artificial-intelligence/cerebras-launches-900000-core-125-petaflops-wafer-scale-processor-for-ai-theoretically-equivalent-to-about-62-nvidia-h100-gpus https://www.tomshardware.com/tech-industry/artificial-intell...
- xcv123 3y ago> 8 exaflops per sec Your number is off by 64x. It can do 125 petaflops at FP16 https://www.tomshardware.com/tech-industry/artificial-intelligence/cerebras-launches-900000-core-125-petaflops-wafer-scale-processor-for-ai-theoretically-equivalent-to-about-62-nvidia-h100-gpus https://www.tomshardware.com/tech-industry/artificial-intell...
- ramshanker 3y agoWhen the limits of digital reaches the boundary of physics, Analogue is going make a comeback. Human brain feels nearer to Analogue than Digital. I will be surprised to reach AGI without nearing the Order-of-Magnitude of brain processing. We need that ONE paper on analogue to end this quest of trillions and counting transistors.
- Teever 3y agoWon't that mean that we just change the quest from higher and higher density of digital elements to higher and higher density of analog elements? Like people weren't trying to make computers out of bigger and bigger tubes before the transistor, they were trying to make them out of smaller and smaller ones.
- bigyikes 3y agoComputers are going to increase in size as they consume more and more power, requiring increasingly elaborate cooling systems. Like how the transistor made the big and hot vacuum tubes obsolete, maybe we’ll see some analog breakthrough do the same thing to transistors, at least for AI. I doubt there is a world where we use analog for general purpose computing, but it seems perfect for messy, probabilistic processes like thinking.
- enlyth 3y agoWhat's amazing is that the human brain does it all on the equivalent of like 20 watts of power. That's basically a couple of LED light bulbs.
- passion__desire 3y agoIs there a comparison of power efficiency of human brain doing 50 digits multiplication vs a multiplier circuit doing it?
- gamacodre 3y agoI think the problem here would be figuring out how much of the brain's power draw to attribute to the multiplication. A brain is more akin to a motherboard than a single CPU, with all kinds of I/O, internal regulation, and other ancillary stuff going on all the time.
- GaggiX 3y agoYeah I also think that the impact of better hardware generates a visible lead in quality in the models that are released, for example: companies like OpenAI have had access to large quantities of H100 for a few months now and Sora is being presented, something I would not have believed a year ago, I would also believe that Claude 3 models were trained on H100, DBRX was trained on 12T tokens, a big difference compared to the 300B for the original GPT-3, the new NovelAI image generation was trained on H100 and compared to the previous model is like day and night. It seems to create a generational jump.
- declaredapple 3y ago> companies like OpenAI have had access to large quantities of H100 for a few months now and Sora is being presented From what I could tell from Nvidia's recent presentation, Nvidia works directly with OpenAI to test their next gen hardware. IIRC they had some slides showing the throughput comparisons with Hopper and Blackwell, suggesting they used OpenAI's workload for testing. H100's have been generally available (not a long waitlist) for only several months, but all the big players had them already 1 year ago. I agree with you, but I think you might be 1 generation behind. > OpenAI used H100’s predecessor — NVIDIA A100 GPUs — to train and run ChatGPT, an AI system optimized for dialogue, which has been used by hundreds of millions of people worldwide in record time. OpenAI will be using H100 on its Azure supercomputer to power its continuing AI research. March 21, 2023 https://nvidianews.nvidia.com/news/nvidia-hopper-gpus-expand-reach-as-demand-for-ai-grows https://nvidianews.nvidia.com/news/nvidia-hopper-gpus-expand...
- GaggiX 3y agoVery interesting, I guess it does make sense that GPT-4 was also trained on the Hopper architecture.
- jameshart 3y agoFor some sense of how far out/close 1 trillion transistors in one GPU is: NVIDIA just announced Blackwell which gets to 208bn transistors on a chip by stitching two dies together into a single GPU. https://www.nvidia.com/en-us/data-center/technologies/blackwell-architecture/ https://www.nvidia.com/en-us/data-center/technologies/blackw... They’re sticking two of them in a board with a Grace CPU in between, then linking 36 of those those boards together in racks with NVLink switches that offer “130TB/s of GPU bandwidth in one 72-GPU NVLink domain”. In terms of marketing, NVidia calls one of those racks a GB200 NVL72 “super GPU”. So on one level NVIDIA would say they already have a GPU with ‘trillions’ of transistors.
- sylware 3y agoGPUs can still scale up by a LOT. The real limiting parameters are power consumption (which includes cooling), bus speed, production level of foundries. Think super-[cross-fire/sli]. Economics will probably forbid that. This is a virtual limit which factors in the previous physical limits... in theory.
- HarHarVeryFunny 3y agoWe already have a 4 trillion transistor "GPU" in the Cerebras WSE-3 (wafer-scale engine), used in Cerebras' data centers. https://www.youtube.com/watch?v=f4Dly8I8lMY https://www.youtube.com/watch?v=f4Dly8I8lMY
- DesiLurker 3y agohow many regular sized GPU (say 4080 or nvidia L4) dies can be cut out from a full-sized wafer? I suppose thats what OP means by reaching integration density of 1T GPU.
- denimnerd42 3y agohere's some discussion about yields for H100 per wafer. I'd assume a 4080 is smaller? regardless, the calculator is supplied. https://news.ycombinator.com/item?id=38588876 https://news.ycombinator.com/item?id=38588876
- moralestapia 3y agoI think the whole premise of TFA is flawed, as there already chips with way more than a trillion transistors (as GP points out). Arguing about what is the size limit to consider something a GPU or not is a bit like bikeshedding. As to why wouldn't a supercomputer be considered for this? Because it's not a single chip.
- IshKebab 3y agoIt's somewhere around 20 if you go as large as possible IIRC.
- ksec 3y agoAt close to reticle limit of around ~810mm2, you could fit around 65 chips.
- IshKebab 3y agoYeah that doesn't really count. It's the equivalent to like 20 GPUs and costs 200x as much.
- highfrequency 3y agoWild that the human brain can squeeze in 100 trillion synapses (very roughly analogous to model parameters / transistors) in a 3lb piece of meat that draws 20 Watts. The power efficiency difference may be explainable by the much slower frequency of brain computation (200 Hz vs. 2GHz). My impression is that the main obstacle to achieving a comparable volumetric density is that we haven't cracked 3d stacking of integrated circuits yet. Very exciting to see TSMC making inroads here: > Recent advances have shown HBM test structures with 12 layers of chips stacked using hybrid bonding, a copper-to-copper connection with a higher density than solder bumps can provide. Bonded at low temperature on top of a larger base logic chip, this memory system has a total thickness of just 600 µm...We’ll need to link all these chiplets together in a 3D stack, but fortunately, industry has been able to rapidly scale down the pitch of vertical interconnects, increasing the density of connections. And there is plenty of room for more. We see no reason why the interconnect density can’t grow by an order of magnitude, and even beyond. It's hard to imagine not getting unbelievable results when in 10-30 years we have GPUs with a comparable number of transistors to brain synapses that support computation speed 10,000x faster than the brain. What a thing to witness!
- londons_explore 3y ago> is it just that we haven't cracked 3-d stacking of integrated circuits yet? Yes. If we could stack transistors in the Z dimension as closely as we do in X and Y, we'd easily exceed the brains density.
- user90131313 3y agoI think biggest difference is millions of years evolutionary development. That's a lot of time difference.
- npalli 3y agoNo neurons are the equivalent of transistors. Synapses are the equivalent of connections between transistors. The total neurons are about 100 Billion but the connections are 100 Trillion. A neuron can have up to 10,000 synapses, while a transistor can have on average only about 3. Impressive nonetheless on power efficiency.
- deleted 3y ago[deleted]
- somewhereoutth 3y agoand how we'll find a way to soak up all those transistors for a perhaps actually worse user experience / societal outcome.
- phkahler 3y agoIf we are just after AI now we should drop the GPU concept and call it what it is - Matrix multipliers. With that framing, we can move to in-memory compute so the data doesn't have to be moving around so much. Memory chips could have lines of MAC units at some intervals and something to compute the nonlinear functions after summing. Fixed sizes could be implemented in hardware and software would spread larger computations over a number of tiles. If this were standardized we might see it end up as a modest premium price on memory chips. nVidia step aside it's going to be Micron, Hynix and friends revolutionizing AI.
- deleted 3y ago[deleted]
- kazinator 3y agoHardware, or at least purely computational hardware, will never get the same accolades as the software that actually makes it do something. Interface hardware, being is perceptible to the senses, gets credit over software. E.g. when people experience a vivid, sharp high resolution display, they attribute all its good properties to the hardware, even if there is some software involved in improving the visuals, like making fonts look better and whatnot. If a mouse works nicely, people attribute it to the hardware, not the drivers. If you work in hardware, and crave the appreciation, make something that people look at, hear, or hold in their hands, not something that crunches away in a closet.
- anonymousDan 3y agoI hear a lot about the energy efficiency of animal brains in comparison to e.g. GPUs. However, as far as I can tell most of the numbers reported are for adult brains, which effectively have been sparsified over time. Does anyone know of how the picture changes if we consider baby animal brains, which as I understand it have much denser connectivity and higher energy consumption than adult brains?