3 ms·
This is where we always assumed the industry was going. Expensive GPUs are great for training, but inference is getting so optimized that it will run on smalle
by jedberg 2y ago
This is where we always assumed the industry was going. Expensive GPUs are great for training, but inference is getting so optimized that it will run on smaller and/or cheaper processors (per token).
- striking 2y ago"wafer scale" inference using 44GB of straight up SRAM per chip does not exactly sound "smaller and cheaper" to me. Just optimized.
- jedberg 2y agoCheaper on a per token basis.
- sanxiyn 2y agoDoubtful. SRAM is not cheap, and this is entirely about SRAM vs HBM.
- jedberg 2y agoThey list the price in this press release. So either they're taking a big loss or they're doing it cheaper per token.
- throwup238 2y agoIt wouldn't be the first time a manufacturer ignored capital amortization to post better numbers.
- sanxiyn 2y agoNot even necessarily a loss let alone big, since their comparison includes margin.
- striking 2y agoI accept that. Your original comment left me under the impression that this represented a shift closer to the edge (I still don't think the hardware is all that much smaller), but I'll agree this is cheaper per token under full utilization.
- itishappy 2y agoCerebras' Wafer Scale Engine is the opposite of small and cheap. https://cerebras.ai/product-chip/ https://cerebras.ai/product-chip/
- jedberg 2y agoI updated my comment to be clearer. I meant smaller and/or cheaper per token. In this case it's cheaper per token.
- throwawaymaths 2y agoThat's totally nuts. How do they deal with the silicon warping around disabled cores or dark silicon? How long of hard running does it take before the chip gets fatally damaged and needs to be replaced in their system? Word on the street is that h100s fail surprisingly often, this can't be better
- bigcat12345678 2y agohttps://www.youtube.com/watch?v=7GV_OdqzmIU&t=1104s https://www.youtube.com/watch?v=7GV_OdqzmIU&t=1104s This video from Cerebras perfectly explain how they solve the interconnect problem, and why their approach greatly reduces the risk of Blackwell-type hardware design challenges.
- prng2021 2y agoSuper informative video!
- TradingPlaces 2y agoThey have a way of bypassing bad cores, and over-provision both logic and memory by 1.5% to account for that. They get 100% yields this way. https://www.anandtech.com/show/16626/cerebras-unveils-wafer-scale-engine-two-wse2-26-trillion-transistors-100-yield https://www.anandtech.com/show/16626/cerebras-unveils-wafer-...
- 2y ago
- deleted 2y ago[deleted]