3 ms·
I've mentioned this before on HN [0][1]. The name of the game has been custom SoCs and ASICs for a couple years now, because inference and model training is an
by alephnerd 1y ago
I've mentioned this before on HN [0][1].
The name of the game has been custom SoCs and ASICs for a couple years now, because inference and model training is an "embarrassingly parallel" problem, and models that are optimized for older hardware can provide similar gains to models that are run on unoptimized but more performant hardware.
Same reason H100s remain a mainstay in the industry today, as their performance profile is well understood now.
[0] - https://news.ycombinator.com/item?id=45275413 https://news.ycombinator.com/item?id=45275413
[1] - https://news.ycombinator.com/item?id=43383418 https://news.ycombinator.com/item?id=43383418
- philipwhiuk 1y ago> The name of the game has been custom SoCs and ASICs for a couple years now, because inference and model training is an "embarrassingly parallel" problem, and models that are optimized for older hardware can provide similar gains to models that are run on unoptimized but more performant hardware. Is anyone else getting crypto flashbacks?
- kcb 1y agoThe difference is crypto wasn't memory and throughput dependent. That's why a small asic on a USB stick could outperform a GPU.
- alephnerd 1y agoOne thing to point out - "ASIC" is more of a business term than a technical term. The teams that work on custom ASIC design at (eg.) Broadcom for Microsoft are basically designing custom GPUs for MS, but these will only meet the requirements that Microsoft lays out, and Microsoft would have full insight and visibility into the entire architecture.