3 ms·
> are we stuck waiting for the 20-25 years for GPU improvements If this turns out to be hard to optimize / improve then there will be a huge economic incentive
by bjornsing 2y ago
> are we stuck waiting for the 20-25 years for GPU improvements
If this turns out to be hard to optimize / improve then there will be a huge economic incentive for efficient ASICs. No freaking way we’ll be running on GPUs for 20-25 years, or even 2.
- coolspot 2y agoLLMs need efficient matrix multiiliers. GPUs are specialized ASICs for massive matrix multiplication.
- vlovich123 2y agoLLMs get to maybe ~20% of the rated max FLOPS for a GPU. It’s not hard to imagine that a purpose built ASIC with maybe adjusted software stack gets us significantly more real performance.
- boroboro4 2y agoThey get more than this. For prefill we can get 70% matmul utilization, for generation less than this but we’ll get to >50 too eventually.
- bjornsing 2y agoAnd even when you get to 100% utilization you’ll still be wasting a crazy amount of gates / die area, plus you’re paying the Nvidia tax. There is no way in hell that will go on for 10 years if we have good AGI but inference is too expensive.