5 ms·
With improvements on the algorithm side and new techniques, even older hardware will become useful.
by cantor_S_drug 1y ago
With improvements on the algorithm side and new techniques, even older hardware will become useful.
- Zigurd 1y agoI get what you're saying and the reasoning behind it, but older hardware has never been useful where power consumption is part of determining usefulness.
- chatmasta 1y agoThis is the biggest threat to the GPU economy – software breakthroughs that enable inference on commodity CPU hardware or specialized ASIC boards that hyperscalers can fabricate themselves. Google has a stockpile of TPUs that seem fairly effective, although it’s hard to tell for certain because they don’t make it easy to rent them.
- xadhominemx 1y agoMore efficient inference = more reasoning token. Hyperscaler ASICs are closing the gap at the hardware/system level, yes.
- Zigurd 1y agoI don't think we will need to wait for anything as unpredictable as a breakthrough. Optimizing inference for the most clearly defined tasks, which are also the tasks where value is most readily quantified, like coding, is underway now.