3 ms·

This shortage is the best thing that could have happened for R&D in model efficiency. The "who has more parameters" race is about to hit the physical wall of ha

• kingstnap 1y ago
  This is a silly take. From day zero everyone and their mother was trying every idea under the sun on trying to get inference costs down. There are a huge number of varients of attention schemes, like seemingly thousands. More posted on arxiv and blogs every single day. There are similarly a huge number of papers and talks on quantization, codebooks, number formats, everything. Like genuinely covering everything from analog compute to training lookup tables in FPGAs. The AI model architects similarly are not sleeping on things. They genuinely take a bottom up approach to the design of their model and make sure that every mm of die area is being used. And top down of course has extreme influence with people spinning up entire hardware companies like groq and cerebras to do AI as fast and efficiently as possible. Everyone wants to be the shovel seller. The idea that anyone is sleeping on increasing inference efficiency when $$$$$$$$$ are being spent is ridiculous. And genuinely they have actually made massive strides. Some models cost almost nothing like GPT OSS 120B which can be used to create slop at the speed of light.