3 ms·
> Frontier AI companies are selling at a loss. There are huge economies to be had by batching requests and using lots of RAM for MoE (sparse models). You can't
by visarga 5mo ago
> Frontier AI companies are selling at a loss.
There are huge economies to be had by batching requests and using lots of RAM for MoE (sparse models). You can't achieve that efficiency at batch size 1 on a single node.
- asjir 5mo agoExactly, they put a lot of money into engineering and it does give results
- eikenberry 5mo agoInference isn't the problem.