5 ms·

It also means the largest models can be scaled up significantly with the same inference budget.

• llm_trw 3y ago
  Depends. The only paper they cite for training: https://arxiv.org/pdf/2310.11453.pdf https://arxiv.org/pdf/2310.11453.pdf doesn't improve training costs much and most models are already training constrained. Not everyone has $200m to throw at training another model from scratch.
  • arunk47 3y ago
    Is there any scope for indie builders?
    • llm_trw 3y ago
      Not really. These are slightly better for memory during pre-training and fine turning but not enough to make a 4090 usable even for a 7b model.