3 ms·

Thoughts About Scaling Law

• kzrdude 2mo ago
  Mirror: https://xcancel.com/jietang/status/2089941544581403107 https://xcancel.com/jietang/status/2089941544581403107
• jarbus 2mo ago
  Maybe just me, but I have a feeling that we are currently stuck at 100-200k effective context window due to a parameter limitation, specifically the hidden dimension of the model, which would be an incredibly expensive dimension to increase vs adding optimizations elsewhere. I don’t think we are done scaling parameters yet.
  • kzrdude 2mo ago
    Why is it connected to number of parameters?