2 ms·
To be clear, this is a 7B model. It's just trained on 1.7 trillion tokens. At first I was confused why they were making such a big deal of of a massive 1.7T mod
by LorenDB 3y ago
To be clear, this is a 7B model. It's just trained on 1.7 trillion tokens. At first I was confused why they were making such a big deal of of a massive 1.7T model outperforming a 7B model.
By the way, GPT-4 has been predicted to be using a 1.7T model, although OpenAI has not confirmed this to my knowledge.
- mmoskal 3y agoThe most interesting bit is that this is RWKV model, meaning constant size state (no quadratic attention). AFAIK the biggest open weights non-transformer model.