8 ms·
> Phi-2: trained with 1400B tokens. Where did you get the information. Not doubting you, just I couldn't find it.
by YetAnotherNick 3y ago
> Phi-2: trained with 1400B tokens.
Where did you get the information. Not doubting you, just I couldn't find it.
- GolDDranks 3y agoIt says so on the website: "Phi-2 is a Transformer-based model with a next-word prediction objective, trained on 1.4T tokens from multiple passes on a mixture of Synthetic and Web datasets for NLP and coding."
- YetAnotherNick 3y agoPhi 1 was trained on 7B tokens. This is quite a jump then.
- nojvek 3y agoI wish they shared their 1.4T dataset. I love that the RedPajama dataset is openly shared on HuggingFace.