4 ms·
It says so on the website: "Phi-2 is a Transformer-based model with a next-word prediction objective, trained on 1.4T tokens from multiple passes on a mixture o
by GolDDranks 3y ago
It says so on the website: "Phi-2 is a Transformer-based model with a next-word prediction objective, trained on 1.4T tokens from multiple passes on a mixture of Synthetic and Web datasets for NLP and coding."
- YetAnotherNick 3y agoPhi 1 was trained on 7B tokens. This is quite a jump then.
- nojvek 3y agoI wish they shared their 1.4T dataset. I love that the RedPajama dataset is openly shared on HuggingFace.