3 ms·
I feel "The Pile" needs a lot more to match what OpenAI has trained their models on. The mysterious "Books2" and so on.
by msoad 3y ago
I feel "The Pile" needs a lot more to match what OpenAI has trained their models on. The mysterious "Books2" and so on.
- harisec 3y agoI think OpenAI has a few hidden advantages that are not obvious at this point. It could be additional training data, filtering/preprocessing that data, some changes to the architecture, who knows? None of the open source models are even close to GPT 3.5, what to speak about GPT 4? I've tried everything and the 60G llama variants so i'm not sure it's about number of parameters. They definitely have some hidden sauce.