3 ms·
How is this model performing better than LLaMa in a lot of tasks[1] even though its trained on a fifth of the data (1 trillion vs 200 billion). [1]https://gith
by newswasboring 3y ago
How is this model performing better than LLaMa in a lot of tasks[1] even though its trained on a fifth of the data (1 trillion vs 200 billion).
[1]https://github.com/openlm-research/open_llama#evaluation https://github.com/openlm-research/open_llama#evaluation
- slekker 3y agoNobody knows :^)
- tarruda 3y agoMaybe it uses a higher quality dataset
- YetAnotherNick 3y agoThey are likely doing some interpolation for 200B or benchmarking it in wrong way. e.g. Hellaswag accuracy for llama 7b is 0.76[1], but it is written 0.56 in the repo. Even at 200B tokens, it is higher than 0.56 for llama looking at the charts. [1]: https://arxiv.org/pdf/2302.13971.pdf https://arxiv.org/pdf/2302.13971.pdf
- byefruit 3y agoThey ran lm-evaluation-harness on both this model and the original llama weights, which is the correct way to do it. Many people have been struggling to reproduce the benchmark numbers included in the original llama paper.