4 ms·
Trying the 8B on translation gives some hilariously garbled results. It’s a small model so perhaps not unexpected but it’s definitely nowhere close to GPT-3. I
by zarzavat 3y ago
Trying the 8B on translation gives some hilariously garbled results. It’s a small model so perhaps not unexpected but it’s definitely nowhere close to GPT-3.
I’d be cautious of synthetic benchmarks, you never know how much of the scores are due to contamination or survivorship bias.
- anon373839 3y agoLanguages other than English are “out of scope” according to the model card, so I wouldn’t expect strong translation performance. In English, though, it’s incredibly capable for its size.
- deleted 3y ago[deleted]
- d-z-m 3y agoI didn't think arena was a synthetic benchmark?