3 ms·
not very good compared to what? Hard to reconcile your comment with its outsize performance on arena/glowing praise from others comparing it to much larger mode
by d-z-m 2y ago
not very good compared to what? Hard to reconcile your comment with its outsize performance on arena/glowing praise from others comparing it to much larger models.
- zarzavat 2y agoTrying the 8B on translation gives some hilariously garbled results. It’s a small model so perhaps not unexpected but it’s definitely nowhere close to GPT-3. I’d be cautious of synthetic benchmarks, you never know how much of the scores are due to contamination or survivorship bias.
- anon373839 2y agoLanguages other than English are “out of scope” according to the model card, so I wouldn’t expect strong translation performance. In English, though, it’s incredibly capable for its size.
- deleted 2y ago[deleted]
- d-z-m 2y agoI didn't think arena was a synthetic benchmark?