4 ms·
There is an attempt to quantify subjective evaluation of models here[1] - the "Arena Elo rating". According to popular vote, Mistral chat is nowhere near GPT 3.
by sorokod 3y ago
There is an attempt to quantify subjective evaluation of models here[1] - the "Arena Elo rating". According to popular vote, Mistral chat is nowhere near GPT 3.5
[1] https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboard https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboar...
- sva_ 3y agoDoesn't seem like that's the mixture of experts model in the list? Or am I blind
- fastball 3y agoIndeed, that is a previous Mistral model.
- orbital-decay 3y agoStarling-7B, OpenChat, OpenHermes in that table are Mistral-7B finetunes, and are all above the current GPT-3.5-Turbo (1106). Note how these tiny 7B models are surrounded by much larger ones. Mixtral 8x7B is not in there yet.
- pama 3y agoELO takes a while to establish. It does not sound likely that the newer GPT3.5 is that much worse than the old one that has a clear gap to all the non proprietary models. In the immediate test, GPT-3.5 clearly outshines these models.
- orbital-decay 3y ago> ELO takes a while to establish. Well, Starling-7B was published two weeks ago; GPT-3.5-turbo-0613 is more than a month old snapshot, which should probably be enough time. OpenChat and OpenHermes are about a month old as well. >It does not sound likely that the newer GPT3.5 is that much worse than the old one In fact, this version received complaints almost immediately. https://community.openai.com/t/496732 https://community.openai.com/t/496732 >In the immediate test, GPT-3.5 clearly outshines these models. It might be so, but it's not clear to me at all. I tested Starling for a bit and was really surprised that it's a 7B model, not a 70B+ one or GPT-3.5.
- whimsicalism 3y agoI disagree - lmsys score for new chatgpt has been relatively constant, and OAI is probably trying to distill the model even further.
- sorokod 3y agoNot my intent to argue about data at any point in time but note that as of today gpt-3.5-turbo-0613 (June 13th 2023) scores 1112, above OpenChat (1075) and OpenHermes(1077).
- jbgt 3y agoThat's not much relative difference. How much does 1% difference make? I am tempted to call it equivalent.
- whimsicalism 3y agonah that is sizeable
- doctorpangloss 3y agoI guess one thing people have learned is that these small differences whatever benchmark turn out to be huge differences qualitatively.