3 ms·
Starling-7B, OpenChat, OpenHermes in that table are Mistral-7B finetunes, and are all above the current GPT-3.5-Turbo (1106). Note how these tiny 7B models are
by orbital-decay 3y ago
Starling-7B, OpenChat, OpenHermes in that table are Mistral-7B finetunes, and are all above the current GPT-3.5-Turbo (1106). Note how these tiny 7B models are surrounded by much larger ones.
Mixtral 8x7B is not in there yet.
- pama 3y agoELO takes a while to establish. It does not sound likely that the newer GPT3.5 is that much worse than the old one that has a clear gap to all the non proprietary models. In the immediate test, GPT-3.5 clearly outshines these models.
- orbital-decay 3y ago> ELO takes a while to establish. Well, Starling-7B was published two weeks ago; GPT-3.5-turbo-0613 is more than a month old snapshot, which should probably be enough time. OpenChat and OpenHermes are about a month old as well. >It does not sound likely that the newer GPT3.5 is that much worse than the old one In fact, this version received complaints almost immediately. https://community.openai.com/t/496732 https://community.openai.com/t/496732 >In the immediate test, GPT-3.5 clearly outshines these models. It might be so, but it's not clear to me at all. I tested Starling for a bit and was really surprised that it's a 7B model, not a 70B+ one or GPT-3.5.
- whimsicalism 3y agoI disagree - lmsys score for new chatgpt has been relatively constant, and OAI is probably trying to distill the model even further.
- sorokod 3y agoNot my intent to argue about data at any point in time but note that as of today gpt-3.5-turbo-0613 (June 13th 2023) scores 1112, above OpenChat (1075) and OpenHermes(1077).
- jbgt 3y agoThat's not much relative difference. How much does 1% difference make? I am tempted to call it equivalent.
- whimsicalism 3y agonah that is sizeable
- doctorpangloss 3y agoI guess one thing people have learned is that these small differences whatever benchmark turn out to be huge differences qualitatively.