4 ms·

Mistral Medium beats 4.5 on the censorship benchmark. It doesn't refuse to help with anything that could be vaguely non-PC or could potentially be used to hurt

Mistral Medium beats 4.5 on the censorship benchmark. It doesn't refuse to help with anything that could be vaguely non-PC or could potentially be used to hurt anyone in the wrong hands, including dangerously hot salsa recipes.

wokwokwok 3y ago
That's not a metric. That's a use case. Certainly, no one here is arguing that there are things openai refuses to allow, and given that the effectiveness of using GPT4 on them is literally zero, a sweet potato connected to a spring and keyboard will "beat" GPT-4, if that's your scoring metric. If you want a meaningful comparison you need tasks that both tools are capable of doing, and then see how effective they are. Claiming that mistral medium beats it is like me claiming the RenderMan beats DALLE2 at rendering 3d models; yes, technically they both generate images, but since it's not possible to use DALLE2 to render a 3d model, it's not really a meaningful comparison is it?
- theshackleford 3y ago
  > If you want a meaningful comparison you need tasks that both tools are capable of doing, and then see how effective they are. The fact it’s incapable of simple requests that an alternative can is absolutely part of a worthwhile comparison.
  - wokwokwok 3y ago
    You’re just twisting what “best” means to suit your bias. That is not a measure of how sophisticated and capable a model is. GPT4 is a more sophisticated, more capable mode than mistral. If that doesn’t make it the “better” for you, that’s fine; but any attempt to argue about the capabilities of the models is misguided. Restrictions placed on a model are an orthogonal concern to its capabilities. …but sure, you can invent some benchmarks to score models on other criteria, which is entirely valid. It’s perfectly fair to say that GPT4 doesn’t top all possible metrics… only the meaningful ones about model capabilities.
- bambax 3y ago
  Semantics. Both tools are generative systems that produce text in response to a prompt. If Mistral was mute on random topics for no other reason that its makers dislike talking about that, would you say it doesn't count?
benreesman 3y ago
I'm a big proponent of freedom in this space (and remain one), but Dolphin is fucking scary. I don't have any use cases for crime in my life at the moment beyond wanting to pirate like Adobe Illustrator before signing up for an uncancelable subscription, but it will do arbitrary things within it's abilities and it's google with a grudge in terms of how to do anything you ask. I stopped wanting to know when it convinced me it could explain how to stage a coup d'etat. I'm back on mixtral-8x7b.

wokwokwok 3y ago

That's not a metric. That's a use case. Certainly, no one here is arguing that there are things openai refuses to allow, and given that the effectiveness of using GPT4 on them is literally zero, a sweet potato connected to a spring and keyboard will "beat" GPT-4, if that's your scoring metric. If you want a meaningful comparison you need tasks that both tools are capable of doing, and then see how effective they are. Claiming that mistral medium beats it is like me claiming the RenderMan beats DALLE2 at rendering 3d models; yes, technically they both generate images, but since it's not possible to use DALLE2 to render a 3d model, it's not really a meaningful comparison is it?

theshackleford 3y ago
> If you want a meaningful comparison you need tasks that both tools are capable of doing, and then see how effective they are. The fact it’s incapable of simple requests that an alternative can is absolutely part of a worthwhile comparison.
- wokwokwok 3y ago
  You’re just twisting what “best” means to suit your bias. That is not a measure of how sophisticated and capable a model is. GPT4 is a more sophisticated, more capable mode than mistral. If that doesn’t make it the “better” for you, that’s fine; but any attempt to argue about the capabilities of the models is misguided. Restrictions placed on a model are an orthogonal concern to its capabilities. …but sure, you can invent some benchmarks to score models on other criteria, which is entirely valid. It’s perfectly fair to say that GPT4 doesn’t top all possible metrics… only the meaningful ones about model capabilities.
bambax 3y ago
Semantics. Both tools are generative systems that produce text in response to a prompt. If Mistral was mute on random topics for no other reason that its makers dislike talking about that, would you say it doesn't count?

benreesman 3y ago

I'm a big proponent of freedom in this space (and remain one), but Dolphin is fucking scary. I don't have any use cases for crime in my life at the moment beyond wanting to pirate like Adobe Illustrator before signing up for an uncancelable subscription, but it will do arbitrary things within it's abilities and it's google with a grudge in terms of how to do anything you ask. I stopped wanting to know when it convinced me it could explain how to stage a coup d'etat. I'm back on mixtral-8x7b.