4 ms·

They can absolutely outperform gpt4 for specific use cases.

by achille 3y ago

nickthegreek 3y ago
I am very open to believing that. I'd love to see some examples.
- GaggiX 3y ago
  Well it's pretty easy to find examples online, this one using Llama 2, not even Mistral or fancy techniques: https://www.anyscale.com/blog/fine-tuning-llama-2-a-comprehensive-case-study-for-tailoring-models-to-unique-applications https://www.anyscale.com/blog/fine-tuning-llama-2-a-comprehe...
- turnsout 3y ago
  I agree, I think they need an example or two on that blog post to back up the claim. I'm ready to believe it, but I need something more than "diverse customer tasks" to understand what we're talking about.
- shiftpgdn 3y ago
  They're quite close in arena format: https://chat.lmsys.org/?arena https://chat.lmsys.org/?arena
  - TOMDM 3y ago
    To be clear, Mixtral is very competitive, Mistral while certainly way better than most 7B models performs far worse than ChatGPT3.5 Turbo.
    - shiftpgdn 3y ago
      Apologies, that's what I get for skimming through the thread.
- bugglebeetle 3y ago
  You can fine-tune a small model yourself and see. GPT-4 is an amazing general model, but won’t perform the best at every task you throw at it, out of the box. I have a fine-tuned Mistral 7B model that outperforms GPT 4 on a specific type of structured data extraction. Maybe if I fine-tuned GPT-4 it could beat it, but that costs a lot of money for what I can now do locally for the cost of electricity.
TOMDM 3y ago
Yeah, a 7B foundation model is of course going to be worse when expected to perform on every task. But finetuning on just a few tasks? Depending on the task, it's totally reasonable to expect that a 7B model might eke out a win against stock GPT4. Especially if there's domain knowledge in the finetune, and the given task is light on demand for logical skills.
- deleted 3y ago
  [deleted]
holoduke 3y ago
Not for translations. Did a lot of experimenting different local models. None come even a bit close to the capabilities of chatgpt. Most local models just outputting plain wrong intormation. I am still hoping one day it will be possible. For our business a huge opportunity.
- ijk 3y ago
  For translation, you're probably better off with a model that's specifically designed for translation, like MADLAD-400 or DeepL's services.

They can absolutely outperform gpt4 for specific use cases.

nickthegreek 3y ago

I am very open to believing that. I'd love to see some examples.

GaggiX 3y ago
Well it's pretty easy to find examples online, this one using Llama 2, not even Mistral or fancy techniques: https://www.anyscale.com/blog/fine-tuning-llama-2-a-comprehensive-case-study-for-tailoring-models-to-unique-applications https://www.anyscale.com/blog/fine-tuning-llama-2-a-comprehe...
turnsout 3y ago
I agree, I think they need an example or two on that blog post to back up the claim. I'm ready to believe it, but I need something more than "diverse customer tasks" to understand what we're talking about.
shiftpgdn 3y ago
They're quite close in arena format: https://chat.lmsys.org/?arena https://chat.lmsys.org/?arena
- TOMDM 3y ago
  To be clear, Mixtral is very competitive, Mistral while certainly way better than most 7B models performs far worse than ChatGPT3.5 Turbo.
  - shiftpgdn 3y ago
    Apologies, that's what I get for skimming through the thread.
bugglebeetle 3y ago
You can fine-tune a small model yourself and see. GPT-4 is an amazing general model, but won’t perform the best at every task you throw at it, out of the box. I have a fine-tuned Mistral 7B model that outperforms GPT 4 on a specific type of structured data extraction. Maybe if I fine-tuned GPT-4 it could beat it, but that costs a lot of money for what I can now do locally for the cost of electricity.

TOMDM 3y ago

Yeah, a 7B foundation model is of course going to be worse when expected to perform on every task. But finetuning on just a few tasks? Depending on the task, it's totally reasonable to expect that a 7B model might eke out a win against stock GPT4. Especially if there's domain knowledge in the finetune, and the given task is light on demand for logical skills.

deleted 3y ago
[deleted]

holoduke 3y ago

Not for translations. Did a lot of experimenting different local models. None come even a bit close to the capabilities of chatgpt. Most local models just outputting plain wrong intormation. I am still hoping one day it will be possible. For our business a huge opportunity.

ijk 3y ago
For translation, you're probably better off with a model that's specifically designed for translation, like MADLAD-400 or DeepL's services.