3 ms·

Their main advantage for now is their super clean API. Open source alternative are already on par with GPT-3.5 and 4 capabilities, they just don't have as good

Their main advantage for now is their super clean API. Open source alternative are already on par with GPT-3.5 and 4 capabilities, they just don't have as good a package but that could change rather quickly too.

fakedang 3y ago
Mistral's API was designed to be practically interchangeable with the OpenAI API.
generalizations 3y ago
What open source alternative is on par with GPT4?
tombert 3y ago
Is that true? I was running Llamas on my laptop a few days ago, and it was giving measurably worse results than ChatGPT. I think it was the uncensored 13B model, but if you got something that's on par with ChatGPT that I can run on my own hardware I'm pretty interested.
- bhouston 3y ago
  13B models probably cannot directly compare with ChatGPT 4 which maybe +1T parameters or a 5 way MoE of 200B each - or something like that. So you can not likely run a model competitive with ChatGPT locally in the near term.
  - tombert 3y ago
    I have a server with a bunch of PCIe slots and like 4 Nvidia GPUs with 24GB of RAM each. What's the best model I can realistically run?
    - bhouston 3y ago
      Here are some scorecards: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb... https://paperswithcode.com/sota/sentence-completion-on-hellaswag https://paperswithcode.com/sota/sentence-completion-on-hella... https://paperswithcode.com/sota/common-sense-reasoning-on-winogrande https://paperswithcode.com/sota/common-sense-reasoning-on-wi... https://paperswithcode.com/sota/common-sense-reasoning-on-arc-challenge https://paperswithcode.com/sota/common-sense-reasoning-on-ar... https://paperswithcode.com/sota/common-sense-reasoning-on-commonsenseqa https://paperswithcode.com/sota/common-sense-reasoning-on-co...
koito17 3y ago
> Open source alternative are already on par with GPT-3.5 and 4 capabilities I'm not sure if this is true. With GPT-4, I can successfully ask questions in Japanese and receive responses in (mostly natural) Japanese. I have also found GPT-4 capable of understanding the semantics of prompts with Japanese and English phrases interleaved. Out of curiosity, I tried doing the same with local models like Mistral 7B and I could never get the model to emit anything other than English. Maybe it's a difference in training data, but even then, GPT-4 has an allegedly small set of training data for non-European languages.

fakedang 3y ago

Mistral's API was designed to be practically interchangeable with the OpenAI API.

generalizations 3y ago

What open source alternative is on par with GPT4?

tombert 3y ago

Is that true? I was running Llamas on my laptop a few days ago, and it was giving measurably worse results than ChatGPT. I think it was the uncensored 13B model, but if you got something that's on par with ChatGPT that I can run on my own hardware I'm pretty interested.

bhouston 3y ago
13B models probably cannot directly compare with ChatGPT 4 which maybe +1T parameters or a 5 way MoE of 200B each - or something like that. So you can not likely run a model competitive with ChatGPT locally in the near term.
- tombert 3y ago
  I have a server with a bunch of PCIe slots and like 4 Nvidia GPUs with 24GB of RAM each. What's the best model I can realistically run?
  - bhouston 3y ago
    Here are some scorecards: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb... https://paperswithcode.com/sota/sentence-completion-on-hellaswag https://paperswithcode.com/sota/sentence-completion-on-hella... https://paperswithcode.com/sota/common-sense-reasoning-on-winogrande https://paperswithcode.com/sota/common-sense-reasoning-on-wi... https://paperswithcode.com/sota/common-sense-reasoning-on-arc-challenge https://paperswithcode.com/sota/common-sense-reasoning-on-ar... https://paperswithcode.com/sota/common-sense-reasoning-on-commonsenseqa https://paperswithcode.com/sota/common-sense-reasoning-on-co...

koito17 3y ago

> Open source alternative are already on par with GPT-3.5 and 4 capabilities I'm not sure if this is true. With GPT-4, I can successfully ask questions in Japanese and receive responses in (mostly natural) Japanese. I have also found GPT-4 capable of understanding the semantics of prompts with Japanese and English phrases interleaved. Out of curiosity, I tried doing the same with local models like Mistral 7B and I could never get the model to emit anything other than English. Maybe it's a difference in training data, but even then, GPT-4 has an allegedly small set of training data for non-European languages.