2 ms·

Anyone see benchmarks comparing this to Gemini Flash or gpt-3.5 or open models that can run on groq?

by ilaksh 2y ago

xendipity 2y ago
I'm not finding any direct sources from OpenAI, but here's this snippet from a Reuters article [1] > Priced at 15 cents per million input tokens and 60 cents per million output tokens, the GPT-4o mini is more than 60% cheaper than GPT-3.5 Turbo, OpenAI said. It currently outperforms the GPT-4 model on chat preferences and scored 82% on Massive Multitask Language Understanding (MMLU), OpenAI said. ... > The GPT-4o mini model's score compared with 77.9% for Google's Gemini Flash and 73.8% for Anthropic's Claude Haiku, according to OpenAI. For some more context: We don't know the size of 4o-mini but Mistral's just released NeMo 12B scores 68% on the MMLU. [2] [1]: https://www.reuters.com/technology/artificial-intelligence/openai-unveils-cheaper-small-ai-model-gpt-4o-mini-2024-07-18/ https://www.reuters.com/technology/artificial-intelligence/o... [2]: https://mistral.ai/news/mistral-nemo/ https://mistral.ai/news/mistral-nemo/
- pzo 2y ago
  Also for some reference: Gemma 2 27B scored: 75.2 in MMLU LLama 3 70B scored: 79.5 in MMLU Haiku scored: 75.2 in MMLU GPT 3.5 scored: 70.0 in MMLU Based on pricing I see in openrouter.ai across different providers this seems like the cheapest model for this kind of performance. ref: [0] https://www-cdn.anthropic.com/de8ba9b01c9ab7cbabf5c33b80b7bbc618857627/Model_Card_Claude_3.pdf https://www-cdn.anthropic.com/de8ba9b01c9ab7cbabf5c33b80b7bb... [1] https://blog.google/technology/developers/google-gemma-2/ https://blog.google/technology/developers/google-gemma-2/
unraveller 2y ago
https://artificialanalysis.ai https://artificialanalysis.ai (groq) Quality to Price graph suggests gpt3.5 was the worst now 4o-mini inched out all others of that lower league. It supposedly gets you flash/llama3-70B tier quality at around llama3-8B price.

Anyone see benchmarks comparing this to Gemini Flash or gpt-3.5 or open models that can run on groq?

by ilaksh 2y ago

Anyone see benchmarks comparing this to Gemini Flash or gpt-3.5 or open models that can run on groq?

I'm not finding any direct sources from OpenAI, but here's this snippet from a Reuters article [1] > Priced at 15 cents per million input tokens and 60 cents per million output tokens, the GPT-4o mini is more than 60% cheaper than GPT-3.5 Turbo, OpenAI said. It currently outperforms the GPT-4 model on chat preferences and scored 82% on Massive Multitask Language Understanding (MMLU), OpenAI said. ... > The GPT-4o mini model's score compared with 77.9% for Google's Gemini Flash and 73.8% for Anthropic's Claude Haiku, according to OpenAI. For some more context: We don't know the size of 4o-mini but Mistral's just released NeMo 12B scores 68% on the MMLU. [2] [1]: https://www.reuters.com/technology/artificial-intelligence/openai-unveils-cheaper-small-ai-model-gpt-4o-mini-2024-07-18/ https://www.reuters.com/technology/artificial-intelligence/o... [2]: https://mistral.ai/news/mistral-nemo/ https://mistral.ai/news/mistral-nemo/

pzo 2y ago
Also for some reference: Gemma 2 27B scored: 75.2 in MMLU LLama 3 70B scored: 79.5 in MMLU Haiku scored: 75.2 in MMLU GPT 3.5 scored: 70.0 in MMLU Based on pricing I see in openrouter.ai across different providers this seems like the cheapest model for this kind of performance. ref: [0] https://www-cdn.anthropic.com/de8ba9b01c9ab7cbabf5c33b80b7bbc618857627/Model_Card_Claude_3.pdf https://www-cdn.anthropic.com/de8ba9b01c9ab7cbabf5c33b80b7bb... [1] https://blog.google/technology/developers/google-gemma-2/ https://blog.google/technology/developers/google-gemma-2/

unraveller 2y ago

https://artificialanalysis.ai https://artificialanalysis.ai (groq) Quality to Price graph suggests gpt3.5 was the worst now 4o-mini inched out all others of that lower league. It supposedly gets you flash/llama3-70B tier quality at around llama3-8B price.