2 ms·

The performance of Gemini 1.5 Flash is the real takeaway. It's priced roughly similar to Claude Haiku, but in this leaderboard it's performing better than Claud

The performance of Gemini 1.5 Flash is the real takeaway. It's priced roughly similar to Claude Haiku, but in this leaderboard it's performing better than Claude Sonnet. I may have to run tests to see if Flash is really that much better than Haiku. From the win rate heat map (https://x.com/lmsysorg/status/1795512212456645074 https://x.com/lmsysorg/status/1795512212456645074) Flash has a 62% win rate vs. Haiku.

YetAnotherNick 2y ago
Better comparison would be llama 3 70b whose pricing is $0.59/$0.79[1]. Gemini Flash pricing is $0.35/$1.05. For similar quality and price I would select open model, if only for the certainty that the model would perform exactly the same and can't be tweaked/deleted without warning. [1]: https://wow.groq.com/ https://wow.groq.com/ [2]: https://ai.google.dev/pricing https://ai.google.dev/pricing
- deleted 2y ago
  [deleted]
- dilap 2y ago
  groq is super fast and cheap, but it seems to have (for normal plans) a very-low 30 requests/minute rate limit; to get a higher-limit, it's "contact us" enterprise pricing (which I haven't done, so I'm not sure what the results will be). edit: ah, currently it's "free beta access with low rate limits" and "enterprise" that is available, but they are planning to release "on demand pay per token" with higher rate limits soon; exact pricing unknown. https://pasteboard.co/OnLHjusiv6QT.png https://pasteboard.co/OnLHjusiv6QT.png another limitation of groq, which may or may not matter for your use case, is there's no way to pre-fill part of the llm response, which can be a very useful technique (there's also no support for the more general feature of "restrict sampling to a grammar", tho that is a much bigger ask).
- harlanlewis 2y ago
  I found myself making these types of comparisons over and over again to keep up with new models and pricing changes, so I made a Google Sheet for it with some charts to visualize outliers and trends in terms of model capability, throughput, and token cost. It's sorted by the Artificial Analysis Index [1], which incorporates both static benchmarks (MMLU) and dynamic crowdsourced elo (chatbot arena). Hope someone finds it useful: https://docs.google.com/spreadsheets/d/1foc98Jtbi0-GUsNySddvL0b2a7EuVQw8MoaQlWaDT-w/ https://docs.google.com/spreadsheets/d/1foc98Jtbi0-GUsNySddv... Specific to Groq - the service is an incredible outlier for speed and cost, but even for quite small projects I've run into rate limits and similar errors. Great option for something like a bring-your-own-keys personal chatbot, but I wouldn't build a service on it. Groq Llama 3 70B and Groq Mixtral 8x7B are in the sheet for comparison but omitted from charts. [1]: https://artificialanalysis.ai/leaderboards/providers https://artificialanalysis.ai/leaderboards/providers
  - nycdatasci 2y ago
    This is great - thanks!

YetAnotherNick 2y ago

Better comparison would be llama 3 70b whose pricing is $0.59/$0.79[1]. Gemini Flash pricing is $0.35/$1.05. For similar quality and price I would select open model, if only for the certainty that the model would perform exactly the same and can't be tweaked/deleted without warning. [1]: https://wow.groq.com/ https://wow.groq.com/ [2]: https://ai.google.dev/pricing https://ai.google.dev/pricing

deleted 2y ago
[deleted]
dilap 2y ago
groq is super fast and cheap, but it seems to have (for normal plans) a very-low 30 requests/minute rate limit; to get a higher-limit, it's "contact us" enterprise pricing (which I haven't done, so I'm not sure what the results will be). edit: ah, currently it's "free beta access with low rate limits" and "enterprise" that is available, but they are planning to release "on demand pay per token" with higher rate limits soon; exact pricing unknown. https://pasteboard.co/OnLHjusiv6QT.png https://pasteboard.co/OnLHjusiv6QT.png another limitation of groq, which may or may not matter for your use case, is there's no way to pre-fill part of the llm response, which can be a very useful technique (there's also no support for the more general feature of "restrict sampling to a grammar", tho that is a much bigger ask).
harlanlewis 2y ago
I found myself making these types of comparisons over and over again to keep up with new models and pricing changes, so I made a Google Sheet for it with some charts to visualize outliers and trends in terms of model capability, throughput, and token cost. It's sorted by the Artificial Analysis Index [1], which incorporates both static benchmarks (MMLU) and dynamic crowdsourced elo (chatbot arena). Hope someone finds it useful: https://docs.google.com/spreadsheets/d/1foc98Jtbi0-GUsNySddvL0b2a7EuVQw8MoaQlWaDT-w/ https://docs.google.com/spreadsheets/d/1foc98Jtbi0-GUsNySddv... Specific to Groq - the service is an incredible outlier for speed and cost, but even for quite small projects I've run into rate limits and similar errors. Great option for something like a bring-your-own-keys personal chatbot, but I wouldn't build a service on it. Groq Llama 3 70B and Groq Mixtral 8x7B are in the sheet for comparison but omitted from charts. [1]: https://artificialanalysis.ai/leaderboards/providers https://artificialanalysis.ai/leaderboards/providers
- nycdatasci 2y ago
  This is great - thanks!