4 ms·

Better comparison would be llama 3 70b whose pricing is $0.59/$0.79[1]. Gemini Flash pricing is $0.35/$1.05. For similar quality and price I would select open m

Better comparison would be llama 3 70b whose pricing is $0.59/$0.79[1]. Gemini Flash pricing is $0.35/$1.05. For similar quality and price I would select open model, if only for the certainty that the model would perform exactly the same and can't be tweaked/deleted without warning. [1]: https://wow.groq.com/ https://wow.groq.com/ [2]: https://ai.google.dev/pricing https://ai.google.dev/pricing

deleted 2y ago
[deleted]
dilap 2y ago
groq is super fast and cheap, but it seems to have (for normal plans) a very-low 30 requests/minute rate limit; to get a higher-limit, it's "contact us" enterprise pricing (which I haven't done, so I'm not sure what the results will be). edit: ah, currently it's "free beta access with low rate limits" and "enterprise" that is available, but they are planning to release "on demand pay per token" with higher rate limits soon; exact pricing unknown. https://pasteboard.co/OnLHjusiv6QT.png https://pasteboard.co/OnLHjusiv6QT.png another limitation of groq, which may or may not matter for your use case, is there's no way to pre-fill part of the llm response, which can be a very useful technique (there's also no support for the more general feature of "restrict sampling to a grammar", tho that is a much bigger ask).
harlanlewis 2y ago
I found myself making these types of comparisons over and over again to keep up with new models and pricing changes, so I made a Google Sheet for it with some charts to visualize outliers and trends in terms of model capability, throughput, and token cost. It's sorted by the Artificial Analysis Index [1], which incorporates both static benchmarks (MMLU) and dynamic crowdsourced elo (chatbot arena). Hope someone finds it useful: https://docs.google.com/spreadsheets/d/1foc98Jtbi0-GUsNySddvL0b2a7EuVQw8MoaQlWaDT-w/ https://docs.google.com/spreadsheets/d/1foc98Jtbi0-GUsNySddv... Specific to Groq - the service is an incredible outlier for speed and cost, but even for quite small projects I've run into rate limits and similar errors. Great option for something like a bring-your-own-keys personal chatbot, but I wouldn't build a service on it. Groq Llama 3 70B and Groq Mixtral 8x7B are in the sheet for comparison but omitted from charts. [1]: https://artificialanalysis.ai/leaderboards/providers https://artificialanalysis.ai/leaderboards/providers
- nycdatasci 2y ago
  This is great - thanks!

deleted 2y ago

[deleted]

dilap 2y ago

groq is super fast and cheap, but it seems to have (for normal plans) a very-low 30 requests/minute rate limit; to get a higher-limit, it's "contact us" enterprise pricing (which I haven't done, so I'm not sure what the results will be). edit: ah, currently it's "free beta access with low rate limits" and "enterprise" that is available, but they are planning to release "on demand pay per token" with higher rate limits soon; exact pricing unknown. https://pasteboard.co/OnLHjusiv6QT.png https://pasteboard.co/OnLHjusiv6QT.png another limitation of groq, which may or may not matter for your use case, is there's no way to pre-fill part of the llm response, which can be a very useful technique (there's also no support for the more general feature of "restrict sampling to a grammar", tho that is a much bigger ask).

harlanlewis 2y ago

I found myself making these types of comparisons over and over again to keep up with new models and pricing changes, so I made a Google Sheet for it with some charts to visualize outliers and trends in terms of model capability, throughput, and token cost. It's sorted by the Artificial Analysis Index [1], which incorporates both static benchmarks (MMLU) and dynamic crowdsourced elo (chatbot arena). Hope someone finds it useful: https://docs.google.com/spreadsheets/d/1foc98Jtbi0-GUsNySddvL0b2a7EuVQw8MoaQlWaDT-w/ https://docs.google.com/spreadsheets/d/1foc98Jtbi0-GUsNySddv... Specific to Groq - the service is an incredible outlier for speed and cost, but even for quite small projects I've run into rate limits and similar errors. Great option for something like a bring-your-own-keys personal chatbot, but I wouldn't build a service on it. Groq Llama 3 70B and Groq Mixtral 8x7B are in the sheet for comparison but omitted from charts. [1]: https://artificialanalysis.ai/leaderboards/providers https://artificialanalysis.ai/leaderboards/providers

nycdatasci 2y ago
This is great - thanks!