4 ms·
as someone building a product with their API, GPT-4 is already smarter than we need for our use case. Request latency is my #1 concern right now.
by mritchie712 2y ago
as someone building a product with their API, GPT-4 is already smarter than we need for our use case. Request latency is my #1 concern right now.
- JimDabell 2y agoIf request latency is your top concern and you don’t need it to be as smart as GPT-4, why are you using OpenAI and not a model hosted by Groq or similar?
- mritchie712 2y agoWe need function calling, which wasn't great in Groq when we last tried it, but they seem to have made progress on it. We'll be testing it again soon.
- voiper1 2y agogroq just released: llama3-groq-70b-8192-tool-use-preview llama3-groq-8b-8192-tool-use-preview Sonnet3.5 also has tool usage, and is slightly cheaper than 4o ($3 input vs $5). If you _only_ need tool usage, then 4o-mini or others will be cheaper... mistal's Mixtral 8x22B has function calling for $2/$6, which is a bit cheaper than 4o.
- d13 2y agoGroq’s rate limits are so bad they’re only good enough for one person using it very rarely with an under 4k context size. It’s just a marketing stunt at the moment.
- fragmede 2y agohow much are you paying them?