5 ms·
This is where inference speed starts to matter. H100 might be cheaper per inference than Groq but cutting down the wait time from 1 minute to 10 seconds could b
by hackerlight 3y ago
This is where inference speed starts to matter. H100 might be cheaper per inference than Groq but cutting down the wait time from 1 minute to 10 seconds could be a big deal.
- anonzzzies 3y agoHave you tried Groq? We did a few days testing on replacing gpt4-turbo with it and, while incredibly fast, the results were horrible, even after a lot of specific prompt engineering. So many hallucinations and such. Our products all have to do with strict generation and software quality; it basically has to fill in the blanks but it was incredibly hit or miss. Some results came in within a second so even a few iterations beat gpt4 when correct, but some needed so many (that we quit) iterations that gpt4 beat it hands down.
- simonvc 3y agothey just run other open models, so you're complaint isn't about Groq, it's about GPT-4 vs mixtral 8x7b accelerated
- anonzzzies 3y agoSure, so when openai moves to groq it might be something. Groq with the current models is impressive but doesn’t work for us is what I am saying. As I don’t actually have access to other models on groq, this is groq as it stands.
- hackerlight 3y agoGroq is hardware not software... It's like saying the H100 hallucinated.
- throwaway11460 3y agoWell, why not. It's like saying that Windows crashed - while it actually was some driver or app that caused it. The hardware (or OS) is useless if it doesn't give good results, even if it theoretically could.