32 ms·
For what reason would you want to use this instead of open source alternatives like Mistral
by nylonstrung 3y ago
For what reason would you want to use this instead of open source alternatives like Mistral
- zozbot234 3y agoIsn't this Apache licensed? Regardless, you can run multiple models concurrently on the same input using well-known ensemble techniques. (Not to be confused with mixture-of-experts, which is more like training a single model where only a few blocks are chosen to be active at any given time - a kind of sparsity.)
- tlb 3y agoNot super easy if they have different tokenizers.
- rvnx 3y agoMistral opened their weights only for very small LLaMA-like model.
- MallocVoidstar 3y agoI'm pretty sure Mixtral outperforms Grok-1 and uses much less memory to do it
- elfbargpt 3y agoI'm a little out of touch, is there a way to see how Grok measures up to other models?
- amrrs 3y agoBenchmarks here https://x.ai/blog/grok https://x.ai/blog/grok
- refulgentis 3y agoAnd to compare, you can sort by MMLU on here: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb.... Edit: to include my self summary after review: There's a good 100 models better than, a couple 1x7b even. Mixtral stomps it, half mixtral are universally better but one is close to same.
- lossolo 3y agoThis benchmark is mostly worthless, some of the top models there were trained on benchmark data, which is a known fact in the community. The only reliable benchmark: https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboard https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboar...
- refulgentis 3y agoNo, it's not "mostly worthless" and yes, some of the top models were removed a few months back from being trained on benchmark data. I urge you to at least think through what alternative you propose before posting so aggressively in these situations. Lmsys doesn't have Grok, or I would have included it. And having _some_ data is better than none. I also had someone arguing with me 6 months back that we can't trust any benchmarks at all from vendors, which would exclude the blog post. Instead of just repeating that back vehemently, I filled a gap. It's important we don't self-peasantize as a species, all data has its issues, that doesn't mean we throw it all out.
- michaelt 3y agoQuantifiable metrics are useful if they're credible, certainly. But does it seem likely, to you, that a 7B-parameter model would outperform a 314B-parameter model? Given that we can look at the chatbot arena leaderboard and it's dominated by proprietary, 70B and 8x7B models? A well regarded and modern model like Mixtral 8x7B, which is ranked 13th on the chatbot arena leaderboard, scores 72.7 'Average' on the open LLM leaderboard - and yet 'pastiche-crown-clown-7b-dare-dpo' scores 76.5. To me, that sounds too good to be true.
- cavisne 3y agoOne of the interesting things when weights are open sourced is the community can often improve the results. See all the bugs fixed in Gemma for an example.
- ein0p 3y agoDoubtful, for purely information theoretic and memory capacity reasons. It may outperform on some synthetic metrics, but in practice, to a human, larger models just feel “smarter” because they have a lot more density in their long tail where metrics never go
- verticalscaler 3y agoWell if nothing else, this one might be significantly less nerfed. Very interesting to compare to the others.
- refulgentis 3y agoIt's not, and I mean it, specifically in groks case. Generally, it's a boring boneheaded talking point that the 1% of us actually working in AI use as a sorting hat for who else is.
- not_really 3y agolol, okay
- mlindner 3y agoCurious why you're so dismissive of something that's pretty important?
- random_cynic 3y agoThe 1% who actually work on AI don't use terms as generic as "AI". Way to reveal yourself as college undergrad who read a couple of popular science books, downloaded MNIST data and thinks they are "experts".
- verticalscaler 3y ago[flagged]
- refulgentis 3y ago(not sure you're going to edit again, but in the current one, I'm not sure what Google's silly stock image warning has to do with anything, and I have generally chosen to avoid engaging people doing their politics hobby via AI discussion, since it became okay to across the ideological spectrum of my peers. So, mu is my answer.) And you're right, I was really surprised to see the harder right people throwing up their hands after the Gemini stuff.