4 ms·
Prediction: GPT-5 will use a consortium of models for parallel reasoning, possibly including their oss versions. Each using different 'channels' from the harmo
by irthomasthomas 1y ago
Prediction: GPT-5 will use a consortium of models for parallel reasoning, possibly including their oss versions. Each using different 'channels' from the harmony spec.
I have a branch of llm-consortium where I was noodling with giving each member model a role. Only problem is it's expensive to evaluate these ideas so I put it on hold. But maybe now with oss models being cheap I can try and it on those.
- Imustaskforhelp 1y agoWhat are your thoughts on some other model like qwen using something like this? Pardon me but are you thinking that this method is superior than mixture of experts? What are your thoughts?
- irthomasthomas 1y agoI tested a consortium of qwens on the brainfuck test and it solved it, while the single models fail. MOEs are a single model. An 'expert' is a subset of layers chosen by a router model for each token. This makes them run faster. A consortium is a type of parallel reasoning that uses multiple of the same or different models to generate parallel response and find the best one. All models have a jagged frontier with weird skill gaps. A consortium can bridge those gaps and increase performance on the frontier.
- onlyrealcuzzo 1y agoHas anyone compared a consortium of leading edge 3B-20B models compared to the most powerful models? I'd love to see how they performed.
- irthomasthomas 1y agoDo you have a favourite benchmark? I may just have the budget for testing some 3b models
- nxobject 1y agoComputer science's favorite move: we've reached the limits of a scaling law meant to benefit single-threaded processes, so let's go parallel...
- BoredPositron 1y agowe are scaling in one direction for 2 years now...
- mindwok 1y agoThis is what Grok 4 Heavy does with apparent success.
- irthomasthomas 1y agoThey may have been inspired by it. It was shared by karpathy... https://x.com/karpathy/status/1870692546969735361 https://x.com/karpathy/status/1870692546969735361 I wish someone would extract the Grok Heavy prompts to confirm, but I guess those jailbreakers don't have the $200 sub.