3 ms·
I also heard this. I unfortunately forget which study it was but yes, their paper spoke of likely diminishing returns at least at around 400-500B parameters for
by jug 3y ago
I also heard this. I unfortunately forget which study it was but yes, their paper spoke of likely diminishing returns at least at around 400-500B parameters for current LLM's. The recent news of GPT-4 running on 8x 220B LLM's (which doesn't equal a 8*220B size) fits that range and it's also questionable how much further we can push LLM's further by introducing multiple models like this, because this too eventually introduces problems due to granularity and picking the right model if I understood this correctly (from an earlier Hacker News discussion). (sorry for altogether no sources lol)
- whimsicalism 3y agoHave those reports from George Hotz been confirmed? It seems plausible to me, but also suggests to me that we have further to go by using that parameter budget for depth rather than for width.
- woeirua 3y agoIt seems consistent with the behavior we see when using GPT4 in chat mode. Every once in a while it will change its answer as it’s generating it, as though it’s switched which model it favors to produce the response. GPT3.5 doesn’t do that.
- famouswaffles 3y agoMoE models don't work like that though
- SgtBastard 3y agoToken-at-time MoE models could result in mid-output-stream directional change. I agree with you that whole-output MoE models don’t.
- letitgo12345 3y agoSeveral people in the community have confirmed it so while we don't "know know", it does seem like the dominant plausible rumor going around