3 ms·
Have those reports from George Hotz been confirmed? It seems plausible to me, but also suggests to me that we have further to go by using that parameter budget
by whimsicalism 3y ago
Have those reports from George Hotz been confirmed? It seems plausible to me, but also suggests to me that we have further to go by using that parameter budget for depth rather than for width.
- woeirua 3y agoIt seems consistent with the behavior we see when using GPT4 in chat mode. Every once in a while it will change its answer as it’s generating it, as though it’s switched which model it favors to produce the response. GPT3.5 doesn’t do that.
- famouswaffles 3y agoMoE models don't work like that though
- SgtBastard 3y agoToken-at-time MoE models could result in mid-output-stream directional change. I agree with you that whole-output MoE models don’t.
- letitgo12345 3y agoSeveral people in the community have confirmed it so while we don't "know know", it does seem like the dominant plausible rumor going around