3 ms·
Not quite there yet, but very close and not done training! It's quite plausible that this model could be state of the art over GPT-4 in some domains when it fin
by modeless 3y ago
Not quite there yet, but very close and not done training! It's quite plausible that this model could be state of the art over GPT-4 in some domains when it finishes training, unless GPT-5 comes out first.
Although 400B will be pretty much out of reach for any PC to run locally, it will still be exciting to have a GPT-4 level model in the open for research so people can try quantizing, pruning, distilling, and other ways of making it more practical to run. And I'm sure startups will build on it as well.
- int_19h 3y agoThe real question will be, how much you can quantize that while still retaining sanity. 400b at 2-bit would be possible to run on a Mac Studio - probably at multiple seconds per token, but sometimes that's "fast enough".
- modeless 3y agoYes. I expect an explosion of research and experimentation in model compression. The good news is I think there are tons of avenues that have barely been explored at all. We are at the very beginning of understanding this stuff, and my bet is that in a few years we'll be able to compress these models 10x or more.
- brandall10 3y agoThere are rumors about an upcoming M3 or M4 Extreme chip... which would certainly have enough RAM, and probably a 1600-2000 GB/s bandwidth. Still wouldn't be super performant AFA token gen, ~4-6 per second, but certainly runnable. Of course by the time that lands in 6-12 months we'll probably have a 70-100G model that is similarly performant.