3 ms·
> Might be time to invest in a Mac studio! The highest end Mac Studio with 196GB of ram won't even be enough to run a Q4 quant of the 400B+ (don't forget the
by Manabu-eo 2y ago
> Might be time to invest in a Mac studio!
The highest end Mac Studio with 196GB of ram won't even be enough to run a Q4 quant of the 400B+ (don't forget the +) model. At this point, one have to consider an Epyc for CPU inference or costlier gpu solutions like the "popular" 8xA100 80GB...
An if it's a dense model like the other llamas, it will be pretty slow..
- nilsherzig 2y agoIt's a dense one, zuck confirms this a couple minutes into the interview posted in this thread
- urbandw311er 2y agoJust FYI on the podcast video Zuck seems to let it slip that the exact number is 405B. (2-3mins in)
- buildbot 2y agoIt might be large enough Q2 won't impact performance too much (not guaranteed), but that's a great point. Time to invest in a Milan-X! (Already have the 512GB of DDR4)