3 ms·
The only system without a discrete GPU I've seen which pulls this off at a usable 8 ish tokens a second is an m3 macbook pro with 128gb unified memory [1], key
by jamal-kumar 3y ago
The only system without a discrete GPU I've seen which pulls this off at a usable 8 ish tokens a second is an m3 macbook pro with 128gb unified memory [1], key word here unified... Something like a 6000$ investment. You're possibly better off renting GPU somewhere or investing in a couple of rtx cards
[1] https://www.nonstopdev.com/llm-performance-on-m3-max/ https://www.nonstopdev.com/llm-performance-on-m3-max/
- abdullahkhalids 3y agoRTX4090 is going for 2-3K on ebay. And some other commenter is saying you need 2, to run a 70B model. So that's about the same price. Though I image the GPU is faster. https://www.ebay.com/sch/i.html?_from=R40&_nkw=RTX+4090&_sacat=0 https://www.ebay.com/sch/i.html?_from=R40&_nkw=RTX+4090&_sac...
- ynniv 3y agoTwo used 3090s aren't as fast as two 4090s, but they're $800 each.
- plasticchris 3y agoJust get 7900xtx cards and use MLC the performance (about 30 tokens/s) is comparable at 1k/card. And the Linux driver support is better.