3 ms·
> That redefines consumer to a 3090 Or a beefy MacBook Pro. I recently bought one with 64gb of memory and Llama 65B infers very promptly as long as I'm using q
by mrtranscendence 3y ago
> That redefines consumer to a 3090
Or a beefy MacBook Pro. I recently bought one with 64gb of memory and Llama 65B infers very promptly as long as I'm using quantized weights (and the Mac's GPU).
- aftbit 3y agoAnd beam size 1?
- b33j0r 3y agoThis is very impressive. I think everyone should pay very close attention to what M1/M2 have given us. But I’m waiting until my friends can afford it. Right now (which in this pace might mean I change my mind tonight) …I am earnestly studying how to make this a thing anyone can install as a part of a product they can use without a subscription.