Any Macbook with 32GB should be able to run Meta-Llama-3-70B-Instruct.Q2_K.llamafile which I uploaded a few minutes ago. It's smart enough to solve math riddles

Any Macbook with 32GB should be able to run Meta-Llama-3-70B-Instruct.Q2_K.llamafile which I uploaded a few minutes ago. It's smart enough to solve math riddles, but at this level of quantization you should expect hallucinations. If you want to run Q4_0 you'll probably be able to squeeze it on a $3,999.00 Macbook Pro M3 Max w/ 48GB of RAM. If you want to run Q5_K_M or or Q8_0 the best choice is probably Mac Studio. I have an Apple M2 Ultra w/ 24‑core CPU, 60‑core GPU, 128GB RAM. It cost me $8000 with the monitor. If I run Meta-Llama-3-70B-Instruct.Q4_0.llamafile then I get 14 tok/sec (prompt eval is 82 tok/sec) thanks to the Metal GPU. You could alternatively go on vast.ai and rent a system with 4x RTX 4090's for a few bucks an hour. That'll run 70b. Or you could build your own, but the graphics cards alone will cost $10k+. AMD Threadripper Pro 7995WX ($10k) does a good job too. I get 5.9 tok/sec eval with Q4_0 and 49 tok/sec prompt. If I use F16 weights then prompt eval goes 65 tok/sec.