3 ms·
Can’t you run larger models easily on MacBook Pro laptops with the bigger memory options? I think I read that people are getting 100 tokens a second on 70B mode
by blackeyeblitzar 2y ago
Can’t you run larger models easily on MacBook Pro laptops with the bigger memory options? I think I read that people are getting 100 tokens a second on 70B models.
- efficax 2y agohaven’t measured but the 70b runs fine on an m1 macbook pro with 64gb of ram, although you can’t do much until it has finished
- politelemon 2y agoYou can get even faster results with GPUs, but that isn't the purpose of this demo. It's showcasing the ability to run such models on commodity hardware, and hopefully with better performance in the future.
- sgt101 2y agoI find 100tps unlikely, I see 13tps on an 80GB A100 for a 70B 4bit quantized model. Can you link - I am answering an email on monday where this info would be very useful!
- JKCalhoun 2y agoPeople on Apple Silicon are running special models for MLX. Maybe start with the MLX Community page: https://huggingface.co/mlx-community https://huggingface.co/mlx-community