3 ms·
I find 100tps unlikely, I see 13tps on an 80GB A100 for a 70B 4bit quantized model. Can you link - I am answering an email on monday where this info would be v
by sgt101 2y ago
I find 100tps unlikely, I see 13tps on an 80GB A100 for a 70B 4bit quantized model.
Can you link - I am answering an email on monday where this info would be very useful!
- JKCalhoun 2y agoPeople on Apple Silicon are running special models for MLX. Maybe start with the MLX Community page: https://huggingface.co/mlx-community https://huggingface.co/mlx-community