Running LLama 2 70b a Mac Studio gets 13 tokens per second. 2 x 4090 is more like 38 tokens per second. But it's harder than that. As the 2 x 4090 is using ht

Running LLama 2 70b a Mac Studio gets 13 tokens per second. 2 x 4090 is more like 38 tokens per second. But it's harder than that. As the 2 x 4090 is using https://llm.mlc.ai/ https://llm.mlc.ai/ which is highly optimized so perhaps the Mac Studio inference could be better when tuned. Then a lot of production inference engines use continous batching, which means they perform really well when processing multiple prompts at a time.