4 ms·

Llama 2 70B on M2 Max at 7 tokens/sec

42 points by tikkun 3y ago

tikkun 3y ago
For context: based on current gpt-4 times,[1] GPT-4 is about 12 tok/s on avg. GPT-3.5 is about 35 tok/s. OP got 13 tok/s on an A100. And 10 tok/s on M2 Ultra. [1]: This site that I run https://openai-status.llm-utils.org https://openai-status.llm-utils.org Step by step guide: https://mlc.ai/mlc-llm/docs/get_started/try_out.html https://mlc.ai/mlc-llm/docs/get_started/try_out.html
- MuffinFlavored 3y ago
  Two variables here when comparing to GPT-3.5 / GPT-4 time taken to generate a token (speed) quality of tokens generated How far off is Llama 2 70B in terms of hallucination/qualify of output compared to GPT-3.5 / GPT-4?
- nailk 3y ago
  Looks great! Are there other benchmarks? How does the speed compare to other LLM engines like llama.cpp / vllm (on GPUs)? Is it able to do continuous batching of incoming requests like vllm?
- Tiberium 3y ago
  You're probably aware, but OpenAI has different speeds for different kinds of customers. The most prioritized ones are accounts that have gpt-4-32k. An easy to see this difference is comparing a trial account's Turbo speed to a pay-as-you-go one. With an account with gpt-4-32k I observed around 125 tokens/s for gpt-3.5-turbo-16k maximum (4k turbo was slower by the way), as for normal gpt-4 on the same account - I don't remember exactly, but it was at least 20 tokens/sec. In short, you can clearly observe the difference in speed (even for Turbo) like this: trial account < paid < paid with gpt-4 access < gpt-4-32k-enabled account
kristianp 3y ago
Similarly: Full F16 precision 34B Code Llama at >20T/s on M2 Ultra https://news.ycombinator.com/item?id=37346727 https://news.ycombinator.com/item?id=37346727 Although that is for an experimental branch of llama.cpp, "PoC for speeding-up inference via speculative sampling".
holtkam2 3y ago
Pretty cool. I'm trying to wrap my head around this. Can someone explain what this mlc tool does? My understanding is that it "compiles" an LLM to C code that can run on any platform? How is this different from "quantization"?
deleted 3y ago
[deleted]

42 points by tikkun 3y ago

For context: based on current gpt-4 times,[1] GPT-4 is about 12 tok/s on avg. GPT-3.5 is about 35 tok/s. OP got 13 tok/s on an A100. And 10 tok/s on M2 Ultra. [1]: This site that I run https://openai-status.llm-utils.org https://openai-status.llm-utils.org Step by step guide: https://mlc.ai/mlc-llm/docs/get_started/try_out.html https://mlc.ai/mlc-llm/docs/get_started/try_out.html

MuffinFlavored 3y ago
Two variables here when comparing to GPT-3.5 / GPT-4 time taken to generate a token (speed) quality of tokens generated How far off is Llama 2 70B in terms of hallucination/qualify of output compared to GPT-3.5 / GPT-4?
nailk 3y ago
Looks great! Are there other benchmarks? How does the speed compare to other LLM engines like llama.cpp / vllm (on GPUs)? Is it able to do continuous batching of incoming requests like vllm?
Tiberium 3y ago
You're probably aware, but OpenAI has different speeds for different kinds of customers. The most prioritized ones are accounts that have gpt-4-32k. An easy to see this difference is comparing a trial account's Turbo speed to a pay-as-you-go one. With an account with gpt-4-32k I observed around 125 tokens/s for gpt-3.5-turbo-16k maximum (4k turbo was slower by the way), as for normal gpt-4 on the same account - I don't remember exactly, but it was at least 20 tokens/sec. In short, you can clearly observe the difference in speed (even for Turbo) like this: trial account < paid < paid with gpt-4 access < gpt-4-32k-enabled account

kristianp 3y ago

Similarly: Full F16 precision 34B Code Llama at >20T/s on M2 Ultra https://news.ycombinator.com/item?id=37346727 https://news.ycombinator.com/item?id=37346727 Although that is for an experimental branch of llama.cpp, "PoC for speeding-up inference via speculative sampling".

holtkam2 3y ago

Pretty cool. I'm trying to wrap my head around this. Can someone explain what this mlc tool does? My understanding is that it "compiles" an LLM to C code that can run on any platform? How is this different from "quantization"?

deleted 3y ago

[deleted]