3 ms·
Similarly: Full F16 precision 34B Code Llama at >20T/s on M2 Ultra https://news.ycombinator.com/item?id=37346727 https://news.ycombinator.com/item?id=37346727
by kristianp 3y ago
Similarly:
Full F16 precision 34B Code Llama at >20T/s on M2 Ultra
https://news.ycombinator.com/item?id=37346727 https://news.ycombinator.com/item?id=37346727
Although that is for an experimental branch of llama.cpp, "PoC for speeding-up inference via speculative sampling".