3 ms·
Right, but how are we supposed to be getting anywhere else unless people start being more specific and stop leaning on anecdotes or repeating what they've heard
by diggan 2y ago
Right, but how are we supposed to be getting anywhere else unless people start being more specific and stop leaning on anecdotes or repeating what they've heard elsewhere?
Saying "Apple seems to be somewhat equal to this other setup" doesn't really contribute to someone getting an accurate picture if it is equal or not, unless we start including raw numbers, even if they aren't directly comparable.
I don't think it's too early to say "I get X tokens/second with this setup + these settings" because then we can at least start comparing, instead of just guessing which seems to be the current SOTA.
- kamranjon 2y agoA great thread with the type of info your looking for lives here: https://github.com/ggerganov/whisper.cpp/issues/89 https://github.com/ggerganov/whisper.cpp/issues/89 But you can likely find similar threads for the llama.cpp benchmark here: https://github.com/ggerganov/llama.cpp/tree/master/examples/llama-bench https://github.com/ggerganov/llama.cpp/tree/master/examples/... These are good examples because the llama.cpp and whisper.cpp benchmarks take full advantage of the Apple hardware but also take full advantage of non-Apple hardware with GPU support, AVX support etc. It’s been true for a while now that the memory bandwidth of modern Apple systems in tandem with the neural cores and gpu has made them very competitive Nvidia for local inference and even basic training.
- diggan 2y agoI guess I'm mostly lamenting about how unscientific these discussions are in general, on HN and elsewhere (besides specific GitHub repositories). Every community is filled with just anecdotal stories, or some numbers but missing to specify a bunch of settings + model + runtime details so people could at least compare it to something. Still, thanks for the links :)
- t1amat 2y agoIn fairness it’s become even more difficult now than ever before. * hardware spec * inference engine * specific model - differences to tokenizer will make models faster/slower with equivalent parameter count * quantization used - and you need to be aware of hardware specific optimizations for particular quants * kv cache settings * input context size * output token count This is probably not a complete list either.
- nickthegreek 2y agoBest place to get that kinda info is gonna be /r/LocalLlama