3 ms·

I think we are somewhat still at the “fuzzy super early adopter” stage of this local LLM game and hard data is not going to be easy to come by. I almost want t

I think we are somewhat still at the “fuzzy super early adopter” stage of this local LLM game and hard data is not going to be easy to come by. I almost want to use the word “hobbiest stage” where almost all of the “data” and “best practice” is anecdotal but I think we are a step above that. Still, it’s way to early and there are simply way to many hardware and software combinations that change almost weekly to establish “the best practice hardware configuration for training / inferencing large language models locally”. Some day there will be established guides with solid. In fact someday there will be be PC’s that specifically target LLMs and will feature all kinds of stats aimed at getting you to bust out your wallet. And I even predict they’ll come up with metrics that all the players will chase well beyond when those metrics make sense (megapixels, clock frequency, etc)… but we aren’t there yet!

diggan 2y ago
Right, but how are we supposed to be getting anywhere else unless people start being more specific and stop leaning on anecdotes or repeating what they've heard elsewhere? Saying "Apple seems to be somewhat equal to this other setup" doesn't really contribute to someone getting an accurate picture if it is equal or not, unless we start including raw numbers, even if they aren't directly comparable. I don't think it's too early to say "I get X tokens/second with this setup + these settings" because then we can at least start comparing, instead of just guessing which seems to be the current SOTA.
- kamranjon 2y ago
  A great thread with the type of info your looking for lives here: https://github.com/ggerganov/whisper.cpp/issues/89 https://github.com/ggerganov/whisper.cpp/issues/89 But you can likely find similar threads for the llama.cpp benchmark here: https://github.com/ggerganov/llama.cpp/tree/master/examples/llama-bench https://github.com/ggerganov/llama.cpp/tree/master/examples/... These are good examples because the llama.cpp and whisper.cpp benchmarks take full advantage of the Apple hardware but also take full advantage of non-Apple hardware with GPU support, AVX support etc. It’s been true for a while now that the memory bandwidth of modern Apple systems in tandem with the neural cores and gpu has made them very competitive Nvidia for local inference and even basic training.
  - diggan 2y ago
    I guess I'm mostly lamenting about how unscientific these discussions are in general, on HN and elsewhere (besides specific GitHub repositories). Every community is filled with just anecdotal stories, or some numbers but missing to specify a bunch of settings + model + runtime details so people could at least compare it to something. Still, thanks for the links :)
    - t1amat 2y ago
      In fairness it’s become even more difficult now than ever before. * hardware spec * inference engine * specific model - differences to tokenizer will make models faster/slower with equivalent parameter count * quantization used - and you need to be aware of hardware specific optimizations for particular quants * kv cache settings * input context size * output token count This is probably not a complete list either.
- nickthegreek 2y ago
  Best place to get that kinda info is gonna be /r/LocalLlama
motorest 2y ago
> I think we are somewhat still at the “fuzzy super early adopter” stage of this local LLM game and hard data is not going to be easy to come by. What's hard about it? You get the hardware, you run the software, you take measurements.
- GTP 2y ago
  Yes, but we don't have enough people doing that to get quality data. Not many people are building this kind of setup, and even less are publishing their results. Additionally, if I just run a test a couple of time and then average the results, this is still far from a solid measurement.
  - diggan 2y ago
    > but we don't have enough people doing that to get quality data But how are we supposed to get enough people doing those things if everyone say "There isn't enough data right now for it to be useful"? We have to start somewhere
    - unshavedyak 2y ago
      I don't think they're saying anything counter to that. The people who don't require the volume of data will run these. Ie the super early adopters.
    - colonCapitalDee 2y ago
      We've already started, we just haven't finished yet

diggan 2y ago

Right, but how are we supposed to be getting anywhere else unless people start being more specific and stop leaning on anecdotes or repeating what they've heard elsewhere? Saying "Apple seems to be somewhat equal to this other setup" doesn't really contribute to someone getting an accurate picture if it is equal or not, unless we start including raw numbers, even if they aren't directly comparable. I don't think it's too early to say "I get X tokens/second with this setup + these settings" because then we can at least start comparing, instead of just guessing which seems to be the current SOTA.

kamranjon 2y ago
A great thread with the type of info your looking for lives here: https://github.com/ggerganov/whisper.cpp/issues/89 https://github.com/ggerganov/whisper.cpp/issues/89 But you can likely find similar threads for the llama.cpp benchmark here: https://github.com/ggerganov/llama.cpp/tree/master/examples/llama-bench https://github.com/ggerganov/llama.cpp/tree/master/examples/... These are good examples because the llama.cpp and whisper.cpp benchmarks take full advantage of the Apple hardware but also take full advantage of non-Apple hardware with GPU support, AVX support etc. It’s been true for a while now that the memory bandwidth of modern Apple systems in tandem with the neural cores and gpu has made them very competitive Nvidia for local inference and even basic training.
- diggan 2y ago
  I guess I'm mostly lamenting about how unscientific these discussions are in general, on HN and elsewhere (besides specific GitHub repositories). Every community is filled with just anecdotal stories, or some numbers but missing to specify a bunch of settings + model + runtime details so people could at least compare it to something. Still, thanks for the links :)
  - t1amat 2y ago
    In fairness it’s become even more difficult now than ever before. * hardware spec * inference engine * specific model - differences to tokenizer will make models faster/slower with equivalent parameter count * quantization used - and you need to be aware of hardware specific optimizations for particular quants * kv cache settings * input context size * output token count This is probably not a complete list either.
nickthegreek 2y ago
Best place to get that kinda info is gonna be /r/LocalLlama

motorest 2y ago

> I think we are somewhat still at the “fuzzy super early adopter” stage of this local LLM game and hard data is not going to be easy to come by. What's hard about it? You get the hardware, you run the software, you take measurements.

GTP 2y ago
Yes, but we don't have enough people doing that to get quality data. Not many people are building this kind of setup, and even less are publishing their results. Additionally, if I just run a test a couple of time and then average the results, this is still far from a solid measurement.
- diggan 2y ago
  > but we don't have enough people doing that to get quality data But how are we supposed to get enough people doing those things if everyone say "There isn't enough data right now for it to be useful"? We have to start somewhere
  - unshavedyak 2y ago
    I don't think they're saying anything counter to that. The people who don't require the volume of data will run these. Ie the super early adopters.
  - colonCapitalDee 2y ago
    We've already started, we just haven't finished yet