3 ms·

Thank you for this really cool and open contribution! I will be watching llama.cpp closely for them to implement this! I’ve been looking for ways to speed up

by dartos 3y ago

Thank you for this really cool and open contribution! I will be watching llama.cpp closely for them to implement this! I’ve been looking for ways to speed up CPU inference and I really like this idea of “effort”

kolinko 3y ago
Hahah, thanks! It was a marathon to get develop this, and I'm glad it reached the front page. The name was proposed by chatgpt :) It claims it doesn't recognise this approach - so there is a chance it's really a new thing. I want to reach out to llama.cpp and the others - I hope it gets implemented. I considered just writing a patch to llama, but c++ and the scale of that project was beyond me. As for CPU inference - it should speed it up just as well. But thanks to the fact that it can load up a fraction of weights (e.g. just 70%, skipping the least important ones), it should be possible now to run models on less VRAM than before (still, Q8 needs to implemented though). Funnily - when I tried comparing benchmarks to llama.cpp, I couldn't find speeds for 7B/FP16 on MB Air 16GB, because it's impossible to run with regular methods. It is possible with Effort. Ditto, I was running full resolution, but cropped, Mixtral on my 96GB M2, even though it usually takes 114GB ram. I just loaded 75% of weights, and it was working smoothly. (before I messed something up with implementation and it now produces crap output - needs a fix)
- dhruvdh 3y ago
  I would imagine the importance of weights depends on the prompt. How do you decide which weights are important?
  - kolinko 3y ago
    Yeah, that is the point more or less - it dynamically chise the weights layer per layer depending on the internal state. A bit technical explaination here. https://kolinko.github.io/effort/equations.html https://kolinko.github.io/effort/equations.html
- indymike 3y ago
  > It is possible with Effort. "All things are possible with enough effort." -- Dad.
  - kolinko 3y ago
    Hahaha :)
- 0x4139 3y ago
  Implementing this approach could significantly enhance the adoption of LLMs within mobile phone libraries and other compact devices. I highly recommend opening an improvement issue for llama.cpp.

by dartos 3y ago

kolinko 3y ago

Hahah, thanks! It was a marathon to get develop this, and I'm glad it reached the front page. The name was proposed by chatgpt :) It claims it doesn't recognise this approach - so there is a chance it's really a new thing. I want to reach out to llama.cpp and the others - I hope it gets implemented. I considered just writing a patch to llama, but c++ and the scale of that project was beyond me. As for CPU inference - it should speed it up just as well. But thanks to the fact that it can load up a fraction of weights (e.g. just 70%, skipping the least important ones), it should be possible now to run models on less VRAM than before (still, Q8 needs to implemented though). Funnily - when I tried comparing benchmarks to llama.cpp, I couldn't find speeds for 7B/FP16 on MB Air 16GB, because it's impossible to run with regular methods. It is possible with Effort. Ditto, I was running full resolution, but cropped, Mixtral on my 96GB M2, even though it usually takes 114GB ram. I just loaded 75% of weights, and it was working smoothly. (before I messed something up with implementation and it now produces crap output - needs a fix)

dhruvdh 3y ago
I would imagine the importance of weights depends on the prompt. How do you decide which weights are important?
- kolinko 3y ago
  Yeah, that is the point more or less - it dynamically chise the weights layer per layer depending on the internal state. A bit technical explaination here. https://kolinko.github.io/effort/equations.html https://kolinko.github.io/effort/equations.html
indymike 3y ago
> It is possible with Effort. "All things are possible with enough effort." -- Dad.
- kolinko 3y ago
  Hahaha :)
0x4139 3y ago
Implementing this approach could significantly enhance the adoption of LLMs within mobile phone libraries and other compact devices. I highly recommend opening an improvement issue for llama.cpp.