4 ms·

llama.cpp support needed ASAP, not just quantized, but I hope it will be albe to run in a way that keeps the same two experts for minimum N tokens generated and

llama.cpp support needed ASAP, not just quantized, but I hope it will be albe to run in a way that keeps the same two experts for minimum N tokens generated and re-valuate the expert to select only then, so that it will be possible to do the inference in low ram doing something similar to "paging".

kristianp 3y ago
There is an issue for it: https://github.com/ggerganov/llama.cpp/issues/4381 https://github.com/ggerganov/llama.cpp/issues/4381
- rgbrgb 3y ago
  what a sick project to be able to attract a billionaire programmer [0] and c royalty. [0]: https://github.com/ggerganov/llama.cpp/issues/4216#issuecomment-1826393578 https://github.com/ggerganov/llama.cpp/issues/4216#issuecomm...
  - jaipilot747 3y ago
    For those out of the loop, who are the billionaire programmer and C royalty people in this link?
    - kostadin 3y ago
      Tobi Shopify and antirez (redis etc)
    - antirez 3y ago
      Tobi is the founder of Shopify.
    - KennyBlanken 3y ago
      I also looked at the threads and had no idea who they were referencing.
    - OJFord 3y ago
      The linked comment is authored by the CEO of Shopify; the top-level comment in this thread is the .. what would you say, author/originator/founder of Redis. (And maybe/presumably more widely known in C-land for other projects or contributions, idk.)
- kristianp 3y ago
  I'm looking forward to a 4-bit quantized version that I can run on my Intel Thinkpad with 32GB RAM, should be almost usable if its as fast as a 14B size model. Whether it's any good is another matter, I guess the leaderboard on HuggingFace will be updated at some point.
tarruda 3y ago
> so that it will be possible to do the inference in low ram doing something similar to "paging". I was also hoping for that, but my initial research suggests that you need to fit everything into VRAM (or RAM when doing llama.cpp CPU inference). The upside is that it should be as fast as running a single 7B model, which on llama.cpp should be in the ballpark of 10 tok/sec when 4-bit quantized. For local usage, I'd say you will have an "OK" experience in a recent laptop with 64GB RAM, which is more realistic than having the necessary VRAM in a Nvidia GPU.
- tarruda 3y ago
  Correction, it should have the performance of a single 12B parameter model: "Concretely, Mixtral has 45B total parameters but only uses 12B parameters per token. It, therefore, processes input and generates output at the same speed and for the same cost as a 12B model." https://mistral.ai/news/mixtral-of-experts/ https://mistral.ai/news/mixtral-of-experts/

kristianp 3y ago

There is an issue for it: https://github.com/ggerganov/llama.cpp/issues/4381 https://github.com/ggerganov/llama.cpp/issues/4381

rgbrgb 3y ago
what a sick project to be able to attract a billionaire programmer [0] and c royalty. [0]: https://github.com/ggerganov/llama.cpp/issues/4216#issuecomment-1826393578 https://github.com/ggerganov/llama.cpp/issues/4216#issuecomm...
- jaipilot747 3y ago
  For those out of the loop, who are the billionaire programmer and C royalty people in this link?
  - kostadin 3y ago
    Tobi Shopify and antirez (redis etc)
  - antirez 3y ago
    Tobi is the founder of Shopify.
  - KennyBlanken 3y ago
    I also looked at the threads and had no idea who they were referencing.
  - OJFord 3y ago
    The linked comment is authored by the CEO of Shopify; the top-level comment in this thread is the .. what would you say, author/originator/founder of Redis. (And maybe/presumably more widely known in C-land for other projects or contributions, idk.)
kristianp 3y ago
I'm looking forward to a 4-bit quantized version that I can run on my Intel Thinkpad with 32GB RAM, should be almost usable if its as fast as a 14B size model. Whether it's any good is another matter, I guess the leaderboard on HuggingFace will be updated at some point.

tarruda 3y ago

> so that it will be possible to do the inference in low ram doing something similar to "paging". I was also hoping for that, but my initial research suggests that you need to fit everything into VRAM (or RAM when doing llama.cpp CPU inference). The upside is that it should be as fast as running a single 7B model, which on llama.cpp should be in the ballpark of 10 tok/sec when 4-bit quantized. For local usage, I'd say you will have an "OK" experience in a recent laptop with 64GB RAM, which is more realistic than having the necessary VRAM in a Nvidia GPU.

tarruda 3y ago
Correction, it should have the performance of a single 12B parameter model: "Concretely, Mixtral has 45B total parameters but only uses 12B parameters per token. It, therefore, processes input and generates output at the same speed and for the same cost as a 12B model." https://mistral.ai/news/mixtral-of-experts/ https://mistral.ai/news/mixtral-of-experts/