3 ms·
Since they mentioned they’re working on Mistral-7B, I’d like to note that my GPU-only implementation of Mistral uses slightly over 5GB of VRAM: https://github.c
by Const-me 3y ago
Since they mentioned they’re working on Mistral-7B, I’d like to note that my GPU-only implementation of Mistral uses slightly over 5GB of VRAM: https://github.com/Const-me/Cgml https://github.com/Const-me/Cgml
Runs pretty good on most consumer-grade GPUs, but so far it only supports Windows OS.
- m1sta_ 3y agoThis looks really really interesting. Any idea whether it would run on a laptop with an Intel Core i7?
- MacsHeadroom 3y agoYes, I run it on CPU using LLMStudio. It's very fast.
- Const-me 3y agoThe performance on integrated GPUs is not stellar, but it should work. On my AMD Ryzen 5 5600u with dual-channel DDR4, I’m getting 2 tokens/second. My friend with Intel Core i3 and single-channel memory was getting 1 token/second.
- ru552 3y agoVRAM is king. If you have the VRAM to hold the model parameters, you can run it.
- v3ss0n 3y agotry ollama , only needs about 4GB it uses llmcpp