4 ms·
Unfortunately the download is taking its time - which kind of base model is it using and what techniques (if any) are they using to offload weights? Since the
by strangecasts 3y ago
Unfortunately the download is taking its time - which kind of base model is it using and what techniques (if any) are they using to offload weights?
Since the demo is 35 GB, my first assumption was it's bundling a ~13B parameter model, but if the requirement is 8 GB VRAM, I assume they're either doing quantization on the user's end or offloading part of the model to the CPU.
(I also hope that Windows 11 is a suggested and not a hard requirement)
- operator-name 3y agoFor some reason it's actually bundling both LLaMA 13b (24.5GB) and Ministral 7b (13.6GB), but only installed Ministral 7b. I have a 3070ti 8GB, so maybe it installs the other one if you have more VRAM?
- ReFruity 3y agoI have 3070 and when I choose LLaMA in config it just changes it back to Mistral on launch