3 ms·
That would be really neat. I’d happily leave my server (RTX3090) or MacBook chugging away for a few hours to get the models I’d like on my phone. Thanks for you
by smcleod 3y ago
That would be really neat. I’d happily leave my server (RTX3090) or MacBook chugging away for a few hours to get the models I’d like on my phone. Thanks for your reply!
- woadwarrior01 3y agoYeah! That's pretty much my quantization workflow for small models. I've got a dual 3090 box under my desk to quantize 7B and smaller models and rent machines on vast.ai for larger models which need > 24GB of RAM for larger models (Mixtral needs an 80GB A100, for instance).