3 ms·
I've been using Private LLM a few weeks now, it's certainly the best of the local options on an iPhone. I'm wondering if you're going to add support for downlo
by smcleod 3y ago
I've been using Private LLM a few weeks now, it's certainly the best of the local options on an iPhone.
I'm wondering if you're going to add support for downloading custom models from HF? (e.g. mlx-community models).
- woadwarrior01 3y agoThanks! One significant reason why the app works as well as it does is OmniQuant quantization. It takes a few hours (depending on the size of the model) to quantize the model with OmniQuant, using an NVIDIA GPU. Using RTN quantized models from mlx-community or perhaps TheBloke on HF, would nullify that advantage. OTOH, OmniQuant is open source and if users can quantize their own models with it, they should be able to side load it. I think this is something that we can add in the near future.
- smcleod 3y agoThat would be really neat. I’d happily leave my server (RTX3090) or MacBook chugging away for a few hours to get the models I’d like on my phone. Thanks for your reply!
- woadwarrior01 3y agoYeah! That's pretty much my quantization workflow for small models. I've got a dual 3090 box under my desk to quantize 7B and smaller models and rent machines on vast.ai for larger models which need > 24GB of RAM for larger models (Mixtral needs an 80GB A100, for instance).