3 ms·

Thanks! One significant reason why the app works as well as it does is OmniQuant quantization. It takes a few hours (depending on the size of the model) to quan

Thanks! One significant reason why the app works as well as it does is OmniQuant quantization. It takes a few hours (depending on the size of the model) to quantize the model with OmniQuant, using an NVIDIA GPU. Using RTN quantized models from mlx-community or perhaps TheBloke on HF, would nullify that advantage. OTOH, OmniQuant is open source and if users can quantize their own models with it, they should be able to side load it. I think this is something that we can add in the near future.

smcleod 3y ago
That would be really neat. I’d happily leave my server (RTX3090) or MacBook chugging away for a few hours to get the models I’d like on my phone. Thanks for your reply!
- woadwarrior01 3y ago
  Yeah! That's pretty much my quantization workflow for small models. I've got a dual 3090 box under my desk to quantize 7B and smaller models and rent machines on vast.ai for larger models which need > 24GB of RAM for larger models (Mixtral needs an 80GB A100, for instance).