This is super helpful! Currently it runs 4-bit quantization models (with names often ending in q4_0). Sorry this isn't more obvious. Will fix that and soon upda

This is super helpful! Currently it runs 4-bit quantization models (with names often ending in q4_0). Sorry this isn't more obvious. Will fix that and soon update the hosted models to support different quantization levels, so you can choose including k quant models. More to come there. It does use GGML under the hood. Will definitely make sure that's more obvious for compatibility. There's about to be an update as well that includes those recent changes. We do vendor the source code since it made building with Go a lot easier, but definitely don't want to be "forking" it (vs supporting it) – it's a brilliant piece of software!