4 ms·

Maybe mention that GGML is under the hood. I see you've forked the library, did you pick up the GQA changes needed for llama2-70b? Also seems like you are pic

Maybe mention that GGML is under the hood. I see you've forked the library, did you pick up the GQA changes needed for llama2-70b? Also seems like you are picking quants on people's behalf, which ones?

jmorgan 3y ago
This is super helpful! Currently it runs 4-bit quantization models (with names often ending in q4_0). Sorry this isn't more obvious. Will fix that and soon update the hosted models to support different quantization levels, so you can choose including k quant models. More to come there. It does use GGML under the hood. Will definitely make sure that's more obvious for compatibility. There's about to be an update as well that includes those recent changes. We do vendor the source code since it made building with Go a lot easier, but definitely don't want to be "forking" it (vs supporting it) – it's a brilliant piece of software!