4 ms·
Pick a decent quant (4-6KM) then use llama-fit-params and try it yourself to see if it's giving you what you need.
by tommy_axle 6mo ago
Pick a decent quant (4-6KM) then use llama-fit-params and try it yourself to see if it's giving you what you need.
- gunalx 6mo agoI habe found llama-fit sometimes just selects a way to conservative load with VRAM to spare.