4 ms·
Fyi, if you're trying to run this under AMD/HIP: PTQ1_0 has no optimized MMQ-Path in their llama-cpp fork, try running PTQ2_0 (needs a bit more vram, but is ab
by nilsherzig 8d ago
Fyi, if you're trying to run this under AMD/HIP:
PTQ1_0 has no optimized MMQ-Path in their llama-cpp fork, try running PTQ2_0 (needs a bit more vram, but is about 2x faster on my 6700 XT)
https://gist.github.com/nilsherzig/b8266d001c5c01bdb3d81d20915572fc https://gist.github.com/nilsherzig/b8266d001c5c01bdb3d81d209...
- jakswa 8d agowhat kinda speeds do you see on 6700 XT? i'm always conflicted on investing time chasing speed-vs-quality tradeoffs. I've got a 7900 XT (about double the IO throughput). I'll probably end up giving it a go when I find time.