2 ms·
I just downloaded the latest llama.cpp and poolside Q4 and within 10k tokens it corrupted "Use ~/p/kevincox.ca as a reference" into "kevinc.ai" and then starts
by kevincox 2mo ago
I just downloaded the latest llama.cpp and poolside Q4 and within 10k tokens it corrupted "Use ~/p/kevincox.ca as a reference" into "kevinc.ai" and then starts doing stupid things because the directory "doesn't exist".
In previous testing the Q8 was better but still had the same problem frequently enough to be very difficult to use. I don't know if it is a runner bug or model bug but at least with llama.cpp this seems to always happen and I haven't seen an issue like this with any other model.
- alfiedotwtf 2mo agoWow that's weird. I have it running sometimes at least 8 hours straight on a single prompt and no weirdness. Could it be something weird like you're hitting your compaction limit?
- kevincox 2mo agoNo compaction happened. (That is set near 256k.) It seems to really not like my domain but I've seen it for various things as well (especially high-entropy tokens). Dropping random characters or various things. I've tried various flash attention and context quantization settings but nothing seemed to resolve it completely.