3 ms·
Tested this on Apple Silicon at 128k context, very poor results with random character hallucinations.
by ukuina 2y ago
Tested this on Apple Silicon at 128k context, very poor results with random character hallucinations.
- growfeather 2y agoThanks for testing it. I wonder if there’s a reasonable line? 64k? How much RAM did the 128k context require? (Maybe future models might be better at that level)
- nicognaw 2y agoIf you got random tokens on Apple Silicon Macs, you should check your stderr for Metal’s error code, like 5 for out of metal memory. Metal lets you use half of your unified memory as metal memory.