Not op, but check out open code; you can turn on K/V quantization to help with increasing context if you have not already. I think K needs to stay at least 8 b

Not op, but check out open code; you can turn on K/V quantization to help with increasing context if you have not already. I think K needs to stay at least 8 but I hear V can go down to 4?