3 ms·

Thinking shouldn't be too hard to deal with---just let the model generate freely until it hits a </think> token, then do constrained decoding, right?

• stymaar 4mo ago
  Sure, but does llama-cpp support that?
  • nl 4mo ago
    It does and this is how I did it. But actually getting that grammar right as well as actually making it work with the correct Jinja template to correctly enable thinking mode and parse it out was a lot more work than I expected.