3 ms·

It can. It's something that is implemented by the thing that runs the model - eg Llama.cpp - rather than the model itself. Note that it is hard to make work i

by nl 4mo ago

It can. It's something that is implemented by the thing that runs the model - eg Llama.cpp - rather than the model itself. Note that it is hard to make work if you turn thinking on because the grammar gets complicated quickly (I don't recall if Qwen 0.6B can do thinking).

aesthesia 4mo ago
Thinking shouldn't be too hard to deal with---just let the model generate freely until it hits a </think> token, then do constrained decoding, right?
- stymaar 4mo ago
  Sure, but does llama-cpp support that?
  - nl 4mo ago
    It does and this is how I did it. But actually getting that grammar right as well as actually making it work with the correct Jinja template to correctly enable thinking mode and parse it out was a lot more work than I expected.
nextaccountic 4mo ago
Just one question. If I'm running a local model, can I do something other than just a context free grammar? Does it makes sense to have something more general, or it would be just too slow? I guess the only hard constraint is to not have backtracking, right? To not waste previously emitted tokens