And to build on this, take a look at the code change. Currently in llama.cpp there are many techniques for sampling the next token: llama_sample_token_greedy -

And to build on this, take a look at the code change. Currently in llama.cpp there are many techniques for sampling the next token: llama_sample_token_greedy - just take the top probability llama_sample_top_k - sample only from the top k probabilities etc ... this code change adds a new sample: llama_sample_grammar - sample only from tokens which match the grammar