You have more inputs to an LLM than just the prompt. For example, people commonly pass parameters which control the sampling of tokens. Implementing grammar ba

You have more inputs to an LLM than just the prompt. For example, people commonly pass parameters which control the sampling of tokens. Implementing grammar based sampling does NOT require "re-prompting until it gets it right". Imagine a point in time when the LLM is generating some particular token. Which token will it produce? To decide that, it evaluates and assigns a score to each potential token. Then it chooses one of these options based on some rules. Rules could be as simple as "pick the token with the highest score". That is called a greedy strategy. Usually more complex strategies are used and they typically have some randomness. That is called sampling. You can imagine a grammar based sampling strategy to force specific tokens at specific positions in the output, for example, to close a bracket in json.