3 ms·

Looking at the playground, it appears the few shot examples in the prompt and CFG are duplicative. What is the relationship between the two? When you say in an

Looking at the playground, it appears the few shot examples in the prompt and CFG are duplicative. What is the relationship between the two? When you say in another comment that using OpenAI functions to output JSON is a waste of tokens, how are you generating the JSON output? And why do your prompts then include few shot examples of JSON objects?

sandkoan 3y ago
The prompt is given to our model as a guiding aid (a suggestion), and the cfg is used to constrain the model to generate only tokens that abide by the schema (an enforcement). That's how we ensure only valid outputs at text generation time. We also prefill some tokens depending on the set of allowed tokens at a given state, so the model doesn't waste resources trying to predict them.
- roseway4 3y ago
  When you say “our model”, are you using a custom LLM for completions vs OpenAI or other LLM vendor?
  - sandkoan 3y ago
    Custom LLM—hence the self-hostability.