3 ms·
How does the token usage compare to vanilla structured output? Many of these libraries do multiple requests to constrain output and measure logprobs.
by potatoman22 2y ago
How does the token usage compare to vanilla structured output? Many of these libraries do multiple requests to constrain output and measure logprobs.
- ngrislain 2y agoSame token usage. Actually OpenAI returns the logprob of each token conditional on the previous ones with the option logprobs=true. This lib simply parses the output json string with `lark` into an AST with value nodes. The value nodes are mapped back to a range of characters in the json string. Then the characters are mapped back to the GPT tokens overlapping the character ranges and the logprobs of the tokens are summed.
- potatoman22 2y agoThat's great to hear, thanks for the explanation! Super excited to try this out.