3 ms·
GPT is still a language model, so at some point it's still just tokens. Is this just a schema validation layer on their end to avoid the round trip (and cost)
by tcdent 2y ago
GPT is still a language model, so at some point it's still just tokens.
Is this just a schema validation layer on their end to avoid the round trip (and cost) of repeating the call?
- gamegoblin 2y agoLanguage models like GPT output a large vector of probabilities for the next token. Then a sampler decides which of those tokens to pick. The simplest algorithm for getting good quality output is to just always pick the highest probability token. If you want more creativity, maybe you pick randomly among the top 5 highest probability tokens or something. There are a lot of methods. All that grammar-constrained decoding does is zero out the probability of any token that would violate the grammar.
- nickreese 2y agoThank you for this explanation. A few things just clicked for me.