3 ms·
An LLM does not generate "the next token" - from an input text, it generates a vector of probabilities where each slot in the vector corresponds to a token. The
by tylerhou 3y ago
An LLM does not generate "the next token" - from an input text, it generates a vector of probabilities where each slot in the vector corresponds to a token. The value in a token's slot is (approximately) the probability that that particular token might appear next in the text.
Programs like ChatGPT "interpret" that vector of probabilities to generate text by selecting (sampling) one of the top tokens. But sometimes this is too flexible -- for example, ChatGPT might generate invalid JSON when you want JSON output because it chose a token that does not conform to the JSON grammar.
A way to "force" an LLM to generate e.g. JSON is to change the sampling process. Instead of choosing any top token, we first filter the tokens to just those that conform to the JSON grammar. Then, we sample one of the top tokens from that subset.
- ttul 3y agoAnd given that the inference code has access to the entire vector, it's the logical place to put this filtering... OpenAI and other LLM APIs probably don't want to return the entire token probability vector to the user because it's a lot of data. That being said, it wouldn't surprise me if Microsoft has such access as part of their deal because of the obviously superior position this puts them in vs. regular API customers.
- bryan0 3y agoAnd to build on this, take a look at the code change. Currently in llama.cpp there are many techniques for sampling the next token: llama_sample_token_greedy - just take the top probability llama_sample_top_k - sample only from the top k probabilities etc ... this code change adds a new sample: llama_sample_grammar - sample only from tokens which match the grammar
- dave1010uk 3y agoIf the performance was ok, is there any reason why a sampler couldn't call an API or use a separate fine tuned model?
- machiaweliczny 3y agoDamn, I thought that making DiffuserAST is smart but sampler based on typescript closures might be much easier to implement. Gotta try this.