4 ms·
Ask HN: How do GPT models generate variable length outputs??
Traditional ML models generally make predictions with a fixed length output , e.g. text of 100 characters. But gpt models can generate a yes/or no and sometimes they generate longer answers. How is it generated?
- sp332 3y agoThey generate one token at a time. The model outputs a probability for each possible output token, and the tooling around the model can sample from that distribution in several ways depending how much "creativity" you want to see in the output. Then that new token is added as part of the input, and the model is run again. That's why it can take tenths of a second to generate each token even on hardware with teraflops of processing power.
- nothrowaways 3y agoBut how is the length of autoregression be determined?
- onesphere 3y agoIf each token leads to another, it eventually leads to what commonly comes out as an ‘end token’. So each token has a certain chance of leading to a next token which might, possibly, say to end. current_token = <prompt> while current_token == true: next_token = statistically[current_token][choose_random_from(common_order)] current_token = next_token s = inf for i in common_order: p = 1 x = i while true: x = statistically[x][common_order[x]] if x == false: break p *= x if p < s: s = p shortest_to_end_start_token = i
- onesphere 3y agoOr: s = - inf ... if p > s: s = p longest_to_end_start_token = i And then maybe you could sort all tokens by their length (if you wanted to find a length of sub-x).