4 ms·
They generate one token at a time. The model outputs a probability for each possible output token, and the tooling around the model can sample from that distrib
by sp332 3y ago
They generate one token at a time. The model outputs a probability for each possible output token, and the tooling around the model can sample from that distribution in several ways depending how much "creativity" you want to see in the output. Then that new token is added as part of the input, and the model is run again. That's why it can take tenths of a second to generate each token even on hardware with teraflops of processing power.
- nothrowaways 3y agoBut how is the length of autoregression be determined?
- onesphere 3y agoIf each token leads to another, it eventually leads to what commonly comes out as an ‘end token’. So each token has a certain chance of leading to a next token which might, possibly, say to end. current_token = <prompt> while current_token == true: next_token = statistically[current_token][choose_random_from(common_order)] current_token = next_token s = inf for i in common_order: p = 1 x = i while true: x = statistically[x][common_order[x]] if x == false: break p *= x if p < s: s = p shortest_to_end_start_token = i
- onesphere 3y agoOr: s = - inf ... if p > s: s = p longest_to_end_start_token = i And then maybe you could sort all tokens by their length (if you wanted to find a length of sub-x).