5 ms·
It's not that shocking, each token outputted is a fixed 'unit' of computation and so there will necessarily be things which can only be handled if there is more
by suddenlybananas 3y ago
It's not that shocking, each token outputted is a fixed 'unit' of computation and so there will necessarily be things which can only be handled if there is more computation. A pause token is one way of allowing the model to spend more time to "consider" the answer.
- explaininjs 3y agoThere seems to already exist a speed up while emitting common strings and a noticeable pause when the next token is more ambiguous. I had attributed it to the model being able to short circuit execution when all probabilities were accumulating in the same bucket.
- Tiberium 3y agoIf you're talking inference speed, it's probably just because simple words are a single token, but complex words or code are much more tokens so it'll output them "slower" (while in reality each token will take the same time)
- explaininjs 3y agoNo, there are significant differences in the per-token time that seem directly correlated with “perplexity” (not necessarily in the exact definition, but so to speak).
- explaininjs 3y agoWho the heck downvoted this. I’ve literally collected this data myself from my production AI applications. But it doesn’t gel with people’s preconceived notions so they downvote ?
- zaptrem 3y agoThis might be a side effect of speculative decoding.
- astrange 3y agoModels themselves are a fixed amount of computation, but they could be doing more advanced decoding that would run through it multiple times to try to get more truthful answers.
- SergeAx 3y agoModel does not "consider" answer, it inferes it from training data. If there are "pause" tokens in trainig data, they will be present in inference data with similar density. In other words, "garbage in - garbage out".