4 ms·
This idea seems to track with the notion we saw early on in the use of ChatGPT where folks suggested that answers with more 'filler' language seemed to produce
by ZeroCool2u 3y ago
This idea seems to track with the notion we saw early on in the use of ChatGPT where folks suggested that answers with more 'filler' language seemed to produce higher quality responses. The 'pause' token seems like a better way to do this, but it's still pretty bizarre how well it works.
- suddenlybananas 3y agoIt's not that shocking, each token outputted is a fixed 'unit' of computation and so there will necessarily be things which can only be handled if there is more computation. A pause token is one way of allowing the model to spend more time to "consider" the answer.
- explaininjs 3y agoThere seems to already exist a speed up while emitting common strings and a noticeable pause when the next token is more ambiguous. I had attributed it to the model being able to short circuit execution when all probabilities were accumulating in the same bucket.
- Tiberium 3y agoIf you're talking inference speed, it's probably just because simple words are a single token, but complex words or code are much more tokens so it'll output them "slower" (while in reality each token will take the same time)
- explaininjs 3y agoNo, there are significant differences in the per-token time that seem directly correlated with “perplexity” (not necessarily in the exact definition, but so to speak).
- explaininjs 3y agoWho the heck downvoted this. I’ve literally collected this data myself from my production AI applications. But it doesn’t gel with people’s preconceived notions so they downvote ?
- zaptrem 3y agoThis might be a side effect of speculative decoding.
- astrange 3y agoModels themselves are a fixed amount of computation, but they could be doing more advanced decoding that would run through it multiple times to try to get more truthful answers.
- SergeAx 3y agoModel does not "consider" answer, it inferes it from training data. If there are "pause" tokens in trainig data, they will be present in inference data with similar density. In other words, "garbage in - garbage out".
- airstrike 3y agoI've gotten good results from asking ChatGPT to "check its own work", although on occasion that seems to throw it into a loop and they can't ever get out of it if it doesn't get the right answer, particularly with quantitative answers that it can sanity check itself
- circuit10 3y agoOr ask it to work step by step
- kridsdale3 3y agoThis prompt was a key step forward in the field of human pedagogy as well.
- SergeAx 3y agoModel doesn't "know" "the right" answer. It is statistically guessing, like a bad student with good memory.
- airstrike 3y agoRight, but through sufficient statistical guessing, a capacity to solve problems emerges from its ability to use language, which functions well enough as a proxy for some values of "reasoning", for some set of problems, with varying degrees of accuracy.