5 ms·
Training language models with pause tokens
- odyssey7 3y agoWoah, very cool idea.
- deleted 3y ago[deleted]
- ZeroCool2u 3y agoThis idea seems to track with the notion we saw early on in the use of ChatGPT where folks suggested that answers with more 'filler' language seemed to produce higher quality responses. The 'pause' token seems like a better way to do this, but it's still pretty bizarre how well it works.
- suddenlybananas 3y agoIt's not that shocking, each token outputted is a fixed 'unit' of computation and so there will necessarily be things which can only be handled if there is more computation. A pause token is one way of allowing the model to spend more time to "consider" the answer.
- explaininjs 3y agoThere seems to already exist a speed up while emitting common strings and a noticeable pause when the next token is more ambiguous. I had attributed it to the model being able to short circuit execution when all probabilities were accumulating in the same bucket.
- Tiberium 3y agoIf you're talking inference speed, it's probably just because simple words are a single token, but complex words or code are much more tokens so it'll output them "slower" (while in reality each token will take the same time)
- explaininjs 3y agoNo, there are significant differences in the per-token time that seem directly correlated with “perplexity” (not necessarily in the exact definition, but so to speak).
- explaininjs 3y agoWho the heck downvoted this. I’ve literally collected this data myself from my production AI applications. But it doesn’t gel with people’s preconceived notions so they downvote ?
- zaptrem 3y agoThis might be a side effect of speculative decoding.
- astrange 3y agoModels themselves are a fixed amount of computation, but they could be doing more advanced decoding that would run through it multiple times to try to get more truthful answers.
- SergeAx 3y agoModel does not "consider" answer, it inferes it from training data. If there are "pause" tokens in trainig data, they will be present in inference data with similar density. In other words, "garbage in - garbage out".
- airstrike 3y agoI've gotten good results from asking ChatGPT to "check its own work", although on occasion that seems to throw it into a loop and they can't ever get out of it if it doesn't get the right answer, particularly with quantitative answers that it can sanity check itself
- circuit10 3y agoOr ask it to work step by step
- kridsdale3 3y agoThis prompt was a key step forward in the field of human pedagogy as well.
- SergeAx 3y agoModel doesn't "know" "the right" answer. It is statistically guessing, like a bad student with good memory.
- airstrike 3y agoRight, but through sufficient statistical guessing, a capacity to solve problems emerges from its ability to use language, which functions well enough as a proxy for some values of "reasoning", for some set of problems, with varying degrees of accuracy.
- famouswaffles 3y agoThis seems somewhat related to https://arxiv.org/abs/2309.16588 https://arxiv.org/abs/2309.16588. Some Researchers found that Vision Transformers would repurpose low-informative background patches of images for internal computations. They gave them an explicit token for this to better results!
- HammadB 3y agoYeah, was thinking the same thing. Something deeper to be explored here esp due to the consistency across domains. I wonder how far we can push "externalizing" higher level information the model wants to store while in the forward pass.
- mkaic 3y agoYou beat me to it, I was just about to post this here haha. It's fascinating to me that these models can take advantage of "scratch paper" this way!
- deleted 3y ago[deleted]
- bradfox2 3y agoThis likely has a tangent to quantization research from Dettmers et al that corresponds to outlier weight values in models >6B params. Might allude to why the outliers emerge.
- toxik 3y agoThis isn’t exactly news. The question is if it’s better use of FLOPs than just having a larger network or embedding.
- famouswaffles 3y agoEach token outputted is a fixed 'unit' of computation. There will always be predictions that could use more computation and therefore need more tokens, regardless of how large the model gets. It just makes sense to add this option.
- toxik 3y agoI think I don’t understand what you are trying to say. Yes, FLOPs per token is the number we care about. Adding ~10x FLOPs and saying “look it’s better” seems odd. What’s worse, memory use is quadratic in sequence length so adding 10x tokens is just not very smart.
- deleted 3y ago[deleted]
- deleted 3y ago[deleted]
- famouswaffles 3y ago>I think I don’t understand what you are trying to say. Yes, FLOPs per token is the number we care about. Adding ~10x FLOPs and saying “look it’s better” seems odd. You don't have to use 10. That's just what they found optimal for a particular benchmark. For others it was lower. >What’s worse, memory use is quadratic in sequence length so adding 10x tokens is just not very smart. Everyone uses flash attention these days. Memory is no longer quadratic in scaling.
- toxik 3y agoYou still haven’t answered my one question: how is this different than just increasing embedding size? I can even think of a sketch proof to show this equivalence for single heads. The paper should have answered this question and well. It is an obvious question. WRT FlashAttention, surely you understand that the matrix-matrix multiplication must still happen even if it’s a fused kernel that doesn’t need to hold several intermediary variables in memory.
- kaibee 3y agoIt would be really funny if the whole "Go step by step and think it through" trick only worked because it was adding more tokens for the AI to reuse.
- famouswaffles 3y agoI'm sure that's part of it but also in the same way out of distribution tokens can derail its output, certain tokens will guide it. In the paper, the model doesn't really learn to use padding tokens effectively till after training so it's the only thing.
- justinzollars 3y agois this your prompt? "Go step by step and think it through".
- joshka 3y agoThis seems like a slight variation of "Let’s think step by step" [1] or "Let’s work this out in a step by step way to be sure we have the right answer" [2]. There's a good review of a bunch of these in https://arxiv.org/abs/2305.02897 https://arxiv.org/abs/2305.02897 1: https://arxiv.org/abs/2205.11916 https://arxiv.org/abs/2205.11916 2: https://openreview.net/forum?id=92gvk82DE- https://openreview.net/forum?id=92gvk82DE-
- danbruc 3y agoIt does. Those models are glorified look-up tables, output = answers[input]. It would of course be insane to try to store this mapping explicitly but it is still there implicitly in the weight of the network. But no matter what, it is just feed-forward, the system will always be limited when it comes to learning things that require iteration. This can be overcome to some extend by doing something like chain of thought that allows dumping some state into the output stream that will then be fed back through the autoregressive loop wrapping the network. Pause tokens are more or less the same thing just that they do not pollute the output with the internal thoughts of the network.
- 3y ago
- Nevermark 3y agoI propose four training stages: 1. Train without pauses for fast response 2. Train with variable numbers of inserted pauses, so the model can take more time (computation) to improve its performance. Where pauses are represented both by pause tokens, and a new pause countdown element. I.e: Sequence: …, token, pause, pause, pause, token, … Pause element: …, 0, 3, 2, 1, 0, … 3. Train the model model to generate its own next pause value (insert its own pauses), to optimize a meta-performance tradeoff function that ranges from the regular performance measure (independent of delay) vs. delay. The tradeoff is based on an “urgency” value between 0 and 1, that adjusts the tradeoff meta-performance weighting between just delay-independent accuracy vs. just speed. The urgency is used at each step to calculate the tradeoff meta-performance, and as a new urgency input element so the model knows what tradeoff level it is supposed to optimize. 4. Train the model to generate its own urgency values, based on examples of prompts which either communicate urgency based on request or context. “Quick, give me an estimated value for …” “The reactor is about to explode, what are the shutdown codes?” Note, that the model generated urgency value and the next urgency input don’t have to be the same. The generated urgency could be pushed higher or lower algorithmically, to get the next urgency input, if the model is being used in a context that has its own urgency relevant information. —- So many good ideas. It’s clear models are going to improve quickly. Next step beyond a model that can leverage pauses for better performance, and a model that controls its own pauses, is a model that can be interrupted. Interruptions could be urgency related, cancellations, modifications, or irrelevant: “Hurry up” “Forget that, do this other thing.” “Oh, and the answer should be odd.” “While you think, I am enjoying a nice cup of tea.”
- silveraxe93 3y agoDamn that's cool. Can we implement a version of System 1 / System 2 [1] thinking with this? From the future work section: > better determining the number of <pause> tokens (perhaps using model confidence) If the number of pause are learned, using model confidence + regularisation (so that the model doesn't always use the maximum number of pauses), then we effectively have the System 1/2 switch. If it's a task the model has seen tons of times before, it just goes with the first inference pass. If it's low confidence, then it keeps expending more inference passes until it reaches an acceptable confidence threshold. - [1] https://en.wikipedia.org/wiki/Thinking,_Fast_and_Slow https://en.wikipedia.org/wiki/Thinking,_Fast_and_Slow
- striking 3y agoWould that be in an effort to more accurately model human thinking? If so, I hate to be the one to tell you that > Readers of “Thinking: Fast and Slow” should read the book as a subjective account by an eminent psychologists, rather than an objective summary of scientific evidence https://replicationindex.com/2020/12/30/a-meta-scientific-perspective-on-thinking-fast-and-slow/ https://replicationindex.com/2020/12/30/a-meta-scientific-pe...
- silveraxe93 3y agoI actually read that one before! Kahneman is overconfident, as we all were before the replication crisis. While most of the psychology field is crumbling and filled with bullshit (even though most people didn't catch on yet). There's still _some_ truth lying in there. It might look bad in isolation, but Kahneman's work is one of the few that actually holds up to scrutiny. This post by Scott Alexander is pretty good in summarising the few good bits left. https://www.astralcodexten.com/p/heres-why-automaticity-is-real-actually https://www.astralcodexten.com/p/heres-why-automaticity-is-r... --- But regarding the initial point. The goal is not to fully emulate human thinking. It might sound wishy-washy, but there's _obviously_ _some_ truth to the system 1/2 model. It's not perfect! But I think it's useful. We as humans do most decisions without thinking (hard), but we have a way to _switch_ into a more reliable but expensive mode. We see some evidence that artificially inducing LLMs to 'think' more improves their output. So it stands to reason that adding this capability to a model would make it better. And what better way to do it than swallowing the bitter pill and have that decision be made by the model itself, on a case-by-case basis by learning it from data.
- iNic 3y agoReminds me of PonderNet [1], which dynamically changed how often the network did a calculation. [1]: https://arxiv.org/abs/2107.05407 https://arxiv.org/abs/2107.05407
- olalonde 3y agoI wonder if we will eventually end up with "internal monologue" tokens that are not meant to be part of the output but just perform computation, a bit the way humans think.
- stavros 3y agoNot all humans think like that, though.
- optimalsolver 3y agoI find that hard to believe.
- olalonde 3y agoSo do I. I think it has more to do with people choosing different words and metaphors to describe their subjective experience.
- stavros 3y agoI don't have an internal monologue.
- Loeffelmann 3y agoWhat do you imagine a inner monologue to be like?
- stavros 3y agoA voice in my head that narrates thoughts.
- wilg 3y agoAre your thoughts or ideas ever in language before you say or write them? I think this is the same misunderstanding as “the mind’s eye” https://x.com/stuartjritchie/status/1708871999924179024?s=46 https://x.com/stuartjritchie/status/1708871999924179024?s=46
- PartiallyTyped 3y agoSeems related to “infinite context” by adding attention sinks. This seems to be the same mechanism at play. https://arxiv.org/abs/2309.17453.pdf https://arxiv.org/abs/2309.17453.pdf
- arxiv_papers 3y ago[dead]
- intalentive 3y agoAn attention layer is a kind of meta neural network whose weights depend on input. If you think of each token as a neuron and the attention matrix as defining the weights, then extra tokens simply mean a wider network.
- SergeAx 3y agoPause have no meaningful value for language models. If pause tokens are produced from speech analysis, LLMs will just become better at imitating speaking, not smarter.