9 ms·
Better and Faster Large Language Models via Multi-Token Prediction
- Havoc 2y agoHow does that still end up making grammatical sense? If token/word +1 and +2 are predicted independently then surely often it won’t ?
- deleted 2y ago[deleted]
- vletal 2y agoThe n+1-th token is discarded if it is unlikely given the n-th token.
- deleted 2y ago[deleted]
- wongarsu 2y agoThey just throw the predictions for +1 and +2 away, and only generate them for more efficient training. The abstract doesn't make that clear, but from the description of figure 1: "During inference, we employ only the next-token output head. Optionally, the other three heads may be used to speed-up inference time" Maybe you can use all three heads if you take the top prediction from all of them, but that prevents you from doing any of the common sampling strategies. I'm not sure how many people actually run an LLM with temperature 0 outside of benchmarks, unless they do something even better than applying a temperature
- throw310822 2y agoApologies in advance for the super naive question; but assuming that we can create vectors that encode for the meaning of entire sentences, what prevents us from training llms to predict those vectors instead of single words?
- deleted 2y ago[deleted]
- yorwba 2y agoYou still need to convert between words and sentence vectors somehow. You could try using a faster model for that, but I suspect that the output quality will suffer.
- magicalhippo 2y agoLLMs somehow need to do this anyway, implicitly or not. Has anyone tried to do it more explicitly? That is to break out the idea that characters are formed into words, and words into sentences, and a sentence is a sequence of "concepts" for the lack of a better description. So have one NN which takes a sequence of tokens and predicts an moderately-dimensional "word vector", which is fed into another which predicts a high-dimensional "concept vector". Then the "thinking layer" would map a sequence of "concept vectors" to "concept vectors", and then you'd have some layers which does the reverse of the input layers to output tokens which can be printed. Thought being that by splitting it up like this you could swap out the decode and encode layers independently to translate, for example, and so on. Just a shower thought.
- flawsofar 2y agoA sentence is both a sequence and a tree of phrases. Phrases have a head and one or more valents; they’re relations. If you wanted to create an embedding algorithm for phrases, you could and you could throw a transformer at it. I don’t know how you get the output of higher levels to diffuse to phrase and word levels.
- marcyb5st 2y ago
- bradley13 2y agoI read an article that pointed out that LLMs literally have a one-dimensional window onto the world. Everything is just a sequence of tokens. Maybe this sort of multi more fiction takes their view into 1.1 dimensions? In any gas, there us s real argument for expanding that window, somehow, into two or more dimensions.
- mike_hearn 2y agoWell, it feels like architecturally there's a lot of scope to do better for coding tasks specifically. Like, if you had FAIR level resources and wanted to train a really great Java coding model for example it would make sense to train the model to predict ASTs rather than tokens. You'd still need some kind of joint normal LLM for predicting comments, identifier names and so on, but you wouldn't model the program itself as a stream of tokens. Instead it would predict things like "add an if block", "add a method call block with 4 parameters" and so on. You could also train the model to expect certain context window positions to be reserved for things like "type members at the current cursor" and then integrate the inferencing loop with IDE/LSP-style static analysis. This would allow the model to see more information than is actually contained in the text. I think the reason we're not seeing models like this right now is the cost of doing such research combined with the fact that AI people are all Python-heads, and Python doesn't benefit from much IDEs.
- bradley13 2y agoThat sounds right. My vague idea of a "second dimension" could well be some sort of structure - be it ASTs for programming languages or for natural language. Another possibility would be some sort of fixed knowledge base, which could be program language documentation or "common sense" like CYC wants to provide.
- albertzeyer 2y agoFor those who know speculative decoding: This is basically self-speculative decoding. It still auto-regressively feeds the predicted label sequence through the network again, and only keeps the prediction up to the point where it matches. So it will not get worse in performance but only faster (here up to 3 times, which is normal for speculative decoding). Due to the multi-task training, it will however also get better. (This idea is already quite old, to predict multiple targets into the future as an auxiliary loss.) Nice work.
- imtringued 2y agoThe problem with speculative decoding is that there are hardly any models that support it and adding support takes extra GPU time. If speculative decoding also improves planning performance, then it will be more readily adopted.
- albertzeyer 2y agoWhat do you mean? Speculative decoding can be done with any auto-regressive model. Normally you use another much faster model to predict the next N subwords, and then you use the big model to verify whether it gets the same output, or maybe just reranked. Evaluating N subwords in one go is much faster compared to doing it subword by subword. That's why this is faster. Not all N words might match, so then you might need to redo the prediction for M < N subwords, but there are many simple cases where a faster and weaker model is still accurate enough. In the very extreme case, where N-1 subwords are always wrongly predicted, it would be slightly slower, but usually you get quite a big speedup, e.g. 3x faster or so. The nice thing here is that you actually don't need another smaller model but the model itself already predicts the next N subwords. Or maybe you mean it's not implemented in some of the common software? I'm not sure about that, but I thought it's a quite popular feature now.
- HanClinto 2y agoFor anyone interested in exploring this, llama.cpp has an example implementation here: https://github.com/ggerganov/llama.cpp/tree/master/examples/speculative https://github.com/ggerganov/llama.cpp/tree/master/examples/...
- deleted 2y ago[deleted]
- mg 2y agoCurrently, LLMs start from scratch for each output token, right? Lets say you ask an LLM What makes bananas yellow? And it replies Bananas are yellow due to a pigment called bromelain. I would think that the concept of "pigment" and "bromelain" are already somehow activated in the neural net when it outputs "a". Because now it can't change its mind anymore and follow up with "an optical illusion that makes humans perceive every bent object as yellow". So it seems to have already planned ahead to talk about the pigment called bromelain. Would it be possible to capitalize on the work that has already been done when the LLM outputs "a"? Could the state of the neural net be somehow preserved for the next answer?
- pbh101 2y agoThe alternative theory is that any word starting with a vowel sound is exceedingly uncommon after ’a’ in its training set, so it doesn’t need to plan ahead, just predict the distribution of the most likely next words and choose. Which is my understanding of how they work and the dynamic at play.
- faabian 2y agoTo some degree, attention is already a mechanism to make computations from previous tokens useful later. (You can think of the KV cache as a representation of the text so far and all the models thoughts on it.) And since language models are trained on sequences end-to-end, I think this is likely to happen. Multi-token prediction encourages this behavior explicitly but only for the small n token window you define. That said, there are many works attempting to increase the compute utilization of transformer language models (early exit, mixture of depths) and novel architectures (SSMs etc.).
- jacobsimon 2y agoThanks for highlighting the KV cache, I’ve been wondering the same thing and hadn’t come across that or didn’t remember.
- edmara 2y agoTransformers are still stateless, KV cache is just a compute-saving measure (but otherwise correctly described)
- hhcoder 2y agoI am curious what happens if the multiple tokens predicted interfere with one another. Say I ask "What are the colors of the rainbow?", if one of the tokens is a repeated color, how do we resolve that?
- bjornsing 2y agoI’ve been thinking about this, but I’m leaning more towards letting the LLM output a small PixelCNN or similar model over the next N tokens. That way the LLM can describe conditional probabilities over the coming tokens.
- deskamess 2y agoSide track: There is so much going on this space. I wish there was a chronological flow of a machine learning scenario/story with all the terms being introduced as we meet them (data, pre-training, training, inference, mixture of experts, RAG). Like someone walking me through a factory explaining what happens at each stage (like Mr Rogers used to do). Most of the time I do not know where the terms fit in the big picture. When I first came across pre-training I thought it was something done to the data before training happened but it was actually another training.
- blennon 2y agoI highly recommend Harmonious.ai written by the folks at AI2 Incubator who are brilliant and reputable.
- berkes 2y ago> Most of the time I do not know where the terms fit in the big picture. Nor do the majority of "AI" experts and consultants that I see on LinkedIn, Twitter or in podcasts. The S/N ratio is very low in this field. Just pick some documentation from "industry leaders" like Langchain and see that not only is it already and always outdated, it sometimes simply contradicts itself. In the "blockchain hype" this was similar, so I guess it's a trait of the hype train.
- highwaylights 2y agoTotally agree with the above, although I’m not sure that documentation on tools like Langchain is a reflection of the hype in the way social media is. I think in that case it’s just a reflection of the pace things are moving at.
- pixl97 2y ago>Nor do the majority of "AI" experts... I mean yes, this is what a rapidly expanding field looks like that's probing the boundaries of its problem space. Kind of like following physics in the early to mid 1900s. Different classes of problems have barely been tested against each other, much less fully explored themselves. In some ways it reminds me of the earlier days of the internet when progress was still very rapid.
- nicklecompte 2y agoI haven't read the paper in full detail yet, but I do have a minor editorial comment: while the appendix L.2 was satisfactory, I thought the condensed argument in 5.2 was a bit too sloppy. In particular, H(X) + H(Y) = H(X | Y) + 2I(X ; Y) + H(Y | X) By discarding H(Y | X) - which appears again when predicting at the following position - we observe that 2-token prediction increases the importance of I(X ; Y) by a factor of 2. The argument about "discarding" was not clear to me - if you're predicting the third token Z, then shouldn't H(Y | X) be contained in the implicit context C, and therefore can't be freely discarded? I don't think this argument was clarified in the appendix. But this is mostly about presentation, I wasn't so confused as to doubt the gist of the argument.
- faabian 2y agoThanks for the feedback! Let me try to state it better: In the end, we only use the next-token head for generating. So which parts of the 2-token target H(X) + H(Y) are "auxiliary" in the sense that they help learning and which are "wasted"? H(X | Y) and I(X; Y) are useful for next-token generation while, by definition, H(Y | X) is the information quantity not related to the next token X. So we could say: "multi-token prediction trades the useful information I(X; Y) from H(Y) for the wasted computations on H(Y | X)". However, note that H(Y | X) is a next-token entropy for predicting Y from the prefix (C, X). If the attention mechanism allows to transfer computations already made for predicting Y|X to the next step, these computations may actually not have been wasted -- it was just pre-computations.
- stealthcat 2y agoDid you have some small toy experiments to prove this?
- Xcelerate 2y agoDo LLMs not consider the probability distribution over all combinations of tokens up to a certain output length with regard to sequence prediction? I assumed they did that already. If they don’t, I’m amazed they work as well as they do. Consider 2-bit sequence prediction with the following possible outcomes and associated probabilities: 00: p=0.36 01: p=0.04 10: p=0.30 11: p=0.30 So the most likely 2-bit sequence is 00. But on the basis of predicting the next token (bit) alone, we have: 0: p=0.40 1: p=0.60 which suggests that 1 is the next bit and leads to a suboptimal starting point for predicting the bit after that. The error is even more prominent with longer sequences as the joint probability distribution becomes more unfactorizable into marginal distributions (as I would expect any minimal algorithmic description of real-world data to be). Edit: now that I think about this a bit more, a cool research project that would be really simple to carry out might be to modify the cross-entropy loss function to consider only the nth future token in the text training data, and then plot LLM performance vs n, assuming that for all current LLM models we just have n=1. My hypothesis is that you can mostly bypass all of the resource blow-up involved in predicting the joint probability distribution over the next 1 through n tokens (which scales as x^n) by just predicting the nth token directly, since doing so would implicitly require a better data model (at least for human-generated text; this wouldn’t be the case for all types of data).
- henderson98 2y ago....uhh Isnt this what Beam search attempts to fix(approximate) over greedy decoding. You know greedy optimum vs global optimum. And most LM already use beam search. Because finding the true optimum is intractable even over modest lengths.
- Xcelerate 2y agoSure, beam search makes the result less greedy than it originally was, but it’s still an extremely greedy approach overall. It would be sort of like trying to make local optimization techniques less local by running the process multiple times from different starting points and choosing the minimal basin the process ended up in from out of the different runs. Quite a bit better, but in many cases not even close to the global optimum*. For a slightly more apt analogy, it would be like: 1) Choose multiple starting points 2) For each starting point, perform local optimization until you get stuck in a basin (local minimum) 3) Keep the n overall lowest points, create m different perturbations for each point that pushes the point out of its basin, and go to 1), using these m*n points as your new set of starting points for the next round. Note this process is totally agnostic to whatever the local optimization algorithm is. That’s why I called the beam search part of an LLM’s post-training prediction a “meta-model”, because it doesn’t matter if the core inference is performed by a transformer architecture or something entirely different. *I say “in many cases”. But I am extremely curious for this particular case of inferring sequences of human-generated text the degree to which we fail to capture the true joint probability distribution via single-token prediction + beam search. It’s quite possible we are very far off—perhaps this is the missing “ingredient” for generalized reasoning. On the other hand, as I said in my original post, I never would have guessed current SOTA LLMs only use single-token prediction, and I’m astonished they work as well as they do based on that, so maybe we’re not actually too far off. Without further research, it’s just speculation either way though.
- ralusek 2y agoGiven that LLMs appear to, in large part, "think" by virtue of feeding its own input into itself, people have consistently noticed that insisting that the model "think out loud" results in higher quality reasoning. i.e. "chain of thought" reasoning will contrast simply having the model answer a question directly with first having it write out things like: - restating what it thinks is being asked of it - expressing a high level strategy over what sort of information it might need in order to answer that question - stating the information it knows - describing how that information might inform its initial reasoning etc... I'd be concerned that going about this by having the model predict the next multiple tokens at any given time would essentially have the opposite effect. Chain of thought prompting appears to indicate that a model is "smarter" when it has n + m tokens than when it just has n tokens as input. As such, getting the next 5 tokens for a given n might net worse results than getting the next 1 token at n, then the next 1 token at n + 1, and so on.
- imtringued 2y agoIf the LLM had an affordable model it would always generate enough tokens for the task at hand. The fact that this particular method would require more tokens would be irrelevant. If you don't have an affordable model, then you would always be at the mercy of the LLM being biased towards answering with an estimate instead of the actual answer. Also, most speculative decoding strategies produce identical output compared to running the model sequentially. If the prediction is wrong, the token gets discarded and the speedup is lost.
- bravura 2y agoI wonder if, instead of just predicting the next n tokens, it could also predict like 128, 512, 2048 etc tokens ahead. Thus learning long-term discourse structure.
- HanClinto 2y agoMight be good to have some flexibility in where those particular tokens are placed, but yeah -- I could see value in creating a "pool" of tokens that should be used at some point in the future in the answer.
- lucidrains 2y agowow, so prophet net does work! i spent so much time experimenting with it back in the day, but just lacked the scale to see a positive result.
- riku_iki 2y agoIts interesting that they got good results on 200B and 0.8 epoch training set, but once scaled it to 1T and 4 epoch, got degradation in vast majority of benchmarks (Table 1).
- jmount 2y agoAfter inventing multi-token, one then invents a useful language oriented hierarchy (such as sections, paragraphs, sentences, and words).
- WhitneyLand 2y agoThe use of the word “head” in machine learning does not seem consistent, in case anyone else is confused by that. There’s multihead attention and multiple output heads as a concept in the paper. Multihead attention is about focusing on different areas of the input in transformer architectures, and the biological analogy here is head as a central processing unit. An output head refers to the final layer of a neural network, of which you could have more than one producing different outputs based on the same previous layers. This is also a loose biological analogy, but instead of head as cpu, think more along the lines of head being on one end of the body. In neither case is there any analogy to a tape head that reads data.