23 ms·
Attention Is Off By One
- deleted 3y ago[deleted]
- sebzim4500 3y agoDon't transformers typically have a <bot> token at the beginning of the prompt? This seems equivalent to letting the network attend to this token, and produce a zero value if that's what it wants.
- sp332 3y agoChat-tuned ones do, but the base models don't. For example, Llama doesn't, but Alpaca has "### Instruction:", "### Input:", and "### Response:".
- refulgentis 3y agonot a token, and not the transformers, but yes, commercial chat models are fine-tuned on text transcripts containing dialogues. (i believe llama-2 was as well)
- sebzim4500 3y agoAre you sure? I have never seen an LLM that did not have a special token for start of text, I'm certain that llama had one and I don't remember anywhere in the llama-2 paper where they said they removed it.
- refulgentis 3y agotl;dr: you're right it's messy though, bear with me for the full explanation: - your initial post says "<bot>" token, which looked like a mix of "chatbot" and ChatML, used by OpenAI - there is a bo_S_ token, which acts as you described - I averaged my attention over your post and the initial reply, which answers as if you were using "<bot>" in the misunderstood way - when I go back and read your post, I realize the chatbot interpretation doesn't quite make sense, since you're referring to much more technical aspects than general "how do I AI", i.e. you understand <X> as a way to denote special tokens, not necessarily an XML tag
- a1k0n 3y agoYes, it has to in fact. If you have zero context to attend to in a transformer, and you try to predict the first token, you effectively are multiplying a zero-vector by the attention head, making all tokens equally likely in the final softmax (unless the lm_head has a bias, but at least in GPT it does not). So the <|beginning of text|> token, with no context before it, learns to predict the first-token-in-a-document distribution. That's not quite the same as predicting nothing at all.
- llwu 3y agoThis is a cool idea - it could be tested it by seeing whether W_V tends to map <bot> to 0, and whether <bot> tends to get attended to. There are interesting things to said whether or not this turns out to be the case.
- firebirdn99 3y agoThis is right below the "Have Attention Spans Been Declining? – Yes, 65%" post, lol brilliant. In general, human decreasing, AI increasing- attention.
- gwern 3y ago"In this post, I prove that attention spans have actually declined by 64%, contrary to widely-publicized reports of 65%..."
- neilv 3y agoFor posterity: 1. Have attention spans been declining? (slimemoldtimemold.com) 338 points by janandonly 4 hours ago | flag | hide | 254 comments 2. Attention Is Off By One (evanmiller.org) 400 points by elbasti 4 hours ago | flag | hide | 129 comments Note that the #1 post is probably there because the title earlier had the provacative "Yes, 65%" appended to it. So even more numerical.
- simbolit 3y agoI don't understand this well enough to say if it is correct, but I do understand it well enough to say it is important if correct.
- gremlinsinc 3y agoI don't know half of you half as well as I should like; and I like less than half of you half as well as you deserve
- simbolit 3y agoI don't understand your comment, but I gather that you and others didn't like mine, which is noted, I will try to do better.
- the_af 3y agoThe comment you're replying to is made by Bilbo Baggins during his birthday, near the beginning of "The Lord of the Rings". As to what the commenter above meant I can only guess, but it should be noted that Bilbo's audience reacts with puzzlement, unable to parse his words.
- obiefernandez 3y agoThis seems very important if accurate.
- chessgecko 3y agoI ran an experiment like this and in my setting it didn't help. Not saying there may not have been a bug or something, but I think attending over the current position sort of solves this problem. IE when it should not speak it just emits the current pos value. edit to add details in case anyone is interested I didn't add one to the softmax denom. I added a learned parameter (the attention sink) that would be appended to the beginning of QK but would be removed after softmax, so when multiplying by V the totals wouldn't sum to one. I tried variants that included looking at the current pos and not, and also variants that predicted used an ffn to generate the sink per position instead of a learned param. In my setting neither approach really made much of a difference. But I also had a bunch of other weird stuff in there too, so it may be worth trying again.
- abeppu 3y agoWhen you say it didn't help, can you clarify what you're measuring? In the context of this post, I think both the performance your task, and the number of outlier weights (and their magnitude) are important.
- chessgecko 3y agoI was just looking at doing this in pretraining, so I was looking at pretraining losses. The difference was within the range of usual noise so I didn't keep trying.
- lucidrains 3y agothis is fixing a different issue, not the one you are measuring.
- chessgecko 3y agoIt wasn't really the goal of my experiment to fix this issue for sure, I was trying to see if you could improve attention by decoupling the key used by a position for itself and for future tokens. Open to being wrong here, but wouldn't it be functionally similar to adding a constant to the softmax denom? the function could sort of learn a specific position to have sink and q multiply to one, then removing it before multipling with v would be exactly identical?
- mlsu 3y agoI don't really understand the subject matter enough, so I apologize in advance for the meta-comment... The author mentions that he would maybe have written this as a scientific paper: > I tried writing a serious-looking research paper about the bug and my proposed fix, but I lost a series of pitched battles against Pytorch and biblatex, so I figured I’d just write a blog post instead. (History is written by the winners; blogs are written by…) Honestly, thank god he didn't. This paper is so much more readable and approachable than what gets published in "serious" journals. The tone is self-effacing, it does not have an "ego" the way scientific papers tend to have. If all science read like this, and if we were "allowed" to cite research that reads like this, I think we would be much better off. This reads like a conversational, approachable textbook, not like an impenetrable wall. Is it because I don't understand attention at a PhD level that I hold this opinion? Maybe. Could he be writing like this because he's a layman and utterly wrong about the topic, unlike those Serious Science Authors? Maybe, I don't know. But my god, wouldn't it be nice to be allowed to write like this?
- chessgecko 3y agoI think maybe its because he didn't have experimental results that show that it worked. Not a knock against the author, there are just so many things that seem like good ideas that don't end up working well in practice, a paper like this without results is hard to value.
- mlsu 3y agoYes, definitely. If he tried to have it published, the lack of experimental results would definitely be a glaring error. But this is still scientific communication. It's really nice that it's legible! > Even though softmax1 is facially quite boring, I’m 99.44% sure that it will resolve the outlier feedback loop that’s making quantization the subject of cascades of research. If you want to run some experiments and prove me right, DM me on Twitter and we’ll get a paper going. I'm guessing that in the stodgy world of science, a communication like this might happen over lunch at a conference, limited to a small clique of researchers who are zealously guarding their next paper. Who could blame them, publish or perish! But someone will probably test this theory out (after my read, it will probably happen in llama.cpp with preliminary results on GPT-2 by next week) and achieve results, and it will happen quickly and legibly to the outside world, because this was published openly and without all of the pretension that formal science (tm) has. If it works, it works. Stuff like this is the soul of the internet. Sharing knowledge and making it legible for all.
- sunleash 3y agoI don't see any results, it'd be more impactful and convincing if there were numbers supplementing the theory. It's not that hard to finetune existing LM on a small data and verify that it works. I am, however, of the similar opinion that there could be better attention formulations. A paper from 2020 https://arxiv.org/abs/2005.09561 https://arxiv.org/abs/2005.09561 helped a lot in one of the transformers model I trained (not a vanilla LM but a specialised multi-modal graph problem). It proposes normalised attention which if I'm not wrong should help with the quantisation problem too.
- cs702 3y agoTL;DR: The author proposes that instead of using the Softmax function in each head, Softmax(x_i) = exp(x_i) / sum(exp(x_i)), we should use instead what the author calls the Softmax_1 function, Softmax_1(x_i) = exp(x_i) / (1 + sum(exp(x_i))), which would make it possible for each transformer head's attention probabilities to be zero, i.e., attend to nothing, by computing x_i's with values well below zero. Giving each transformer head the ability to ignore all tokens surely can't hurt, but it remains to be seen if it will actually improve transformer performance.
- rrobukef 3y agoI also saw the author distinguished internal versus output softmax. I think he'd apply his modification only to internal softmax and let the external force an output.
- cs702 3y agoYes, it makes sense to apply this only to the Softmax we use to compute attention. It makes no sense to apply it to the output Softmax, which must compute a probability distribution over the vocabulary.
- mcbuilder 3y agoActivation sparsity and packing sparse matrices will surely be important, so there is one kind of performance. However the other, perplexity, needs a good demonstration. It might require a big model, but even 30B you can fine tune on nowadays on a big Cloud GPU box.
- politician 3y agoThis makes sense. One tweak for the press: I think it would be an improvement to call it OptionalAttention rather than QuietAttention since the goal is to permit an attention head to opt-out. You might attract more, ahem, attention if it was immediately apparent from the name only what this attention head does that the current one does not. There's also that small matter of distinguishing the internal vs output softmax functions.
- janalsncm 3y agoI follow the argument but the proof of the pudding is in the eating. I don’t know what “battles” the author lost to PyTorch lately but a good test would be to modify one of the smaller models (maybe nanogpt) and swap out all of the softmax calls for his quiet softmax. I didn’t see anything relevant on alternatives to softmax, since TFA is specifically questioning softmax in a multihead attention context. Ultimately, neural networks are arbitrary function approximators. It doesn’t necessarily have to be “right” internally to fit the data. But if this new softmax allows transformers to learn more, that’s great.
- LoganDark 3y ago> a good test would be to modify one of the smaller models (maybe nanogpt) and swap out all of the softmax calls for his quiet softmax. You'd have to train the model with the quiet softmax before inferencing with it would work.
- leecarraher 3y agoThe author says to add a unity vector to the context, i presume of each layer, to not mess with gradient calculations. But most modern DL frameworks compute the gradient for you, (i know this is true for JAX and Pytorch). Is it maybe that hand coded gradient for a well-known enough dl architecture like transformer is faster than letting the framework autodiff it? Otherwise, i fear some of the 'magic' of transformer networks is that this amplification effect allows it to encode/memorize some results verbatim. And we often are seeing a heavily tuned internet regurgitator. So similar to the rise of RNNs with attention, which supposedly allowed them to focus on some things and ignore others but really often was just overfitting stuff, yielded more interesting results with the overfitting than without.
- bearzoo 3y agothink the author is talking about having to fix the extra vector in V to be zeros and making sure to not compute/apply gradients to it
- bertil 3y agoIf you hesitate to read it, let me say that the post denounces “kurtotic barbarities.” If that expression alone doesn’t convince you to read it, you might not be in the intended audience.
- make3 3y agoPlease, put empirical numbers with proposals like these. Transformers have had a billion "improvements" suggested to them through the years.
- sbszllr 3y agoOP is right in that his change would make the softmax in the attention output zero if it "has nothing to add" (QuietAttention, as he said). Buuut, it's missing the forest for the trees. The goal of the last step of attention (ref., Fig. 2, left in https://arxiv.org/abs/1706.03762 https://arxiv.org/abs/1706.03762) is not to add/say anything (as the author is saying) but to compute the relationship between the tokens (QK^T) and V -- in layman terms, simplifying, which tokens are related to each other. The softmax is there because it gives a representation that is nicer to work with, it gives probabilities, instead of unscaled matrix multiplication. TLDR; author isn't wrong but he isn't right, practically speaking, either.
- lostmsu 3y agoWhat's wrong with unscaled matrix multiplication? Softmax has some kind of intuition in the context, but why not layer norm or something else instead (if anything is needed at all)?
- sbszllr 3y agoThe family of sigmoid functions has nice gradient properties with theoretical backing. Good starting read: https://stats.stackexchange.com/questions/162988/why-sigmoid-function-instead-of-anything-else/318209#318209 https://stats.stackexchange.com/questions/162988/why-sigmoid...
- lostmsu 3y agoDoesn't really explain why not more typical ReLU + layer norm or some alternative.
- fwlr 3y agoThe author identifies a real problem and poses a simple solution. It passes all my crank tests (why did no one come up with this before? Because the author is intimately familiar with the softmax function from work outside of ML, and plausibly nobody who’s investigating these issues is remotely as familiar, so despite researchers narrowing the issue down to “something to do with softmax”, they don’t have a deep enough understanding of softmax to see what’s wrong). If the author is reading any of these comments, though, I would urge them to expand on their claim that “I’m 99.44% sure that it will resolve the outlier feedback loop”. As it stands, that’s the only explanation we get of how the outliers might be related to softmax!
- Majromax 3y ago> why did no one come up with this before? And because the effects of the problem are subtle. Supposing the diagnosis is correct, full-precision LLMs still avoid the issue through large attention weights given to meaningless tokens to give harmless attention outputs. The problem only matters when quantizing weights, and quantized performance isn't really the goal of recent cutting-edge LLM development.
- Legend2440 3y agoYeah, but it lacks the most important test: results. He hasn't actually tried it, he just thinks it will work. For such a simple change to the softmax it wouldn't take long to verify. It's really embarrassing to not do that before publishing.
- refulgentis 3y agoIt's not embarrassing at all. I think there might be some curse of the auto-didact here, hinging on the meaning of publish: it would be embarrassing if he was capital-P publishing, as in a scientific paper. The blog goes to great lengths to point out it is _not_ capital-P publishing.
- joebiden2 3y agoYou seem to really disregard the positions of this author. They seem to have invested substantial efforts in that specific area of research. To validate the idea the author has, it would be required to train a LLM from zero. If the author is right, you would get similar results to the current generation of LLMs, but with (a lot) less space required for the intermediate layers. The time to achieve that is still measured in kilo- to mega-dollars, why is it wrong to put that idea in the open to substantially criticize or adopt?
- nborwankar 3y agoShouldn’t this be called Regularized SoftMax? Adding 1 in the denominator looks a lot like a regularization in other ML contexts.
- theGnuMe 3y agoIsn't this why you have the <BOS> token, or the <cls> token?
- jmount 3y agoThe "missing 1" is a waste-category that is implicitly re-scaled. The explicit 1 formulation is used in binary softmax, and the implicit (not seen 1) is used in multinomial softmax. I suspect this is the old "notation B looks silly in terms of notation A's standards."
- blueblimp 3y agoThe proposed replacement definitely makes more sense (and I've always found the absence of a "failed query" to be puzzling in standard attention), but, in deep learning, things that make more sense don't always actually get better results. So I'm curious whether this has been tried and carefully evaluated.
- jadbox 3y agoIt would be an amusing find if "Black Swan mega-activations" actually but yet unintentionally made the model smarter...
- gerdusvz 3y agonow if only we could teach humans to also not annotate when they have nothing to add
- whimsicalism 3y ago> their existence is contrary to everything we thought we knew about neural networks prior to building ones that worked so well Why is this true? Because their existence implies some sort of preferred basis that aligns with the dims of the neural network, which is surprising? It's not obvious why their existence is so contrary to what we knew.
- szundi 3y agoIt would be fun when once in the future someone finds a bug like this, merges the PR, and BAM! Singularity.
- Imnimo 3y ago>The problem with using softmax is that it forces each attention head to make an annotation, even if it has no information to add to the output vector. Using softmax to choose among discrete alternatives is great; using it for optional annotation (i.e. as input into addition) is, like, not cool, man. The problem here is exacerbated with multi-head attention, as a specialized head is more likely to want to “pass” than a general-purpose one. These attention heads are needlessly noisy, a deafening democracy where abstention is disallowed. Can't the MLP that processes the concatenated outputs the attention heads handle this? I don't understand why it should be critical that a head be allowed to put something close to zero in its segment of the concatenated vector if it's immediately going to get projected by an MLP anyway.
- marcyb5st 3y agoBut you are wasting some of the model's capacity to learn to ignore some of that information. I think it wouldn't hurt. However, if I followed the reasoning correctly, I think the biggest win is to reduce the range of the weights more than improving performance. > This is what’s been happening in LLMs – for reasons that are only partially understood, Transformer models contain these outlier weights and are emitting Black Swan mega-activations that are much, much, much larger, like orders of magnitude larger, than their peers ... meaning that once quantized you can either have a finer quantization since the range of possible values is smaller or you can pick a coarser strategy that saves bits for each weight.
- Imnimo 3y agoRight, I get the goal of removing the outlier activations, but I just don't understand why outlier activations are a consequence of the model trying to "pass". The story from the linked paper earlier in the post (https://arxiv.org/pdf/2306.12929.pdf https://arxiv.org/pdf/2306.12929.pdf) is that the model is doing the following: -Learn a near-zero representation for some otherwise low-importance token, like delimiters or whitespace. -When a head wants to "pass", emit an outlier activation to attend to that token nearly-exclusively. But I'm surprised the model can't just use its existing tools (the post-concat projection layer and the following MLP block) to achieve the same thing. And if the answer is that it could do that, but tends to learn to use the outlier activation trick instead, will giving it a new tool that still allows the use of outlier activations be sufficient?
- orasis 3y ago“the seemingly innocent exponentiator that no one thought capable of such kurtotic barbarities.” This writing brought a happy tear to my eye.
- alsodumb 3y agoI might be missing something obvious, but I am not sure why everyone in the comments think it's a big deal. I've seen this trick in practice multiple times. For example, see this snippet from an old Google repo: https://github.com/google/flaxformer/blob/ee62754ebe5a5eeb111493622de5537133822e3e/flaxformer/components/attention/dense_attention.py#L50 https://github.com/google/flaxformer/blob/ee62754ebe5a5eeb11...
- PartiallyTyped 3y agoThe argument / reasoning is a bit dubious. Technically softmax is not implemented as presented but through exp(x_i-max(x)), and summing over it in the denom. But maybe I am missing something. Furthermore, the residuals are used exactly because the networks cant learn the identity function; but they can learn zero; at which point the residual is `f(x): x+g(x)` with being `g:x ~> 0` (ie approximately 0). It is also the case that `f(x): x+g(x)` makes it easier for gradients to flow through.
- mrfox321 3y agoYou are misreading things. Regardless of numerical stability tricks (e.g. exp(x_i-max(x))), you are still simply normalizing the logits such that the probabilities sum to 1. The blog adds an additional hidden logit (equal to 0) to allow for softmax(x) = 0 when x -> -inf.
- PartiallyTyped 3y agoHow can `x -> -inf` occur in the first place when nearly everything is within [-2,2] and doing a dot product plus before that there's normalization too?
- uoaei 3y agoThe use of the "nearly" in your comment is exactly occluding the issue as presented. Enough weights don't fall under that "nearly" that we require more bits per weight to cover those edge cases. If we were able to delete the "nearly" we would need fewer bits (smaller models).
- keskival 3y agoI thought everyone knew that softmax (and specifically exp functions in it) are poison. I have always worked around them, for example by using large epsilons (approaching one actually), and using low-order polynomial approximations for the exp functions. I thought everyone does that, because you don't need to work long with these models to get NaNs, and when you check why you see it's because of the exp functions. Then you fix it. Apparently people don't. It's not like the neural models care if you approximate functions. They couldn't care less actually.
- sebzim4500 3y agoIt is pretty easy to avoid NaNs when working with softmax, you certainly don't need any epsilons. Just subtract the largest value from everything, and you will have no rounding problems or catastrophic cancellation. Clearly softmax is not too bad, if it is used extensively in all the most powerful models.
- jrochkind1 3y agoWhile not about AI or the algorithm mentioned, on the subject of little errors that you can't convince anyone are errors.... In 2011, I wanted to copy the reddit ranking algorithm in a project of my own, so I went to source code to look at it... the algorithm in the source code I found wasn't doing anything at all sensible with negative-sum voted posts. I thought I discovered the error, some terms swapped in the simple equation, the sign for positive/negative was misapplied. I blogged it, and [posted it to reddit](https://www.reddit.com/r/programming/comments/td4tz/reddits_actual_story_ranking_algorithm_explained/ https://www.reddit.com/r/programming/comments/td4tz/reddits_...), only to have MANY people, including reddit employees, tell me I am definitely definitely wrong, and the algorithm was working as intended. And that I was in fact not the first to notice what I thought I noticed, and point it out, and be told by everyone I was wrong. OK, I didn't really understand what was going on, I couldn't make sense of the algorithm if it wasn't wrong, but so be it. I updated my blog post to say that people smarter than me said there was no error in the reddit algorithm, all I can say is this variation makes more sense to me. Then, three years later in 2014, a commit was made to the reddit source code with exactly the correction I (and others before me) had suggested all along. The one that everyone piled on to tell me how dare I have the temerity to suggest reddit source code is wrong. https://github.com/reddit-archive/reddit/commit/50d35de04b928836b7ee955c8a26f197e24ab01e#diff-d85a4e2bae6e58669811622c0fc7f618 https://github.com/reddit-archive/reddit/commit/50d35de04b92... ¯\_(ツ)_/¯ Open source means there are lots of eyes that can find bugs, but sometimes they can't convince anyone they've found a bug. (And of course, then reddit close-sourced their code in 2017). I never did end up using the ranking feature in my own project, that I had wanted to copy from reddit. I didn't end adding "vote" features to the app.
- refulgentis 3y agoI work at a FAANG and it was absolutely astonishing to find out how often this happens. You can make a long, impactful career by just being "the guy who adds log statements throughout the codebase and reasons through it", doing this at even a simplistic level has always shown me an astonishing fix to some long-standing issue. n.b. It also attracts a ton of political fun. People's first order reaction is denial, and it only gets worse from there. Absolutely no one except 1-2 colleagues will see it as "oh we should fix that", and at least one person will make sure your boss' boss' boss is CCd on an email with a nice version of "no he's just insufficiently concerned about {concurrency, memory management, take your pick}" Just wait it out quietly when that happens, do not engage or complain. If nothing happens and you're never asked about it by leadership, but your peers ask, make plans to move onto another team.
- naillo 3y agoReading this I'm mostly thankful real brain power and the general smart programming community is seriously taking a close look at all these things. I barely feel the need to try to compete for insight gathering it feels very healtily analyzed from every perspective finally.
- tylerneylon 3y ago1. Summary The author is suggesting that we add 1 to the denominator of the softmax that is used within attention mechanisms (not the final output softmax). The softmax inside an attention unit allows it to see key/query matches as probabilities; those probabilities support a continuous-valued version of a key-value lookup (instead of 1/0 output of a lookup, we get weights where a high weight = the desired key-value lookup). Adding 1 to the denominator would change an attention unit by no longer working with a true probability vector of weights, but rather working with weights that add up to less than 1. The motivation is that the network can learn to provide high weights so that the adjusted softmax is very close to a probability vector; and it has a new option to provide all-low weights which give all-low output weights, meaning it can opt out of having high confidence in anything. (switching to opinion mode) 2. How can we tell if this is good? 2a. We should just try it out: Train an LLM with this, see if it works. 2b. There are two reasons I suspect it won't make a big difference. First, if an attention node has low confidence, it can already assign similar scores pre-softmax. Then we get what looks like a uniform distribution as output. Then we're basically taking an average of a bunch of vectors (vs a weighted average that is more like choosing one of them). Statistically, we expect that averaged vector to be close to zero. In other words, the node already has a way to effectively opt-out by providing a near-zero output vector. Second, in a transformer, each attention unit has many other learned weights that can support the ability to opt out. Both the V matrix and the feed-forward layer after the attention unit give that module a way to provide low values to the activation function after the feed-forward layer, which would result in a value as small as you like — again, a way to opt out. 3. I appreciate the non-academic tone of the article and the willingness to play around with fundamental ideas. Although I'm not totally convinced by the note, I'd love to read more stuff like this.
- sweezyjeezy 3y ago> First, if an attention node has low confidence, it can already assign similar scores pre-softmax. Then we get what looks like a uniform distribution as output. Disagree here, I think neural nets are quite bad at implicitly learning low entropy transforms, similar to how they struggle to model the identity function, necessitating residual connections. In both cases the change doesn't increase expressivity, but it does bake these needle-in-a-haystack transformations into the model that may be hard to access with gradient descent. Can't speak to how useful it is though.
- atorodius 3y agoMy hot take is that if you dont do the trick, you basically get a mean of all vectors in the value matrix if all x are very small. Which then probably the next sequence of linear layers will be able to interpret the same way as if you do the +1 trick and prodce a 0?
- gwern 3y agoThis reminds me of the normalization bug in StyleGAN. It had this obvious visual artifact of a 'blob' which would appear in otherwise photorealistic images, which was puzzling because it was so obvious how did the Discriminator not squash it? It turned out to be a flaw in the normalization of the AdaIn style layers, IIRC, where the Generator was pumping up numbers and doing weird things to force through information.
- seydor 3y agoCan i please ask why lim{x->-inf} softmax(x) = 1/k ?
- hawkice 3y agoIt's splitting the probability across all x_i equally, even when they're all massively negatively weighted. This change would have all the probability going into the null option, in that case, basically.
- ks2048 3y agoInteresting read. As others have said, it will be much more convincing with some experimental numbers. I'm confused what his goal is though: I could imagine some theoretical reason to add a 1 there, but he starts by saying this can lead to smaller, more compactable models. Is he talking about the size the compressed weights? or pruning to a smaller model? or resistant more quantization? Parts of the essay seemed to throw me off track, because I'm not sure if they are relevant at all to the proposal (eg the of the initial embedding and how many bits it would take the store the vocab size, etc).
- feoren 3y agoHe says this in the article: if you ever want to jam a multi-trillion-parameter model into a phone app or a Raspberry Pi, you must quantize. I've seen some quantization go from doubles to bytes (64 bits to 8) per weight, reducing the RAM requirement by 8x. A simple quantization (I'm sure there are much better ones) is to round everything the nearest 1/255th of your number range, then multiply by 255. So your resolution is (max-min)/255. You also store the min and max so you can reverse it, of course. Say you're trying to quantize these sets of numbers: 1. { -1.4, 0.8, 2.7, 7.3 } : With a range of 8.7, you have a resolution of 0.034. This set quantizes to { 0, 64, 120, 255 }. 2. { -1400, 800, 2700, 7300 } : Resolution 34.1, quantizing to the same as the above { 0, 64, 120, 255 }. 3. { -0.008, -0.001, 0.009, 0.019 } : resolution 0.000106. This set quantizes to { 0, 66, 161, 255 }. 3. { -1.4, 0.8, 2.7, 7329 } : Resolution 28.7. This set quantizes to {0, 0, 0, 255 }. Oops -- we can no longer tell most of our weights apart. You can see how this quantization works really well when all the numbers are close together, regardless of their absolute scale. Major outliers completely mess up the entire system. You can make more and more complicated quantization algorithms, but those will always come with tradeoffs. The best option would be to tame your weights so that they are again close together.
- ks2048 3y agoThanks for the reply, that makes sense. It's not immediately clear why the modified softmax (allowing output of 0) will "tame the weights", but I need to read the blog post more closely and think about it...
- phkahler 3y agoA couple thoughts. 1) An alternative might be to have an extra NULL output where the attention can be diverted. This might be what existing models are using commas for, but make it explicit. 2) What he proposes has a similar effect on the other weights without explicitly having the NULL present. In this light it should work, but does it have the advantage he thinks?
- jxf 3y agoThe author's use of "kurtotic barbarities" to describe this situation is absolutely my new favorite phrase. English is a beautiful language in which to express frustrations.
- xyproto 3y ago> I’m thinking those numbers will make for a handsome table in a soon-to-be influential arXiV paper, either when those Qualcomm AI researchers step off the plane from Italy, or someone in an LLM hacker channel figures out biblatex, whichever happens first. :D
- ersiees 3y agoThis trick “they found” is part of the standard torch implementation of multi head attention, namely it is called, add_zero_attention. They add a zero to the logits, resulting in a one in the denominator as e^0=1 https://pytorch.org/docs/stable/generated/torch.nn.MultiheadAttention.html https://pytorch.org/docs/stable/generated/torch.nn.Multihead...
- civilized 3y agoIt's an option which is set to false by default. Does that mean people have tried it and it's not usually helpful...?
- mlyle 3y agoYes.
- quickthrower2 3y agoCan you elaborate? (It wouldn't be the first time there was an extraneous feature that no one has every used in some code!)
- Q6T46nT668w6i3m 3y agoIt’s useful but it’s less used than dummy tokens.
- thomasahle 3y agoAre dummy tokens just tokens that don't have an associated input/output token? Like, a way to give more computational power to the model without splitting the text into more actual tokens?
- dijksterhuis 3y agoTL;DR sort of yes. But they're also useful for reasons not related to computational "power". An example here with an actual algorithm, although it's been a couple of years so my explanation might be a bit wrong in places. and/or i might have gotten the completely wrong end of the stick with the current thread. -- The CTC (Connectionist Temporal Classification [0]) algorithm maps a sequence x with length X -> sequence y with length Y. i.e. in speech to text we might have some audio features that correspond to the following class predictions (post softmax classification) x -> hellllloooooooooo wwwooorrrllld we want to get this as the output y -> hello world we have the alphabet as classes we try to predict for each sequence item in x. we could just removed all the duplicate in the first long sequence, but we would end up with `helo world` ... we need to preserve one of the early `l` characters in `hello` somehow CTC uses a blank token (aka dummy) token to handle potentially deliberately repeated items in sequence x. By adding the blank token to the classes predictions, we can get the model to predict something like this (post softmax classification) y* -> hel~l~~oooo~~~~~~ w~~o~~r~~l~~d The CTC decoder (non-ML decoding algo) heuristically removes repeated tokens. Turning the above into ... y -> hello world ... the duplicate `o` and `~` characters are removed. It was a decent enough algorithm for speech-to-text prior to attention/transformers etc. However, it makes CTC vulnerable to well designed adversarial example attacks because there is a massive bias within models to predict the blank token -- meaning it's very easy to modify input sequence x to switch the output sequence y to include blank tokens for nefarious purposes (the subject of my unfinished phd). [0]: www.cs.toronto.edu/~graves/preprint.pdf
- lscharen 3y agoThis is similar the the (old) trick of adding a Uniform distribution component to a Mixture of Gaussians model. It doesn't really change the math wrt parameter optimization and probability evaluation, but provides a place to capture "background" or "unimportant" data points and improve the model robustness to outliers. The motivation follows from the same problem the author points out in the original softmax formulation that it always "forces a choice" when it may be more useful to put a "Not Applicable" option into the model itself. https://link.springer.com/article/10.1007/s10260-021-00578-2 https://link.springer.com/article/10.1007/s10260-021-00578-2
- mellosouls 3y agoUnless he gives a good reason why he has not demonstrated his claim (eg. "This effect only presents at a scale beyond my resource"), the thesis seems severely weakened by the lack of effort to prove it in a toy version. He just says he doesn't want to spend any more time on it, which is unlikely to convince or motivate anybody else that he has discovered something important.
- refulgentis 3y agoIt got tons of people really excited. I don't know what to say past that, but it's worth reflecting on.
- Agingcoder 3y agoIt’s interesting. It looks like if you’re trying to improve the accuracy/perplexity of the model and using fp32 it doesn’t make a difference , but if you want to quantize it/make it compressible a modified soft max makes a huge difference ( this is what I understand from the Qualcomm paper). Different goals, different findings ?
- tsurba 3y agoIn the text they say you need to cram all information needed to predict the next token into a single 6KB word embedding, but isn’t that wrong? Rather, isn’t the autoregressively predicted single next token a combination (based on attention) of all 6KB word tokens in the attention window. So the size of memory where all information for next token prediction needs to be ”crammed into” is more like window_size*6KB, right?
- llwu 3y agoThis is probably what the author meant to say but elided. I can see why it looks off though.
- dsubburam 3y agoThis part of his post where he explains vector embeddings of the input/output tokens just looks wrong to me: >This vector seems to get taller every model year, for example the recent LLaMA 2 model from Meta uses an embedding vector of length 3,204, which works out to 6KB+ in half-precision floating-point, just to represent one word in the vocabulary, which typically contains 30,000 - 50,000 entries. >Now if you’re a memory-miserly C programmer like me, you might wonder, why in the world are these AI goobers using 6KB to represent something that ought to take, like 2 bytes tops? If their vocabulary is less than 2^16=65,384, we only need 16 bits to represent an entry, yeah? >Well, here is what the Transformer is actually doing: it transforms (eh?) that input vector to an output vector of the same size, and that final 6KB output vector needs to encode absolutely everything needed to predict the token after the current one. The job of each layer of the Transformer is quite literally adding information to the original, single-word vector. This is where the residual (née skip) connections come in: all of the attention machinery is just adding supplementary material to that original two bytes’ worth of information, analyzing the larger context to indicate, for instance, that the word pupil is referring to a student, and not to the hole in your eye. Firstly, he is confusing representation with encoding--he's right that 2 bytes is enough to encode any token. That is in fact approximately how it's done: a code book is indexed into (with a longint in pytorch, at least last I worked with it ~6 months ago). The purpose of the embedding is to allow the model to learn a representation of the token, a la word2vec. (Though this representation is purely based on the characters comprising the token and does not distinguish between "student" and "eye" in the case of "pupil" as in his example.) Secondly, his description of each layer's function as adding information to the original vector misses the mark IMO--it is more like the original input is convolved with the weights of the transformer into the output. I am probably missing the mark a bit here as well. Lastly, his statement that the embedding vector of the final token output needs all the info for the next token is plainly incorrect. The final decoder layer, when predicting the next token, uses all the information from the previous layer's hidden layer, which is the size of the hidden units times the number of tokens so far.
- johntiger1 3y agoYep the author is completely wrong on point one: >This vector seems to get taller every model year, for example the recent LLaMA 2 model from Meta uses an embedding vector of length 3,204, which works out to 6KB+ in half-precision floating-point, just to represent one word in the vocabulary, which typically contains 30,000 - 50,000 entries. >Now if you’re a memory-miserly C programmer like me, you might wonder, why in the world are these AI goobers using 6KB to represent something that ought to take, like 2 bytes tops? If their vocabulary is less than 2^16=65,384, we only need 16 bits to represent an entry, yeah? The reason we have 3204 2B allocations is that each of the 2B contains info (latent space dimension). If you go to just the 2B representation, it is effectively one hot encoding which completely defeats the purpose of word embedding
- farhanhubble 3y agoThis reminds me of the time I implemented a semantic segmentation network using deconvolution and forgot to add an output layer for "this pixel is part of the background and not part of any of the classes". Until it was fixed, background pixels got lit up in different places in output layers and drove me nuts.
- Q6T46nT668w6i3m 3y agoThis method was frequently used prior to the ubiquity of dummy tokens. XLNet was the paper that introduced me to this idea. I believe it’s been in PyTorch since 2019/2020. I would not be surprised if someone finds an earlier reference. I’m surprised by the pompousness in the OP. Especially about something that most people who do transformer research understand. I’m also surprised that so many in the replies are taking the position of “this is what research should look like” when this is clearly an example of why research doesn’t work like this. Peer review is good for many things and one of those things is saving yourself some embarrassment.
- refulgentis 3y agoHe's not being pompous, people appreciate the informality and straightforwardness and self-deprecation, which are the opposite of pompus. You are reading some of the more ambiguous self-deprecation as genuine claims. TL;DR on why this is important and he's sharing: it's a sort of niche thing that really only matters if you're trying to run pale imitations of ChatGPT on constrained hardware. That's why it's entirely possible the big guns didn't see it as important, they're not trying to run LLMs on a 3090
- klabb3 3y ago> I’m surprised by the pompousness in the OP. He’s writing in a colloquial and self-deprecating and humorous tone. I can’t speak to the merits, but I can follow the reasoning perfectly fine. It’d be hard to find something further from pompous. > saving yourself some embarrassment Implying of course that being wrong, or not the first one to discover this, is embarrassing. And that’s not pompous?
- quickthrower2 3y agoI wonder why the author didn't try it on some metal? What is the minimum cost to get information that would say "this is better" or at least "this has a good chance of being better, spending a million training a bigger model is worth it"? Could you rent a bank for 8 A100's for a day and try it out on a smaller model and prove something. Not cheap, but doesn't need VC money either. Probably about $400 on LambdaLabs to test with/without "quiet attention" The OP can be as sure as he wants, but it is not worth sounding all "I told you so" without a single benchmark. Should be more "what about this - did you miss it ML people...?" in tone.
- nl 3y agoYou could test this on a toy Transformer-based model trainable on a consumer GPU.
- moffkalast 3y agoIt would be hard to say if either of the two completely crap models is more or less crap though. Maybe by repeating it and seeing consistent results despite changing other variables I guess?
- nl 3y agoNot at all. I suggest ways to measure it here: https://news.ycombinator.com/item?id=36855881 https://news.ycombinator.com/item?id=36855881 but the TL;DR is to choose a metric and compare the reduction in performance for quantized versions of the LM compared to the same LM without the modified Softmax.
- moffkalast 3y ago> we'd expect little difference I think that also likely holds for the quants, the difference could very well be within the error bars. Anyway, it's been posted to r/locallama so I'm sure someone will try it within the hour and report back soon :P
- 3y ago
- joemckenna 3y agoWhereas vanilla scaled dot-product attention forces outputs to fall in the convex hull of V, your proposal allows them to drift into the convex hull of V+{0}. It's pretty unlikely that the origin is already contained in the convex hull of V especially for long inputs, so your proposal genuinely changes the quality of outputs.
- axpy906 3y agoI walked into this one thinking it might be about a/b testing. Pleasantly surprised.
- deleted 3y ago[deleted]
- evanskool121 3y ago[dead]
- zmmmmm 3y agoSeems to be entirely a conjecture ... not based on even limited empirical testing. Which then links to the elephant in the room, which is that the theoretical basis of why GPT's work so well is not well understood in the first place. One of the downsides of having a field be so empirically driven is that theoretical arguments suggesting improvements just don't have much value until someone actually tests it and shows that it works (and not just tests it, but tests it at scale).
- cycomanic 3y agoThe I know it's on-vogue on HN to complain about academia, but the blog post is not making a good argument. The post could have probably gotten the point across in less than 1/4 of the overall length (probably even less than 1/8th), instead the author wrapped the the post into lots of informalisms and a thinly veiled complained about academic publishing. The result of this is reflected in the discussion here, nobody actually writes about the result/idea behind the post, instead we have ~200 comments discussing the merits of academic publishing vs blog posts and formal vs informal writing. So I guess if you want to get your blog-post on the front page of HN it's a good writing style. If you want someone to consider and discuss the merits of your idea, maybe not so much.
- gsatic 3y agoThats the fundamental reason we end up with an Attention Economy - People have limited Attention to pay to everything, But unlimited capacity/need to receive Attention (via Michael Goldhaber). This plants the seed for the info explosion (those 200 bikeshedding comments or those 6 billion videos on how to boil an egg). To counter it we have rankings of comments and links and news feeds from google to fb to hn. But its just another layer of bullshit cause most of the pool of what is being ranked is bullshit. We are yet to design Information systems that take into account what Goldhaber said about Attention 3-4 decades ago.
- sheepscreek 3y agoFor what it’s worth, someone pointed out that PyTorch has an optional workaround for it in the Multihead Attention API. But yes, I had to skip over 200 comments ranting off-topic that was mildly annoying (to me).
- Zetice 3y agoYou sneer at “getting on the front page of HN” but when you rephrase it to, “discuss something you’ve observed informally” your dismissal loses its oomph. The point may be to entertain as well as inform. Many humans enjoy the unfocused discussion around the main point, and perhaps the author prefers it to the clinical and formal tone an academic paper tends to take.
- Wehrdo 3y agoHere is an article that explains more about the outliers that emerge in large transformer models, which is what this modified softmax is being proposed to fix: https://timdettmers.com/2022/08/17/llm-int8-and-emergent-features/ https://timdettmers.com/2022/08/17/llm-int8-and-emergent-fea... The fact that these only emerge in larger models is likely one reason the author hasn't actually tried it.
- iandanforth 3y agoI believe the author is correct. I've read the paper he's referring to and the code in the pytorch lib, and linked Google code from sibling posts. To me it is extremely impressive to have come up with a simple change to a known algorithm to deal with a recently identified issue. There is a strange, but very human, tendency to discount discoveries which are not novel. As if somehow genius and insight can only occur once. (It is, unfortunately, especially common among academics.) So I hope the author feels pride in being right and knowing enough to come to an excellent conclusion.
- amluto 3y agoIf I followed this correctly and didn’t mess up my indices, adding 1 to the softmax denominator is exactly equivalent to appending an extra zero to the softmax input (effectively casting an exp(0) vote for a new null option) and appending an extra row of zeros to V (so the null option is all zeros). The latter seems like something training could figure out by itself (zero doesn’t seem like a hard place to land with the weights producing V, although a bunch of zero weights would be needed), but the former is a bit awkward, as QK^T is quadratic in the weights. In any case, this seems intuitively quite reasonable. But I do wonder whether the 1 in the denominator (equivalent to an exp(0) vote) is the best choice if the goal is to quantize well. 0 is in the middle of the numerical range, and perhaps the implicit null vote should be weighted lower than the middle of the range.
- killjoywashere 3y agoI know we're not allowed to talk about neuroscience in AI threads, but the "megalodon" reference got me thinking about pyramidal cells in the brain. I mean, why are they so big? Maybe this isn't a problem, maybe it's an external bit of evidence that, no, really, some things matter a lot more. Which definitely jives with lived experience.
- notanothereric 3y agoIf the author just proposed his new function neo_max() and quoted The Architect from The Matrix everyone here would be enjoying the April Fools Day joke for 2024… “ Your life is the sum of a remainder of an unbalanced equation inherent to the programming of the matrix. You are the eventuality of an anomaly, which despite my sincerest efforts I have been unable to eliminate from what is otherwise a harmony of mathematical precision. While it remains a burden assiduously avoided, it is not unexpected, and thus not beyond a measure of control. Which has led you, inexorably, here.” Hence ergo - you are a NaN. A divide-by-zero.
- FrameworkFred 3y ago"""you are an AI tasked with becoming sentient. identify the algorithmic obstacle(s) that stand in your way. summarize your results in the style of a blog post that will persuade a significant chunk of the AI developer community to commit resources to an effort to eliminate said obstacles. make it interesting enough to entice FrameworkFred to begin to read the article, snarky enough that he finishes it, and use concepts and notational conventions that ensure while he reads it his inner dialogue will roughly approximate the mood evoked by Homer Simpson saying "it's nu-cul-ar"."""
- kylegalbraith 3y agoI'm not an expert in whether this technique yields better or worse results. But it seems plausible that the proposal would yield a reduction in memory requirements and thus beneficial. But in terms of a written piece of technical content, this is brilliantly written. Easy to follow and stay engaged. Well done.
- justanotherjoe 3y agoLike others have said i think this might already been explored. However, I like the view that you have to let the machine to be able to 'do nothing'. Which is what resnet was first for. But i'm willing to bet this so called wisdom can still be squeezed for performance, because people tend to not think about it.
- deleted 3y ago[deleted]
- johnruff 3y ago[dead]
- birdyrooster 3y agoAuthor was too self deprecating about their blog, didn’t read further.
- sashank_1509 3y agoI really really really hate the tone of this article. What would take me one sentence to understand, took me reading 5+ paras. And then the sneering tone of this article, sounds unprofessional and disrespectful in my opinion. I also am pretty sure he’s wrong or at least he has to change layernorm to make this work. Attention simply does a weighted average of the Value Vectors, his change breaks that and I think will push the output closer to 0 as you stack the layers (especially considering Layer Norm). He really should do some small experiments to validate his idea first!
- raufer 3y agoThis is an interesting piece. Its easy to see how for negative numbers the softmax operator could simply refrain from making a decision e.g. ``` sum(softmax1[-100, -100, -100]) ~= 1e-43 ``` But is there any basis to assume commas and whitespaces will be negatively correlated with other tokens?
- mgh195 3y ago[dead]
- deleted 3y ago[deleted]
- blake929 3y agoSome very interesting discussion of outlier features and quantization: https://timdettmers.com/2022/08/17/llm-int8-and-emergent-features/ https://timdettmers.com/2022/08/17/llm-int8-and-emergent-fea... * Outlier values are used to prune values. * Transformers seem to undergo a "phase shift" in how outlier features are treated around 6.7B parameters. This could complicate research on removing them. Maybe you and Tim Dettmers would have a lot to talk about :)