13 ms·
Scaling Transformer to 1M tokens and beyond with RMT
- cs702 3y agoCool. The idea of augmenting a transformer with an RNN isn't new, but I'll take a look. See also the threads here, mentioning other methods for scaling up context length up to the millions of tokens: https://news.ycombinator.com/item?id=35502187 https://news.ycombinator.com/item?id=35502187
- tartakovsky 3y agoWow! I don't know how accuracy translates, I do see charts that look strong but unless I'm missing something, this is incredible. Would be curious and also terrified to see an endpoint so I can play around with it. I thought we were stopping this kind of research?
- alchemist1e9 3y ago> I thought we were stopping this kind of research? I’m going to, hopefully correctly, assume that was a funny joke.
- tartakovsky 3y agoI’m just saying, it was more a hopeful nod in the direction of wishful thinking. Not a joke, just a bit of having my head in the sand.
- alchemist1e9 3y ago> I’m just saying, it was more a hopeful nod in the direction of wishful thinking. I still don’t understand why this is even wishful thinking, I don’t understand what the point of delaying progress is.
- madmads 3y agoBecause some people believe that sufficient progression in this field would result in a setback for humanity in the long run. Personally, I'm not sure what to believe.
- nullsense 3y agoWhat's your assessment of a world full of computers that can actually tell you "no" will actually be like?
- flangola7 3y agoBecause AI development is the most dangerous thing humans have ever done? Respectfully, have you been under a rock? "Progress" doesn't mean "every massive change to the world and humanity is good." There are undeveloped technologies that we are currently not capable of being responsible with. Google differential technological research.
- alchemist1e9 3y agoI’m using GPT-4 basically everyday multiple times. I’m following closely LLM developments. Yet I have trouble seeing what people find so dangerous. It’s amazingly cool stuff that will create massive productivity gains. I guess I just lack needed imagination to believe this “most dangerous thing humans have ever done” perspective. It looks to me the most dangerous thing is actually Gain Of Function research on pathogens. That seems like most dangerous thing. Close 2nd is nuclear weapons. LLMs seems really nice and fuzzy and warm to me in comparison.
- kubanczyk 3y agoThis convinces me: > Intelligence is the only advantage we have over lions, who are otherwise much bigger and stronger and faster than we are. But we have total control over lions, keeping them in zoos [...] The "intelligence" is a mouthful, but I think the advantage described above is, at its root, caused by a slightly better "predict" step in the observe-predict-act loop. That's it. And LLMs aren't exactly bad predictors, are they? And it looks like while our predictive abilities rise over centuries, theirs rise over decades.
- deleted 3y ago[deleted]
- kristintynski 3y agoAnother roadblock to AGI starting to fall. WOW.
- libsaresubhuman 3y ago[dead]
- deleted 3y ago[deleted]
- lumost 3y agoThis paper presents a novel approach to incorporate memory into a transformer. It does not demonstrate that this approach works in a useful manner. While the approach is interesting, I’m skeptical that the RNN has enough capacity to encode the memory in it's output. I would have liked to see more detail on the synthetic benchmark they used. The memory component may be learning the benchmark rather than a generalized feature.
- mysterybox 3y ago[flagged]
- tomohelix 3y agoAn AI can theoretically ingest 1M token of data, analyze, summarize, indexing, and storing the condensed information in a hard drive. Then later on retrieve it based on the metadata for specific situations that needs it. At millions of token, it is basically a superhuman that can learn new things on the fly at the speed unimaginable to any organic based life. We are so close to something amazing, and scary.
- alchemist1e9 3y agoDo we yet have a single example of a transformer based AI, i.e. LLM, learning something new that we didn’t teach it in the training data? Maybe we do. I’m not sure how to define it, but new should be a discovery or insight or even relationship that is not explicitly taught in the training data. If we don’t then it suggests without humans in the loop that the super intelligence is not so close?
- outlace 3y agoGPT can learn things "in context" E.g. you can teach it something by chatting with it, but it will eventually forget it after its context length is exceeded. It cannot continuously learn and remember like biological organisms since its weights are frozen.
- KRAKRISMOTT 3y agoSo we still have catastrophic forgetting, just on a slightly longer scale.
- anon291 3y agoExcept it cannot conceptualize reasonably. It is utterly incapable of symbolic thinking in a general sense.
- LASR 3y agoIt’s incapable of thinking. But what it’s surprisingly good at is symbolic reasoning similar to human common sense. Most people forget that there is no memory or thought independent of whatever the output it generates. But when you ask it to follow a chain of thought and generate it in the output, the eventual conclusions it lands on is scarily human. People asking ChatGPT for word counts and sighing at how wrong it is. Yeah of course, it has no counter variables to hold numbers and increment them. It has no memory. But when you ask it to generate a numbered list of all the words in a passage and then output the word count, it gets it right every time. Because you basically gave it a memory by encoding the counter in the generated output.
- panabee 3y agoAbstract This technical report presents the application of a recurrent memory to extend the context length of BERT, one of the most effective Transformer-based models in natural language processing. By leveraging the Recurrent Memory Transformer architecture, we have successfully increased the model's effective context length to an unprecedented two million tokens, while maintaining high memory retrieval accuracy. Our method allows for the storage and processing of both local and global information and enables information flow between segments of the input sequence through the use of recurrence. Our experiments demonstrate the effectiveness of our approach, which holds significant potential to enhance long-term dependency handling in natural language understanding and generation tasks as well as enable large-scale context processing for memory-intensive applications. Introduction The Transformer model (Vaswani et al., 2017) has been widely adopted and used in various research areas and industrial applications. The most important issue of the model is quadratic complexity of attention operation, that makes large models increasingly difficult to apply to longer inputs. This report we show that by using simple token-based memory mechanism introduced in (Bulatov et al., 2022) can be combined with pretrained transformer models like BERT (Devlin et al., 2019) with full attention and full precision operations can be applied to sequences longer than 1 million tokens using a single Nvidia GTX 1080Ti GPU. Contributions 1. We enhance BERT by incorporating token-based memory storage and segment-level recurrence with recurrent memory (RMT). 2. We demonstrate that the memory-augmented BERT can be trained to tackle tasks on sequences with lengths up to seven times its originally designed input length (512 tokens). 3. We discovered the trained RMT’s capacity to successfully extrapolate to tasks of varying lengths, including those exceeding 1 million tokens with linear scaling of computations required. 4. Through attention pattern analysis, we found the operations RMT employs with memory, enabling its success in handling exceptionally long sequences. Discussion The problem of long inputs in Transformers has been extensively researched since the popularization of this architecture. In this work, we demonstrate that applying Transformers to long texts does not necessarily require large amounts of memory. By employing a recurrent approach and memory, the quadratic complexity can be reduced to linear. Furthermore, models trained on sufficiently large inputs can extrapolate their abilities to texts orders of magnitude longer. Synthetic tasks explored in this study serve as the first milestone for enabling RMT to generalize to tasks with unseen properties, including language modelling. In our future work, we aim to tailor the recurrent memory approach to the most commonly used Transformers to improve their effective context size.
- politician 3y agoI read the paper, but I’m not sure that I understand how the memory unit works. It seems like they reserve special tokens to read and write to this memory between layers, as guided by the self-attention heads. In this way, the model can learn to squirrel away important details from the input prompt for later recall in subsequent layers. A global chalkboard mechanism. Am I on the right track?
- deleted 3y ago[deleted]
- Imnimo 3y agoI don't buy this. I don't think it's reasonable to set up the task such that the "fact" can be distinguished from the background without knowing the question. For example, if your fact is always "[name] went to the [room]", and question is always "where is [name]?" the background text should be other sentences of the same form about different names. If your background text is instead drawn from a different distribution (as in this paper), then the model can simply cheat by identifying the fact (which constitutes only a few bits of information) and ignoring everything else. That isn't scaling to 1M tokens - the model should be expected to answer questions about any text within its context window.
- panabee 3y agothis is a great point. do you know of any benchmarks doing this today? given the acute need to evaluate models on contextual factuality, we're exploring how to create a benchmark for this purpose but prefer existing benchmarks if possible. openai's truthfulqa[0] is close but does not focus on contextual factuality and targets a much harder problem of absolute truth. if none exist, and people are interested in contributing, please reach out. [0] https://github.com/sylinrl/TruthfulQA https://github.com/sylinrl/TruthfulQA
- nl 3y agoFor those who don't completely get the impact of this: We already know Large Language Models (LLMs) can learn at runtime (ie, separately to the training process.) This is called "In Context Learning". See [1], [2] for more details. (BTW, when anyone says "LLMs are stochastic parrots" you know they are ignorant of this) In context learning is wonderful because it means you can "train" a LLM at run time by filling the context with examples. Traditionally this "context window" has been a few thousand tokens, and GPT-4 recently extended that to 32,000 tokens. That is useful, but if you wanted to say load all of a companies documents and ask questions it doesn't really work because this overflows the context. But at 2M tokens there's a whole range of applications that become possible. [1] Language Models are Few-Shot Learners: https://arxiv.org/abs/2005.14165 https://arxiv.org/abs/2005.14165 [2] Language Models Secretly Perform Gradient Descent as Meta-Optimizers: https://arxiv.org/abs/2212.10559 https://arxiv.org/abs/2212.10559
- kgwgk 3y ago> (BTW, when anyone says "LLMs are stochastic parrots" you know they are ignorant of this) Do you? Some of them -at least- understand that the output is conditional on the input.
- nl 3y ago"Conditioned" with the ability to introduce new facts and have the model infer impacts on them. As an example, tell chatGPT that the Queen of England died (which occurred after the data cut off) and then ask it who the head of state of Australia is. It's able to infer the head of state of Australia is now Charles III (and gives a good explanation of how this is mostly ceremonial.) See https://twitter.com/nlothian/status/1646699207506685953 https://twitter.com/nlothian/status/1646699207506685953 At some point the word "stochastic" doesn't really capture that behavior in any useful sense.
- geraneum 3y agoInterestingly, the example provided is not necessarily indicative of the ability to infer the actual physical transfer of power. It’s possible that the terms “After Elisabeth, Charles will be king” are accompanied with “Elisabeth” and “Die” or “Death” in the same context before cut off (Wikipedia, news papers, tabloids). You cannot draw any scientifically sound conclusions without knowing what was actually in the training data and use a control group/instance.
- zwaps 3y agoI really wish that CS would switch to discussing related literature before (or with) contributions. The authors only cite themselves until the very end of the paper, and the whole time you are thinking: "Hasn't this been done before? Transformer-XL (recurrent Memory) is a years old paper!" And you need to read to the very and to find out that the contribution is actually: Keep your architecture (but still need to train it on specific tasks)... It's like a blog-post recipe in research paper form
- ad404b8a372f2b9 3y agoI feel like it used to be much more common to have "Related Works" as part 2 or way earlier in papers in earlier years.
- macleginn 3y agoThey cite the paper where the architecture was introduced. If you go to that paper, you'll see that it mostly consists of a very detailed and careful comparison with Transformer-XL. In the new paper, they plug their memory system into vanilla BERT. This makes the resulting model essentially nothing like Transformer-XL, which was a strictly decoder-only generative language model.
- albertzeyer 3y agoThis looks like a potential NeurIPS submission. But it will probably be rejected. The quality bar for NeurIPS is quite high. Some reasons: The experiments are very weak: There are just a few figures, basically figure 1 and figure 5, which show some results. There are no tables with numbers. But more importantly: There are no comparisons (in terms of experiments/numbers) to similar models, like: - Block-Recurrent Transformers (https://arxiv.org/abs/2203.07852 https://arxiv.org/abs/2203.07852) and related approaches to make the Transformer recurrent, so effectively getting infinite context length. - All the work on sparse attention, or linear attention, like Longformer, etc, which should also allow for such context lengths. I don't mean that they just mention them in related work (they partly do, although it looks very short, so I'm quite sure they leave out a lot of other related work, without looking too much into it). I mean that they really run experiments and show numbers. And also, to look at the model definitions of those alternatives, and compare and analyze the differences. Analysis also seems to be a bit weak. Not sure about novelty. So, while the presented approach could be interesting, it's hard to tell how good it really performs, and how it compares to alternatives. (This is now a 10 min review by me. If I had to review this, I usually would spend more like 1-2h at least, so giving much more details. But I don't expect that my initial impression would change too much.)
- deleted 3y ago[deleted]
- brucethemoose2 3y agoI feel like this is a thing in ML land, like everyone is in such a rush to publish something "revolutionary" because the whole field is already moving at warp speed. Under such pressure, its easy to lose focus on metrics and comparisons.
- macleginn 3y agoThe original architecture used in this model was accepted at last year's NeurIPS: https://proceedings.neurips.cc/paper_files/paper/2022/file/47e288629a6996a17ce50b90a056a0e1-Paper-Conference.pdf https://proceedings.neurips.cc/paper_files/paper/2022/file/4... That paper is written very differently.
- 3y ago
- singularity2001 3y agoAn alternative would be to enable LauRa fine tuning per customer / use case (?)
- Tepix 3y agoYou probably mean LoRa (Low Rank adaption)
- vrglvrglvrgl 3y ago[dead]
- deleted 3y ago[deleted]
- mysterybox 3y ago[flagged]
- l33tman 3y ago"Figure 2: Recurrent memory mechanism. Memory is passed to Transformer along input sequence embeddings, and memory output is passed to the next segment. During training gradients flow from the current segment through memory to the previous segment." The whole point of the 2017 paper "Attention is all you need" was to show that the recurrent structure previously used (with attention mechanisms) wasn't needed. It's pretty cool if it does turn out that both mechanisms are needed and powerful together. After all, the brain doesn't linearly process the entire book you're reading from page 1 to where you are, for every new word you read, it stashes the important parts into short/long term memory along the way. Still, passing gradients back through several stages like that is typically a tricky thing to do in practice.
- cced 3y agoIs any of this open source? I just recently started experimenting with summarization of youtube video audio that has been passed through openai/whisper and to my surprise I’m told that the maximum token lengths are in the 1k range. I figured that I’d be able to roll my own summarization much like I was able to do so with transcription. I’m hoping that the findings of this work are open source. Does anyone have any extra insights?
- Borrible 3y agoGiven the example in the paper: Fact1: The hallway is east of the bathroom. Fact2: The bedroom is west of the bathroom. Question: What is the bathroom east of? Answer: bedroom Asked for better understanding, if the two facts given above are embedded in,say the Bible, would the system be able to answer the question correctly? Does that mean it can track the state of an object and determine it at a certain point in time?
- cs702 3y agoHere's a list of tools for scaling up transformer context that have github repos: * FlashAttention: In my experience, the current best solution for n² attention, but it's very hard to scale up beyond the low tens of thousands of tokens. Memory use is O(n) but compute is O(n²). Code: https://github.com/HazyResearch/flash-attention https://github.com/HazyResearch/flash-attention * Heinsen Routing: In my experience, the current best solution for n×m attention, i.e., mapping n tokens to m tokens. It's like a souped-up version of attention. I've used it to pull up more than a million tokens as context. Memory use and compute are O(nm). It works, but in my (limited) experience, it doesn't work out-of-the-box as well as FlashAttention for n² attention. Code: https://github.com/glassroom/heinsen_routing https://github.com/glassroom/heinsen_routing * RWKV: A sort-of-recurrent model which claims to have performance comparable to n² attention in transformers. In my (limited) experience, it doesn't. Others seem to agree: https://twitter.com/arankomatsuzaki/status/1639000379978403853 https://twitter.com/arankomatsuzaki/status/16390003799784038... . Code: https://github.com/BlinkDL/RWKV-LM https://github.com/BlinkDL/RWKV-LM * RMT (this method): I'm skeptical that the recurrent connections will work as well as n² attention or n×m routing in practice, but I'm going to give it a try. Code: https://github.com/booydar/t5-experiments/tree/scaling-report https://github.com/booydar/t5-experiments/tree/scaling-repor... In addition, the group that developed FlashAttention is working on state-space models (SSMs) that look promising to me. The idea is to approximate n² attention dynamically using only O(n log n) compute. There's no code available, but here's a blog post about it: https://hazyresearch.stanford.edu/blog/2023-03-27-long-learning https://hazyresearch.stanford.edu/blog/2023-03-27-long-learn... [CORRECTION: Code is available. See comment by lucidrains below. I'm hopeful this will go to the top of my list.] If anyone here has other suggestions for working with long sequences (hundreds of thousands to millions of tokens), I'd love to learn about them.
- lucidrains 3y agothe code is here https://github.com/hazyresearch/safari https://github.com/hazyresearch/safari you should try it and let us know your verdict.
- cs702 3y agoThank you. Somehow I missed that. I'm still making my way through the related papers, trying to get a superficial but accurate/intuitive understanding of these models. Embarrassingly, the work hasn't quite 'clicked' for me yet. Looking forward to tinkering with the code!
- dalys 3y ago1M token is around 750k words, in English. According to Wolfram Alpha that is: Single-spaced document: 1500 pages Double-spaced document: 3000 pages Book: 1028 pages So around 1-5 books. I'm assuming they're using OpenAI's tiktoken tokenizer. (??)