13 ms·
RWKV: Reinventing RNNs for the Transformer Era
- whimsicalism 3y agoAwesome to see this published. Work on transformer alternatives, especially parallelizable ones like this, is incredibly important - it would suck if we get sucked down a local optima in architecture without actually looking at nearby viable alternatives.
- pico_creator 3y agoYup. I’m all here for infinite scaling of context size
- mindwok 3y agoAs a (mostly) layperson, this seems like it could be a very significant paper. What are the odds we see the next few years of machine learning models based on RWKV like we have seen with transformers since the attention is all you need paper?
- m3kw9 3y agoThis model seem to be good when input context is large in comparison to OpenAI, would just wait till if someone productize it. Most likely it wine be just one paper but many new and old ideas combined
- visarga 3y agoBeing a RNN there is another trick: caching a long prompt, because RNNs only look back one step while transformers see the whole sequence. So you can load your long context only once and reuse it many times.
- pico_creator 3y agoYup. This is commonly done in the community for the chat models as well (due to the huge amount of reuse for each reply)
- FallDead 3y agoPretty high, we are going to be deploying something similar to Sono's GPT based LLM, and leverage RWKV for that, still playing around with it though!
- andy_xor_andrew 3y agoOne thing I'm keen to understand is: how well does attention hold across huge context sizes, with respect to the usual transformer models, and also these proposed RNN models? All these 2k/4k/8k context sizes that we've had recently are able to map pretty well to what a human could reasonably remember. What I mean is, you could ask a human to read some text with 8k tokens, and for the most part they could answer questions about the text coherently. But what about 32k contexts, or beyond? At some point, as token size increases, the ability of a human to give a highly precise and detailed answer decreases. They must start to generalize. A human could not read Infinite Jest in one pass and then answer details about every single sentence. But could a transformer, or a RNN? As the context grows, is it harder to keep a high granularity of detail? Or am I wrong in trying to think of these models the way I think about the human mind, and they are actually able to handle this problem just fine? I'm aware that we can cheat a bit, by adding a lookup step into an embedding database, to provide "infinite" context with "infinite" precision. But to me, that is analogous to a human looking up information in a library in order to answer a question. I'm interested in the inherent, emergent memory that these models have available to them in just one forward pass.
- humbleharbinger 3y agoNot sure if it answers your question but the paper notes something similar in its discussion of limitations: > the linear attention of RWKV leads to significant efficiency gains but still, it may also limit the model’s performance on tasks that require recalling minutiae information over very long contexts. This is due to the funneling of information through a single vector representation over many time steps, compared with the full information maintained by the quadratic attention of standard Transformers. In other words, the model’s recurrent architecture inherently limits its ability to “look back” at previous tokens, as opposed to traditional self-attention mechanisms. While learned time decay helps prevent the loss of information, it is mechanistically limited compared to full self- attention. > Another limitation of this work is the increased importance of prompt engineering in comparison to standard Transformer models. The linear attention mechanism used in RWKV limits the information from the prompt that will be carried over to the model’s continuation. As a result, carefully designed prompts may be even more crucial for the model to perform well on tasks
- bee_rider 3y agoDumb arxiv question (sorry); is it possible to see what journal a paper was actually submitted to, to help find the reviewed version when it comes out?
- ftxbro 3y agothat's what arxiv used to be, but that's not what it is anymore
- pico_creator 3y agoThis is straight to ARXIV (its not even the final copy yet)
- rsfern 3y agoSometimes that is included in the journal LaTeX template, but otherwise authors (in my field at least) typically won’t say where a paper is submitted before it’s accepted for publication there, or at the very least if it’s made it past the editorial rejection hurdle and been sent out for review Probably your best bet for getting a feed heads up is to set a GScholar alert on the first author?
- stellaathena 3y agoThe paper is going to be submitted to EMNLP next month. An early version is being released now to garner feedback and improve the paper before submission.
- bee_rider 3y agoThanks!
- sbierwagen 3y agoIt's not required for arxiv authors to submit papers to journals. As far as I can tell "Language Models are Unsupervised Multitask Learners", (the GPT-2 paper) which has 9,786 citations, was never published in a journal.
- bigdict 3y agoJesus H. Christ, first time I see a collaboration this big on a ML paper. How does a team like that even come together? This isn't the LHC.
- lordofgibbons 3y agoThey're a group of volunteers who came together over Discord.
- ftxbro 3y agoI think it's just, everyone in their discord channel.
- Buttons840 3y agoWhat channel? I have an application for sequence models I think might be novel, and I'd like to be able to get credit and help research it if possible. Probably somebody has already done it, but I cannot search well enough to find related literature.
- ftxbro 3y agoWrite it and submit it yourself. Sole authorship is given more weight these days for whatever reason. In a multi author publication, even if you are first author or listed as equal contribution, if there is a more famous person on the paper everyone will assume they did it.
- Buttons840 3y agoWrite it and submit it to who? I'm not familiar enough with the field to find any prior work or related work.
- ftxbro 3y agofind the most closely related paper that you know of, even if it's not very close, and submit your idea to the same journal that published that other idea
- ftxbro 3y ago> "Our experiments reveal that RWKV performs on par with similarly sized Transformers"
- zh217 3y agoThis sounds pretty bad, right? Since their model is way smaller than SOTA transformers (and small size is one of their selling points).
- pico_creator 3y agoThe paper is meant to compare architecture vs architecture with similar model size, and dataset - to inform decisions in future architecture designs Its main benefits being presented with the above staying closely the same, is that it has significantly lower running and training cost without performance penalty If you want to compare any <20B model with GPT 3.5 / 4 / 100B models evals, thats another paper altogether
- elromulous 3y agohttps://arxiv.org/pdf/2305.13048 https://arxiv.org/pdf/2305.13048 ^direct pdf link
- satvikpendem 3y agoSee also, ChatRWKV, an open source LLM powered by RWKV instead of transformers. https://github.com/BlinkDL/ChatRWKV https://github.com/BlinkDL/ChatRWKV
- cs702 3y agoAlas, it doesn't appear to work well for longer contexts: https://twitter.com/arankomatsuzaki/status/1639000379978403853 https://twitter.com/arankomatsuzaki/status/16390003799784038... Has anyone here experimented with this recently to confirm?
- FallDead 3y agoone of the authors here!, I think someone in our discord did experiments to prove that it does work for longer contexts, The pace of this work moves really fast. This might have been an earlier models in the series. RWKV it needs to be trained for longer contexts lengths in order to obtain that skill a context tuning if you will. IRCC there will be a follow up paper for it.
- cs702 3y agoThank you for taking the time to comment here! That sounds promising. Maybe scale (of model and training samples) is all you need. And RNNs are obviously so much more efficient at inference. I'm going to take a closer look :-)
- pico_creator 3y agoIt has already been confirmed that with the right dataset we can scale it effectively from 2k to 4K, and 4K to 8k via fine tuning (you dun even need to train a new foundation model) We believe this can be done for 16k to way beyond 100k Research in how RWKV handle the hidden state shows that it is barely used (imo: <5%??) meaning lots of headroom for scaling context size (This is actively being experimented on - we dun really know the limit yet)
- cs702 3y agoThank you. This is great to hear! I'm going to take a closer look :-)
- pico_creator 3y agoHi Everyone, I'm a regular involved with the RWKV community. AMA, on RWKV, and I will do my best to answer them here for the next hour (one at a time) PS: you can find our discord here : https://discord.gg/qt9egFA7ve https://discord.gg/qt9egFA7ve
- ftxbro 3y agoThe paper says it's comparable to transformers right now but that means that it might be better later. Do you guys have concrete plans to make it better? Are they secret? Also, what's the deal with that foundation? Is it a cult or like the new OpenAI that will turn closed or maybe it's to reap the value of random contributors to the project?
- FallDead 3y ago(Note: My comments do not represent or project those of my collaborators) I remember talking to Blink DL about this, I think the plan is just to build an ecosystem, provide more diversity in the DL space. There are plans to make a RWKV5, they are in the open in the RWKV5 channel. From an engineering standpoint I don't really see the "reap" the value of random contributors to the project. Most of us I believe ... are hackers and tinkerers that just want to learn and contribute and be apart of something that can change the current
- pico_creator 3y agoCompletely the opposite. - it is NOT backed directly or owned by any VC funded company - it is 100% OSS driven by the community (Apache 2 license) - it’s currently the top OSS chat model that can be used commercially on the chatbot arena score board - IMO it is undertrained, so expanding the training data alone will make it much better (however for the sake of this paper, we wanted to focus on architecture not training data, so we compared similarly trained models) And yes we do have multiple experiments and plans to make it better. It’s a list, and we will not know which is final until we try. Individual members can go to great lengths on what they are working on For better or worse, being truly OSS means our initiatives are more disorganized then a centrally planned org
- noduerme 3y agoI wrote a home-brew neural network around 2006, just to see what would happen. I'd read no papers on it, and just kind of made it up as I went along. The result was basically a cube of "neurons" which had stronger and weaker trigger points to their neighbors and would propagate "spark" to one or more neighbors based on the strength and direction of spark they got from their other neighbors. Each of the connections and weights had a "certainty" attached to it which would go up when the output was closer to the ideal and would go down when the output was garbage. When those certainty values hit zero, the weights for that neuron would be re-randomized. I quickly realized that the only thing this could really do was try to transform one 10x10 pixel bitmap into a different one. But it was fascinating that it actually seemed to "learn" patterns as they got baked in. What would be called "attention" now, though, basically didn't exist in that system. Once you started training it on something new it lost everything. For anyone wondering, it was written in Actionscript 3, and ridiculously, each neuron was bound to a display class that displayed as a semitransparent cube that lit up as the inputs propagated through them. A thoroughly ridiculous side project. But other than scaling that from 1000 neurons to billions, I'm curious what has changed about the concepts of pathing or tolerance to make these models better? Maybe my concept of the principle behind modern LLMs is too archaic or rooted in a cartoon understanding of our own wetware that I tried to reproduce. [edit: I'm describing an ancient home project... for anyone downvoting this, I'm more than receptive to hearing your reasons why it's stupid. I'm the first to admit it seems stupid!]
- cma 3y agoWhat determined whether it's output was "ideal" for the certainty measure to go off of? Backprop?
- noduerme 3y agoI'd need to go back and look at the code, but it was something primitive but similar to backprop. There was an evaluation routine that compared what lit up on the back of the cube to the desired output. The farther off it was, the more the certainty got docked from any of the [connected] neurons one layer up from the bad pixel, and that dragged down the certainties on the next layer and so on. It didn't have the ability to back check which neurons were involved in which specific output node. But I guess it was a scatter shot attempt at that.
- stellaathena 3y agoTwitter thread announcing the paper with some additional color commentary: https://twitter.com/AiEleuther/status/1660811179239849986 https://twitter.com/AiEleuther/status/1660811179239849986
- pico_creator 3y agoLayperson summary Good - substantially cheaper and faster to run / train - scalable to ridiculous context size Bad - you will need to change how you prompt this model sadly (it works differently)
- deleted 3y ago[deleted]
- deleted 3y ago[deleted]
- quickthrower2 3y agoMy goal this year: to understand what this is about :-)
- pico_creator 3y agoIf you are familiar with how transformer network works There is RWKV in 150 lines to help understand all the nitty gritty https://github.com/BlinkDL/ChatRWKV/blob/main/RWKV_in_150_lines.py https://github.com/BlinkDL/ChatRWKV/blob/main/RWKV_in_150_li...
- quickthrower2 3y agoAlmost familiar: one more Karpathy lecture to go. Thanks!
- p-e-w 3y agoThe paper lists the first author's institutional affiliation as "RWKV Foundation". However, I cannot find anything about this supposed "foundation" online, and as far as I can tell, the term RWKV originates in this very paper. What's going on here?
- pico_creator 3y agoIt does not exists (yet maybe) Blink is an individual and does not represent a company (aka not google, not eleuther, not <insert VC company>, etc) So he had to fill something up i guess haha The idea of a foundation has been tossed around. So it may eventually happen Also the RWKV project predates this paper by over 2 years - nothing is truely new and revealed today on this paper Your simply just finding out about it now in a formal paper consolidating various things that was learnt in this project
- ftxbro 3y ago> Blink is an individual and does not represent a company (aka not google, not eleuther, not <insert VC company>, etc) So he had to fill something up i guess haha I saw there is no one listed in the author list as independent researcher or no affiliation or whatever. It seems hard to believe, if it's a lot of random contributors on a discord channel. Did they all have to make up some kind of 'DBA' (doing business as) name?
- pico_creator 3y agoI would not really call it random. While it was open for feedback/contributions. There is a strong requirements for substantial contribution to the paper itself to qualify for authorship. So unfortunately that does limit it in part to folks who are more familiar in writing such papers, have the resources for doing benchmarks, and charting, etc And less so for folks who for eg, tinkers, contributed to dataset, porting the implementation to X lang, doing LoRa, or various other experiments that has nothing to do with the core architecture
- 3y ago
- RC_ITR 3y ago> We propose a novel model architecture, Receptance Weighted Key Value (RWKV), that combines the efficient parallelizable training of Transformers with the efficient inference of RNNs. Just to be clear to everyone: this is “use attention to train parameters, use recurrence for inference” It’s a very cool idea and I hope we get more interesting approaches to inference, but attention is here to stay for training.
- matteogrella 3y agoAs one of the authors, I'd like to clarify: the equations of the RWKV model enable computational parallelization, provided that the sequence is predetermined. This parallelization occurs during both the training and inference stages, specifically during the prompt reading process (consider it an "encoding"), right before the generation (or decoding phase).
- RC_ITR 3y agoHow can something recurrent be parallelized? > the equations of the RWKV model enable computational parallelization, provided that the sequence is predetermined. And sure, this is the core concept of self attention, no?
- hiddencost 3y agoEleutherAI is stunning. The only other papers I've seen with such a diversity of institutions are high energy particle physics.
- chrgy 3y agoHere is a summary of all comments by Transformers, wonder how RNN does: RWKV is a new language model architecture that is comparable to transformers in terms of performance. RWKV is more efficient than transformers, which makes it possible to train larger models on smaller datasets. The RWKV community is open source and welcomes contributions from anyone. There are plans to create larger versions of RWKV, but this will require more computational resources. Here are some additional details about the chinchilla law and the dataset problem: The chinchilla law states that the amount of data required to train a language model grows exponentially with the model size. This means that it is very expensive to train large language models, even with the latest hardware. The RWKV community is working on developing new methods for training large language models more efficiently. There are a number of datasets available to the RWKV community, including: The Pile: A massive dataset of text and code. The Chinchilla: A smaller dataset of text and code that is designed for training RWKV models. The Red Pajamas: A dataset of text and code that is being used to train a 65B RWKV model. These datasets are stored in a variety of locations, including: The RWKV GitHub repository The Chinchilla website The Red Pajamas website The RWKV community is constantly updating the datasets and adding new ones. If you are interested in contributing, please visit the RWKV GitHub repository.
- PoignardAzur 3y ago> The chinchilla law states that the amount of data required to train a language model grows exponentially with the model size. This means that it is very expensive to train large language models, even with the latest hardware. The RWKV community is working on developing new methods for training large language models more efficiently. There are a number of datasets available to the RWKV community, including: What? I though the chinchilla-optimal regime was something like "20 tokens per weight". That's not remotely exponential, even by the word's colloquial use.
- stellaathena 3y agoIt seems that the user is posting ChatGPT-generated text and not a real summary. It's complete nonsense, with about half of the sentences containing a factual error.
- zwaps 3y agoI wish this was written with more care. None of the symbols are defined. Worst of all, they use "channel dimension" in a sequence model. What even is a channel in a sequence of tokens? This happens as soon as you have a single person with CNN background on the team and it makes zero sense. What if you actually have channels in your data? What then?
- sorz 3y agoThey said the paper is still working in progress and will improve it. https://twitter.com/AiEleuther/status/1660811180901019648 https://twitter.com/AiEleuther/status/1660811180901019648
- pico_creator 3y agoIf you have more specific feedback, like a specific digram or page, and how it can be made better. I will gladly forward that info, to improve the paper draft. Because channel mixing, is a core component of this architecture, and that keyword "channel"is all over the place. I have no idea what is it you are critiquing specifically (i could not find the mention of "channel dimension" in the paper)