7 ms·

One thing I'm keen to understand is: how well does attention hold across huge context sizes, with respect to the usual transformer models, and also these propos

One thing I'm keen to understand is: how well does attention hold across huge context sizes, with respect to the usual transformer models, and also these proposed RNN models? All these 2k/4k/8k context sizes that we've had recently are able to map pretty well to what a human could reasonably remember. What I mean is, you could ask a human to read some text with 8k tokens, and for the most part they could answer questions about the text coherently. But what about 32k contexts, or beyond? At some point, as token size increases, the ability of a human to give a highly precise and detailed answer decreases. They must start to generalize. A human could not read Infinite Jest in one pass and then answer details about every single sentence. But could a transformer, or a RNN? As the context grows, is it harder to keep a high granularity of detail? Or am I wrong in trying to think of these models the way I think about the human mind, and they are actually able to handle this problem just fine? I'm aware that we can cheat a bit, by adding a lookup step into an embedding database, to provide "infinite" context with "infinite" precision. But to me, that is analogous to a human looking up information in a library in order to answer a question. I'm interested in the inherent, emergent memory that these models have available to them in just one forward pass.

humbleharbinger 3y ago
Not sure if it answers your question but the paper notes something similar in its discussion of limitations: > the linear attention of RWKV leads to significant efficiency gains but still, it may also limit the model’s performance on tasks that require recalling minutiae information over very long contexts. This is due to the funneling of information through a single vector representation over many time steps, compared with the full information maintained by the quadratic attention of standard Transformers. In other words, the model’s recurrent architecture inherently limits its ability to “look back” at previous tokens, as opposed to traditional self-attention mechanisms. While learned time decay helps prevent the loss of information, it is mechanistically limited compared to full self- attention. > Another limitation of this work is the increased importance of prompt engineering in comparison to standard Transformer models. The linear attention mechanism used in RWKV limits the information from the prompt that will be carried over to the model’s continuation. As a result, carefully designed prompts may be even more crucial for the model to perform well on tasks
- pico_creator 3y ago
  Prompt design is definitely a huge one shifting to rwkv Sadly too many folks copy and paste what works for openAI and move on when it fails
foota 3y ago
The primary benefit of transformers is that they generally can reference things within the context, but RNNs have trouble with that. The trade off is that transformers have memory and compute requirements quadratic in the context length.
teruakohatu 3y ago
> But what about 32k contexts, or beyond? At some point, as token size increases, the ability of a human to give a highly precise and detailed answer decreases War and Peace is over 580,000 words long. Chapter one is ~2,020 words which encoded for GPT3 is ~2,956 tokens (lots of longer, older words and proper nouns eg. "scarlet-liveried footman" is six tokens), so we might expect the entire book to be ~750,000 tokens long. Many people could reason about the book in its entirety. They would not have an encyclopedic recall of random dates and side characters or be able to quote any passage, but they could perform deep analysis on it.
- dcl 3y ago
  Similarly, consider a series like A Song of Ice and Fire. A human reader is still consciously aware of (and waiting for) the answers to questions raised in the very first book. This is millions of tokens ago, and that's if our brains turn off when not reading the books. I think this highlights a hurdle on the path to more human-like AGI. We keep track of so much stuff for very long periods of time, albeit perhaps with some loss of fidelity. My guess is that there will need to be an advancement or two before we can can get an AI to read all of the ASIOF books so far and ask "What really happened at the Tower of Joy?"
  - zamnos 3y ago
    ASIOF spoiler below! > At the Tower of Joy, Ned Stark defeated three members of the Kingsguard and discovered his dying sister, Lyanna, who made him promise to protect her son, Jon Snow, whose true parentage remained a closely guarded secret. Seems like ChatGPT-3 already knows, unless there's a deeper secret that I'm not deep enough into ASIOF fandom to know.
    - jokteur 3y ago
      But this is becaude ASIOF was in the training dataset. Chatgpt wouldn't be able to say anything about this book if it wasn't in his dataset, and you wouldn't be able to have enough tokens to present the whole book to chatgpt.
Buttons840 3y ago
To answer your question: https://news.ycombinator.com/item?id=35904773 https://news.ycombinator.com/item?id=35904773 > we loaded the entire text of The Great Gatsby into Claude-Instant (72K tokens) and modified one line to say Mr. Carraway was “a software engineer that works on machine learning tooling at Anthropic.” When we asked the model to spot what was different, it responded with the correct answer in 22 seconds.
noduerme 3y ago
I know a guy who memorized "Fahrenheit 451" back in the '60s, and went on to memorize a dozen other books - I think it was a requirement of the anti-establishment cult he was in at the time. Anyhow, his recall is still pretty fantastic.
- tanseydavid 3y ago
  Sounds like your friend was sort of the Charles Babbage of Large Language Models.
chaxor 3y ago
I'll give a very simplistic answer to your question and skirt some detail. Essentially, Transformers can 'look at' everything within the context with perfect fidelity (only one 'hop' between tokens to 'get the info'), while RNNs struggle with this because the information gets 'muddied' with all of the tokens in between. Like telephone between you and one friend (transformer) vs you and a sea of people (RNN). Hopefully this gets the point across without the mathematics and a precise description.
- pico_creator 3y ago
  Yea using the analogy of the eyes can see the entire document Vs I need to memorise everything as it’s spoken out, and then answer on it Is a good approximate on the difference. The kicker though as people pointed out, there are individuals on earth who can memorise things to great length. So the same will apply here (need more training) It’s also why it is recommended by the community to ask the question first, then put your context document. In a document QnA task
  - soulofmischief 3y ago
    In the shower today I was thinking about Jungian archetypes as a visual programming language for the subconscious. For example, if someone says, "I should be a better mother", and hold this impression in their mind over time, the associations their brain makes with that image and various behaviors can be used as a guiding force during the physical reorganization which occurs when one creates a new habit, or adjusts a behavior or muscle memory. And these associations depend upon observations made by the individual, which they have either been taught or innately know can be categorized as mother-like. The brain is typically trained to make this association in the case of the mother by mirroring and attaching to the first female who exhibits nursing behavior. These archetypes are a visual representation of deeper mental models which we are primed to carry on between generations. And then I wondered how this might apply to neural models, if some system of archetypes naturally emerges, if they're useful for programming or memorization/compression, or how we might explore imbuing existing models with an archetypical system.
pico_creator 3y ago
One of the biggest issue with incredibly large context sizes is the lack of training / dataset meant specifically to use such large context sizes. And this apply to all models. So when a specific model now does badly even at 32k or 50k it’s hard to say if it’s an architecture design issue, or a dataset issue
- ofermend 3y ago
  Yes; and the amount of training examples that are really long (relative to the other training examples) becomes small really fast, so it's also a problem of the long-tail in a sense.
  - mike_hearn 3y ago
    It feels like this is where training on code is going to go from important to critical. Most human texts won't require you to look back 100,000 tokens to understand what it means, but if you dump 1000 source files one after the other than it will.
mlsu 3y ago
These things are very different from the human mind. With your mind, you read the words one by one, building up intuition as you go. As part of that, there's a kind of continuous synthesis of information. The synthesis is highly temporal, because as you go you are training, and the hardware is changing (your emotions, dictated by your stomach or the sound you can hear or...) underneath you. These things are very different. The mental model I have is that they put the entire book into a big hypercube and then *flash*, instantly, out comes the next token. And likewise, when they encounter text that is 8001 tokens long, token #1 is completely erased, gone, like it never existed.
- p-e-w 3y ago
  > These things are very different from the human mind. With your mind, you read the words one by one, building up intuition as you go. You're confusing your perception of how your mind works with how your mind actually works. Simply put, we have no idea how the human mind works. For all we know, its underlying principles could be very similar to LLMs, or they could be something nobody has thought of yet. But under no circumstances is human intuition about the human mind a reliable indicator for what is actually going on. This is like asking ChatGPT why it came up with a specific response. It will give you an answer, of course, but that answer is generated the same way all its answers are – ChatGPT doesn't suddenly turn on a magic "introspection mode" that allows it to examine its internals. And there's no reason to assume that humans have such an introspection mode either. In fact, many results from experimental psychology seem to indicate that humans don't understand their own mental operations at all.
  - anon373839 3y ago
    While it's true that we may not be able to observe the elemental building blocks of our own minds, metacognition is a real capability that people have and use.
    - p-e-w 3y ago
      Is there any hard evidence that "metacognition" reflects actual cognitive processes, rather than being something that the mind pulls out of its ass?
thomasahle 3y ago
> One thing I'm keen to understand is: how well does attention hold across huge context sizes, with respect to the usual transformer models, and also these proposed RNN models? We won't know before we've tried it. Reasoning by analogy with humans is not useful. In a year we'll have tried lots of things and can give a much better answer.
pico_creator 3y ago
Taking into account the limits of how much can be stored in a hidden state. My theory is that the model will start generalising the information stored once it starts going past its limits (like real life humans) So if we take books as an example, we probably do not remember every single word. But we might vaguely remember which section or book events occur Combine this with the agent model, and we may have an alternative for embeddings. Where we can ask which pages the model recall is relevant to the question. Bring those pages up again. And get the answers (Once again like how a human might answer in real life)
coppsilgold 3y ago
The word "attention" has been stretched pretty far to explain what is happening inside a transformer. What's actually happening is that every token embedding interacts with every other token embedding before it and as the product of this interaction (dot product + softmax) it takes a fraction of every other token embedding and adds it to itself. Technically, it's different transforms/functions of the embedding. You can view it as every token embedding mixing information from other embeddings into itself. Done ~100 times in parallel ("attention heads"), ~100 times in sequence (layers). As per GPT-3 model (175B).
- pico_creator 3y ago
  Agreed. IMO - A part of me even argue we should stop calling it attention (but what to call it instead is a mess) But since this was derived from apple lite attention paper. The name is gonna stick, due to a lack of better alternative
  - xiphias2 3y ago
    I believe the naming is perfect: the word attention clearly maches the intention of the model structure. It’s still clearly missing more efficiency, but we had to have ChatGPT / ChatGPT4 working and used to make the further research directions clear (increasing context length and decreasing hallucinations).
    - pico_creator 3y ago
      Haha, yea - naming things is hard Every-time someone comes up and say "this is not attention, because it does X and not Y" My response is, ok, what would you call it then? Because no one (including me) seem to to able to find a better term, that fits its use case.
  - candiodari 3y ago
    Naming is a perpetual problem. My issue is with "hallucination", which everyone takes to be the "problem" with GPT style networks making things up. Never mind that transformers are just trying to predict the next likely token, NOT the truth PLUS that they're trained from the internet. As everyone knows, the internet is not known for correctness and truth. If you want any neural network to figure out the truth independently, it'll obviously need the ability to go out into the real world and even needs to be allowed to experiment for most things. Hallucination used to mean the following. A basic neural network is: f(x) = y = repeat(nonlinearity(ax[0] + bx[1] + ...)) And then you adjust a, b, c, ... until y is reasonable, according to the cost function. But look! The very same backpropagation can adjust x[0], x[1] ... with the same cost function and only a small change in the code. This allows you to reverse the question neural networks answer. Which can be an incredibly powerful way to answer questions. And that used to be called hallucination in Neural networks. Instead of "change these network weights to transform x into y, keeping x constant" you ask "change x to transform x into y, keeping the network weights constant". Now it's impossible finding half the papers on the this topic. AARGH!
Orange1688 3y ago
I think what you're saying is warranted, but you should be very careful putting machine learning architectures in the context of the human mind. They are inherently different, although their output may be similar at some points. What you're saying addressed this directly. Machine learning is optimizing mathematical weights and biases, and the human mind is messy wetware. One of the first things you will read in any machine learning textbook is to throw away the notion of AI trying to simulate the human mind.
tlb 3y ago
People condense information from more than several thousand tokens ago into higher level chunks. So the whole first chapter of Infinite Jest becomes “child tennis prodigy is weird and annoying”. It’s not clear how to learn this. Transformers work so well because there are only a few layers between input tokens and output. Adding a summarization step is a whole new research project.
candiodari 3y ago
Okay ... the answer really is "much better than other RNNs of any type". And transformers do: LSTMs and GRUs networks effectively have a context size of 1. Note that what people claim here is not true: just because you go past the context window does NOT mean everything is forgotten, it means that you're back to RNN levels of performance. In transformers, just like in RNNs, processing the nth token influences the nth token AND the (n+1)th token. So the influence of the nth token "slowly dies down". It influences the (n+1)th token a lot, the (n+2)th token a bit less and so on. The special thing about attention is that it doesn't start dying until you go past the context size. Or with a bit of math notation. If a neural network is a function f, then: RNN: f(x[n]) = f( f(x[n-1]), x[n] ) Transformer: f(x[n]) = f(sum(i=0...context size, f(x-i)), x[n]) The difference between LSTMs and GRUs is in the structure of f. And "sum" is, like a lot of things in here, uh, basically accurate but missing a lot of detail.
- MontyCarloHall 3y ago
  RNNs have a theoretically infinite context size, but in practice it’s limited by vanishing gradients due to too much recursion. That’s why the recursive units are usually LSTMs or GRUs that have explicit functionality to cut off the recursion, using a learnable threshold that’s a function of both the current and recursive inputs. In your example, x[n-1] = f(x[n-2], x[n-1]). So really, your expression should be RNN: x[n] = f( … f( f( f( f(k0, x[0]), x[1]), x[2]), x[3]), …, x[n-1]) where k0 is a learnable parameter.
stevetesla 3y ago
Hey side question here, are there any papers or anything really I can read on the part about cheating by using a lookup that you mentioned?

humbleharbinger 3y ago

Not sure if it answers your question but the paper notes something similar in its discussion of limitations: > the linear attention of RWKV leads to significant efficiency gains but still, it may also limit the model’s performance on tasks that require recalling minutiae information over very long contexts. This is due to the funneling of information through a single vector representation over many time steps, compared with the full information maintained by the quadratic attention of standard Transformers. In other words, the model’s recurrent architecture inherently limits its ability to “look back” at previous tokens, as opposed to traditional self-attention mechanisms. While learned time decay helps prevent the loss of information, it is mechanistically limited compared to full self- attention. > Another limitation of this work is the increased importance of prompt engineering in comparison to standard Transformer models. The linear attention mechanism used in RWKV limits the information from the prompt that will be carried over to the model’s continuation. As a result, carefully designed prompts may be even more crucial for the model to perform well on tasks

pico_creator 3y ago
Prompt design is definitely a huge one shifting to rwkv Sadly too many folks copy and paste what works for openAI and move on when it fails

foota 3y ago

The primary benefit of transformers is that they generally can reference things within the context, but RNNs have trouble with that. The trade off is that transformers have memory and compute requirements quadratic in the context length.

teruakohatu 3y ago

> But what about 32k contexts, or beyond? At some point, as token size increases, the ability of a human to give a highly precise and detailed answer decreases War and Peace is over 580,000 words long. Chapter one is ~2,020 words which encoded for GPT3 is ~2,956 tokens (lots of longer, older words and proper nouns eg. "scarlet-liveried footman" is six tokens), so we might expect the entire book to be ~750,000 tokens long. Many people could reason about the book in its entirety. They would not have an encyclopedic recall of random dates and side characters or be able to quote any passage, but they could perform deep analysis on it.

dcl 3y ago
Similarly, consider a series like A Song of Ice and Fire. A human reader is still consciously aware of (and waiting for) the answers to questions raised in the very first book. This is millions of tokens ago, and that's if our brains turn off when not reading the books. I think this highlights a hurdle on the path to more human-like AGI. We keep track of so much stuff for very long periods of time, albeit perhaps with some loss of fidelity. My guess is that there will need to be an advancement or two before we can can get an AI to read all of the ASIOF books so far and ask "What really happened at the Tower of Joy?"
- zamnos 3y ago
  ASIOF spoiler below! > At the Tower of Joy, Ned Stark defeated three members of the Kingsguard and discovered his dying sister, Lyanna, who made him promise to protect her son, Jon Snow, whose true parentage remained a closely guarded secret. Seems like ChatGPT-3 already knows, unless there's a deeper secret that I'm not deep enough into ASIOF fandom to know.
  - jokteur 3y ago
    But this is becaude ASIOF was in the training dataset. Chatgpt wouldn't be able to say anything about this book if it wasn't in his dataset, and you wouldn't be able to have enough tokens to present the whole book to chatgpt.

Buttons840 3y ago

To answer your question: https://news.ycombinator.com/item?id=35904773 https://news.ycombinator.com/item?id=35904773 > we loaded the entire text of The Great Gatsby into Claude-Instant (72K tokens) and modified one line to say Mr. Carraway was “a software engineer that works on machine learning tooling at Anthropic.” When we asked the model to spot what was different, it responded with the correct answer in 22 seconds.

noduerme 3y ago

I know a guy who memorized "Fahrenheit 451" back in the '60s, and went on to memorize a dozen other books - I think it was a requirement of the anti-establishment cult he was in at the time. Anyhow, his recall is still pretty fantastic.

tanseydavid 3y ago
Sounds like your friend was sort of the Charles Babbage of Large Language Models.

chaxor 3y ago

I'll give a very simplistic answer to your question and skirt some detail. Essentially, Transformers can 'look at' everything within the context with perfect fidelity (only one 'hop' between tokens to 'get the info'), while RNNs struggle with this because the information gets 'muddied' with all of the tokens in between. Like telephone between you and one friend (transformer) vs you and a sea of people (RNN). Hopefully this gets the point across without the mathematics and a precise description.

pico_creator 3y ago
Yea using the analogy of the eyes can see the entire document Vs I need to memorise everything as it’s spoken out, and then answer on it Is a good approximate on the difference. The kicker though as people pointed out, there are individuals on earth who can memorise things to great length. So the same will apply here (need more training) It’s also why it is recommended by the community to ask the question first, then put your context document. In a document QnA task
- soulofmischief 3y ago
  In the shower today I was thinking about Jungian archetypes as a visual programming language for the subconscious. For example, if someone says, "I should be a better mother", and hold this impression in their mind over time, the associations their brain makes with that image and various behaviors can be used as a guiding force during the physical reorganization which occurs when one creates a new habit, or adjusts a behavior or muscle memory. And these associations depend upon observations made by the individual, which they have either been taught or innately know can be categorized as mother-like. The brain is typically trained to make this association in the case of the mother by mirroring and attaching to the first female who exhibits nursing behavior. These archetypes are a visual representation of deeper mental models which we are primed to carry on between generations. And then I wondered how this might apply to neural models, if some system of archetypes naturally emerges, if they're useful for programming or memorization/compression, or how we might explore imbuing existing models with an archetypical system.

pico_creator 3y ago

One of the biggest issue with incredibly large context sizes is the lack of training / dataset meant specifically to use such large context sizes. And this apply to all models. So when a specific model now does badly even at 32k or 50k it’s hard to say if it’s an architecture design issue, or a dataset issue

ofermend 3y ago
Yes; and the amount of training examples that are really long (relative to the other training examples) becomes small really fast, so it's also a problem of the long-tail in a sense.
- mike_hearn 3y ago
  It feels like this is where training on code is going to go from important to critical. Most human texts won't require you to look back 100,000 tokens to understand what it means, but if you dump 1000 source files one after the other than it will.

mlsu 3y ago

These things are very different from the human mind. With your mind, you read the words one by one, building up intuition as you go. As part of that, there's a kind of continuous synthesis of information. The synthesis is highly temporal, because as you go you are training, and the hardware is changing (your emotions, dictated by your stomach or the sound you can hear or...) underneath you. These things are very different. The mental model I have is that they put the entire book into a big hypercube and then *flash*, instantly, out comes the next token. And likewise, when they encounter text that is 8001 tokens long, token #1 is completely erased, gone, like it never existed.

p-e-w 3y ago
> These things are very different from the human mind. With your mind, you read the words one by one, building up intuition as you go. You're confusing your perception of how your mind works with how your mind actually works. Simply put, we have no idea how the human mind works. For all we know, its underlying principles could be very similar to LLMs, or they could be something nobody has thought of yet. But under no circumstances is human intuition about the human mind a reliable indicator for what is actually going on. This is like asking ChatGPT why it came up with a specific response. It will give you an answer, of course, but that answer is generated the same way all its answers are – ChatGPT doesn't suddenly turn on a magic "introspection mode" that allows it to examine its internals. And there's no reason to assume that humans have such an introspection mode either. In fact, many results from experimental psychology seem to indicate that humans don't understand their own mental operations at all.
- anon373839 3y ago
  While it's true that we may not be able to observe the elemental building blocks of our own minds, metacognition is a real capability that people have and use.
  - p-e-w 3y ago
    Is there any hard evidence that "metacognition" reflects actual cognitive processes, rather than being something that the mind pulls out of its ass?

thomasahle 3y ago

> One thing I'm keen to understand is: how well does attention hold across huge context sizes, with respect to the usual transformer models, and also these proposed RNN models? We won't know before we've tried it. Reasoning by analogy with humans is not useful. In a year we'll have tried lots of things and can give a much better answer.

pico_creator 3y ago

Taking into account the limits of how much can be stored in a hidden state. My theory is that the model will start generalising the information stored once it starts going past its limits (like real life humans) So if we take books as an example, we probably do not remember every single word. But we might vaguely remember which section or book events occur Combine this with the agent model, and we may have an alternative for embeddings. Where we can ask which pages the model recall is relevant to the question. Bring those pages up again. And get the answers (Once again like how a human might answer in real life)

coppsilgold 3y ago

The word "attention" has been stretched pretty far to explain what is happening inside a transformer. What's actually happening is that every token embedding interacts with every other token embedding before it and as the product of this interaction (dot product + softmax) it takes a fraction of every other token embedding and adds it to itself. Technically, it's different transforms/functions of the embedding. You can view it as every token embedding mixing information from other embeddings into itself. Done ~100 times in parallel ("attention heads"), ~100 times in sequence (layers). As per GPT-3 model (175B).

pico_creator 3y ago
Agreed. IMO - A part of me even argue we should stop calling it attention (but what to call it instead is a mess) But since this was derived from apple lite attention paper. The name is gonna stick, due to a lack of better alternative
- xiphias2 3y ago
  I believe the naming is perfect: the word attention clearly maches the intention of the model structure. It’s still clearly missing more efficiency, but we had to have ChatGPT / ChatGPT4 working and used to make the further research directions clear (increasing context length and decreasing hallucinations).
  - pico_creator 3y ago
    Haha, yea - naming things is hard Every-time someone comes up and say "this is not attention, because it does X and not Y" My response is, ok, what would you call it then? Because no one (including me) seem to to able to find a better term, that fits its use case.
- candiodari 3y ago
  Naming is a perpetual problem. My issue is with "hallucination", which everyone takes to be the "problem" with GPT style networks making things up. Never mind that transformers are just trying to predict the next likely token, NOT the truth PLUS that they're trained from the internet. As everyone knows, the internet is not known for correctness and truth. If you want any neural network to figure out the truth independently, it'll obviously need the ability to go out into the real world and even needs to be allowed to experiment for most things. Hallucination used to mean the following. A basic neural network is: f(x) = y = repeat(nonlinearity(ax[0] + bx[1] + ...)) And then you adjust a, b, c, ... until y is reasonable, according to the cost function. But look! The very same backpropagation can adjust x[0], x[1] ... with the same cost function and only a small change in the code. This allows you to reverse the question neural networks answer. Which can be an incredibly powerful way to answer questions. And that used to be called hallucination in Neural networks. Instead of "change these network weights to transform x into y, keeping x constant" you ask "change x to transform x into y, keeping the network weights constant". Now it's impossible finding half the papers on the this topic. AARGH!

Orange1688 3y ago

I think what you're saying is warranted, but you should be very careful putting machine learning architectures in the context of the human mind. They are inherently different, although their output may be similar at some points. What you're saying addressed this directly. Machine learning is optimizing mathematical weights and biases, and the human mind is messy wetware. One of the first things you will read in any machine learning textbook is to throw away the notion of AI trying to simulate the human mind.

tlb 3y ago

People condense information from more than several thousand tokens ago into higher level chunks. So the whole first chapter of Infinite Jest becomes “child tennis prodigy is weird and annoying”. It’s not clear how to learn this. Transformers work so well because there are only a few layers between input tokens and output. Adding a summarization step is a whole new research project.

candiodari 3y ago

Okay ... the answer really is "much better than other RNNs of any type". And transformers do: LSTMs and GRUs networks effectively have a context size of 1. Note that what people claim here is not true: just because you go past the context window does NOT mean everything is forgotten, it means that you're back to RNN levels of performance. In transformers, just like in RNNs, processing the nth token influences the nth token AND the (n+1)th token. So the influence of the nth token "slowly dies down". It influences the (n+1)th token a lot, the (n+2)th token a bit less and so on. The special thing about attention is that it doesn't start dying until you go past the context size. Or with a bit of math notation. If a neural network is a function f, then: RNN: f(x[n]) = f( f(x[n-1]), x[n] ) Transformer: f(x[n]) = f(sum(i=0...context size, f(x-i)), x[n]) The difference between LSTMs and GRUs is in the structure of f. And "sum" is, like a lot of things in here, uh, basically accurate but missing a lot of detail.

MontyCarloHall 3y ago
RNNs have a theoretically infinite context size, but in practice it’s limited by vanishing gradients due to too much recursion. That’s why the recursive units are usually LSTMs or GRUs that have explicit functionality to cut off the recursion, using a learnable threshold that’s a function of both the current and recursive inputs. In your example, x[n-1] = f(x[n-2], x[n-1]). So really, your expression should be RNN: x[n] = f( … f( f( f( f(k0, x[0]), x[1]), x[2]), x[3]), …, x[n-1]) where k0 is a learnable parameter.

stevetesla 3y ago

Hey side question here, are there any papers or anything really I can read on the part about cheating by using a lookup that you mentioned?