4 ms·
> It's not a transformer model though, and old context fades away much faster / is harder to recall because all the new context is layered directly on top of it
by zozbot234 3y ago
> It's not a transformer model though, and old context fades away much faster / is harder to recall because all the new context is layered directly on top of it.
That's a well known limitation. But if you actually know that a "context" comprises multiple sentences (or other elements of syntax) and that any ordering among them is completely arbitrary, the principled approach is to RNN-parse them all in parallel and sum the activations you end up with as vectors - like in bag-of-words model, essentially enforcing commutativity on the network: that's pretty much how attention-based models work under the hood. The really basic intuition is just that a commutative and associative function can be expressed (hence "learned") in terms of vector sum modulo some arbitrary conversion of the inputs and outputs.
- LoganDark 3y ago> That's a well known limitation. I know. I did a lot of work on state handling in rwkv.cpp