7 ms·
Why would asking the question first improve quality? Is it because the model will be better aware of what info it Can and can’t throw away at each step? This se
by zaptrem 3y ago
Why would asking the question first improve quality? Is it because the model will be better aware of what info it Can and can’t throw away at each step? This seems like the opposite of transformers.
- LoganDark 3y agoRWKV does not work like transformers. The "transformer" part here is the training step. RWKV is an RNN with fixed-size state, so old information slightly decays each time it reads a new token. Hence the freshest memory is of the most recent tokens.