3 ms·
state = context The difference between SSMs and GPTs here is how that state/context scales. Per usual in engineering, there are big trade offs!
by a_wild_dandan 3y ago
state = context
The difference between SSMs and GPTs here is how that state/context scales. Per usual in engineering, there are big trade offs!
- kelseyfrog 3y agoI'm not following. State is a multi-dimensional vector and context is a list of tokens. State is perturbed by A and Bx(t), while context is appended to by sampling the predicted token distribution.