3 ms·
Want reliable AI? Stop approximating memory with attention and build reliable memory into the model directly.
by inciampati 2y ago
Want reliable AI? Stop approximating memory with attention and build reliable memory into the model directly.
- logicchains 2y agoStandard LLM quadratic attention isn't an approximation, it's perfect recall. Approaches that compress that memory down into a fixed-size state are an approximation, and generally perform worse, that's why linear transformers aren't widely used.