3 ms·
> This idea is quite similar to retrieval transformers and Hopfield networks which have been known and published for several years now. It's not really that nov
by ShamelessC 3y ago
> This idea is quite similar to retrieval transformers and Hopfield networks which have been known and published for several years now. It's not really that novel.
Is it? I had thought retrieval transformers "merely" used retrieval as a backend of sorts rather than a substitute for the attention itself?
- mxwsn 3y agoYeah, RETRO [0] embeds all an entire question/prompt, and searches for similar text passages with k-NN, then does further processing. This can kind of be understood as attention on paragraphs. This preprint instead does k-NN and calls it attention on single tokens. So not the same. But similar. [0] https://jalammar.github.io/illustrated-retrieval-transformer/ https://jalammar.github.io/illustrated-retrieval-transformer...
- ShamelessC 3y agoAh, I see - thanks for the clarification.
- make3 3y agoretro doesn't attend itself, which is a big difference