4 ms·
Examples of such architectures: RETRO (Retrieval-Enhanced Transformer) https://arxiv.org/abs/2112.04426 https://arxiv.org/abs/2112.04426 And much older (2016)
by halflings 4y ago
Examples of such architectures:
RETRO (Retrieval-Enhanced Transformer)
https://arxiv.org/abs/2112.04426 https://arxiv.org/abs/2112.04426
And much older (2016):
Differential Neural Network
https://www.deepmind.com/blog/differentiable-neural-computers https://www.deepmind.com/blog/differentiable-neural-computer...
- lossolo 4y agoRETRO has external source of data but is something entirely different to what is discussed in the paper. In simplistic terms, in RETRO architecture you store embeddings in an external database and then use kNN search. It is not an end-to-end model capable of things like those with stack or tape memory (what OP asked about), retrieval models are only good for limited Q/A atm.