3 ms·
have an implementation of this over at https://github.com/lucidrains/memorizing-transformers-pytorch https://github.com/lucidrains/memorizing-transformers-pytor
by lucidrains 4y ago
have an implementation of this over at https://github.com/lucidrains/memorizing-transformers-pytorch https://github.com/lucidrains/memorizing-transformers-pytorc..., for any researcher exploring retrieval and memory with attention networks
- knrz 4y agoDude your repo’s are great, marvellous code quality too for cutting edge papers. Keep it up!
- lucidrains 4y agohey thanks! :^) hope someone makes the next big discovery with them
- silencedogood3 4y agoNeat! Can you explain what the KNn is doing? I can’t quite follow the paper.
- visarga 4y agoIt's a sparse attention scheme. They store and reuse activations thus "memorising" the past without the need for training. In order to keep the sequence short enough to fit into memory they only recall the k most similar memories from a much larger log.