3 ms·
I don't think they are generations, but rather samples from The Pile that are semantically close to the input. Actually, as far as I can tell - the RETRO arch
by ShamelessC 4y ago
I don't think they are generations, but rather samples from The Pile that are semantically close to the input.
Actually, as far as I can tell - the RETRO arch itself isn't trained in this article. It focuses more on how to build the retrieval system with a fast KNN index over all of the Pile.