3 ms·
Interestingly, there's very little discussion of the Transformer architecture aside from the known text implementations like AI Dungeon 2 (using GPT-2/GPT-3).
by minimaxir 6y ago
Interestingly, there's very little discussion of the Transformer architecture aside from the known text implementations like AI Dungeon 2 (using GPT-2/GPT-3).
There's no mechanical restriction from using it on text only (with OpenAI's ImageGPT being an extreme example), and it will be interesting to see how/if it competes with LSTMs for nontext domains.
- Der_Einzige 6y agoIt seems to not support going in the reverse though, e.g. giving a vector in a low dimensional space and inverse transforming to a hypothetical generated document. Closest to this is the context starting text but it's not the same. To be clear, I'm describing using transformers as autoencoders