3 ms·
As a sidenote and somewhat related: Suppose we train BackwardsGPT on the same sequences of tokens as GPT but reversed. The question is: What would this model be
by golol 4y ago
As a sidenote and somewhat related: Suppose we train BackwardsGPT on the same sequences of tokens as GPT but reversed. The question is: What would this model be good at? Because it would be ver bad at most things. But some tasks might be easier. By understanding what BackwardsGPT can do, we can understand what GPT naturally can't do.
- chessgecko 4y agoMaybe not super relevant but people used to do that for translation with RNNs. They reversed the input so that the hidden states at the end would be close to what was needed to start translating. Nobody really did it enc-dec transformers but it'd be a funny thought to pretrain a dec only transformer with and without to see if it helped.