3 ms·
Maybe not super relevant but people used to do that for translation with RNNs. They reversed the input so that the hidden states at the end would be close to wh
by chessgecko 4y ago
Maybe not super relevant but people used to do that for translation with RNNs. They reversed the input so that the hidden states at the end would be close to what was needed to start translating. Nobody really did it enc-dec transformers but it'd be a funny thought to pretrain a dec only transformer with and without to see if it helped.