3 ms·
I assume they probably switched from BERT to GPT architecture at some point.
by KRAKRISMOTT 3y ago
I assume they probably switched from BERT to GPT architecture at some point.
- HarHarVeryFunny 3y agoI wouldn't assume that. BERT uses the original encoder-decoder transformer architecture which takes advantage of the entire input sequence being available, so you have the forward context as well as backward context of a word to assist with translation. GPT uses the decoder-only transformer architecture and masked attention so that it only relies on backward context, which allows it to predict word-by-word. It's not a better architecture, just one adapted for different requirements. If you are able to use the forward context, then I'd assume that a BERT-like architecture would do better since it has more information available.