3 ms·
At this point the changelog is surely just stuff along the lines of "retrained with more data" and "slightly tweaked model architecture in a random way that imp
by ladberg 3y ago
At this point the changelog is surely just stuff along the lines of "retrained with more data" and "slightly tweaked model architecture in a random way that improved performance".
And out of curiosity: as someone with a lot of expertise and money on the line, how would you compare Google Translate with LLMs? And also smaller self-hosted models with bigger ones that require API access like OpenAI? Do they perform better or worse and are they cheaper or more expensive?
- KRAKRISMOTT 3y agoI assume they probably switched from BERT to GPT architecture at some point.
- HarHarVeryFunny 3y agoI wouldn't assume that. BERT uses the original encoder-decoder transformer architecture which takes advantage of the entire input sequence being available, so you have the forward context as well as backward context of a word to assist with translation. GPT uses the decoder-only transformer architecture and masked attention so that it only relies on backward context, which allows it to predict word-by-word. It's not a better architecture, just one adapted for different requirements. If you are able to use the forward context, then I'd assume that a BERT-like architecture would do better since it has more information available.