3 ms·

I agree with this. Although there are APIs used in production whose output changes/evolves over time. First one that comes to mind is Google Translate. We spe

by cj 3y ago

I agree with this. Although there are APIs used in production whose output changes/evolves over time. First one that comes to mind is Google Translate. We spend 6-figures with the Google Translate API annually, and recently we went back and checked if Google Translate is improving/changing the translations over time, and found that they indeed are (presumably as they improve their internal models, which aren't versioned and isn't exposed in a changelog anywhere). The majority of translations were different for the same content today compared to 6 months ago. I don't particularly agree with this approach. Speaking as a power user of Google Translate API, it would be nice to be able to pin to specific version/model and then manually upgrade versions (with a changelog to understand what's changing under the hood).

ladberg 3y ago
At this point the changelog is surely just stuff along the lines of "retrained with more data" and "slightly tweaked model architecture in a random way that improved performance". And out of curiosity: as someone with a lot of expertise and money on the line, how would you compare Google Translate with LLMs? And also smaller self-hosted models with bigger ones that require API access like OpenAI? Do they perform better or worse and are they cheaper or more expensive?
- KRAKRISMOTT 3y ago
  I assume they probably switched from BERT to GPT architecture at some point.
  - HarHarVeryFunny 3y ago
    I wouldn't assume that. BERT uses the original encoder-decoder transformer architecture which takes advantage of the entire input sequence being available, so you have the forward context as well as backward context of a word to assist with translation. GPT uses the decoder-only transformer architecture and masked attention so that it only relies on backward context, which allows it to predict word-by-word. It's not a better architecture, just one adapted for different requirements. If you are able to use the forward context, then I'd assume that a BERT-like architecture would do better since it has more information available.

by cj 3y ago

ladberg 3y ago

At this point the changelog is surely just stuff along the lines of "retrained with more data" and "slightly tweaked model architecture in a random way that improved performance". And out of curiosity: as someone with a lot of expertise and money on the line, how would you compare Google Translate with LLMs? And also smaller self-hosted models with bigger ones that require API access like OpenAI? Do they perform better or worse and are they cheaper or more expensive?

KRAKRISMOTT 3y ago
I assume they probably switched from BERT to GPT architecture at some point.
- HarHarVeryFunny 3y ago
  I wouldn't assume that. BERT uses the original encoder-decoder transformer architecture which takes advantage of the entire input sequence being available, so you have the forward context as well as backward context of a word to assist with translation. GPT uses the decoder-only transformer architecture and masked attention so that it only relies on backward context, which allows it to predict word-by-word. It's not a better architecture, just one adapted for different requirements. If you are able to use the forward context, then I'd assume that a BERT-like architecture would do better since it has more information available.