4 ms·
I wonder how they've been intermixing different languages. Like is it all one "huge bucket" or do they tag languages so that it is "supposed" to know English vs
by bombcar 3y ago
I wonder how they've been intermixing different languages. Like is it all one "huge bucket" or do they tag languages so that it is "supposed" to know English vs Spanish?
- wouldbecouldbe 3y agoI don't think there is a language processor before or after it, just based upon the training data it's most likely tokens to return are spanish if question is largely in Spanish
- bombcar 3y agoIt works decently well as a translator, correct? I wonder how it's been doing that - is it "native" to being an LLM or is it somehow processing it before?
- astrange 3y agoIt's native. There is nothing in the LLM except more LLM. (And a safety model afterwards. And a tokenizer. But those things make it behave worse, not better.)
- tgv 3y agoAs far as I can tell, the language is part of the embedding. At least, when you use those for clustering, utterances in the same language group first.
- famouswaffles 3y agoSpanish tokens are just more tokens to predict. No tagging necessary. If the model can write in Spanish fluently then it saw enough Spanish language tokens to be competent. "Enough" is a sliding target. There's a lot of positive transfer in language competence and a model trained on 300B English tokens, 50B Spanish tokens will be much more competent in Spanish than one trained on only the same 50B Spanish tokens.
- kelseyfrog 3y agoIt would seem that way. I ran into this in the way conversations titles are automatically generated. There's a "Dockerfile fuera del contexto" hanging out in my history. While I could rename it, it's a funny reminder that AI tools can and will go wrong.
- fzzzy 3y agoThis seems to be a longstanding bug, I have many chats with the summarization in another language.