4 ms·

I thought ChatGPT and LLMs in general use vectored word embeddings - which captures that dog and puppy are related (so you can take puppy, subtract dog and add

I thought ChatGPT and LLMs in general use vectored word embeddings - which captures that dog and puppy are related (so you can take puppy, subtract dog and add cat to get kitten). Wouldn't this also capture that the Estonian token for dog is related to English token for dog etc? So ChatGPT is 'thinking' in some kind of semantic tokens not English.

ben_w 3y ago
That can work for some words, but not all words in all languages divide all of reality up the same way. Colour is a particularly easy way to notice this (ancient Greeks and their wine-dark seas), but also consider: in English "dog" isn't just an animal, but also a way to contemptuously describe a human. Can the word be used that way in Estonian? Or numbers: is 13 unlucky like in English, and is 4 unlucky like in some east asian languages? Or settlements: in the UK a settlement is a "city" if the monarch says so, apparently the Estonian word "linn" is used for both "a city" and "a town".
- helsinkiandrew 3y ago
  I thought some of that would be captured by the embeddings, crudely: ["english" + "dog" + "person"] = ["estonian" + "bad" + "person"]
tgv 3y ago
I had hoped that would be the case, but I tried multi-lingual clustering, and the clusters were neatly assigned per language, even though the meanings were incredibly close (e.g. short phrases literally meaning "cheap" or "unhealthy"). BTW, that was using ada-002.
a2128 3y ago
It doesn't actually have a token for Estonian dog, the word " koer" is tokenized as two tokens, " ko" and "er". In my language it's three tokens. In Russian it's seven(!). So any relations to dog and puppy in other languages would have to happen deeper within the model. Just as the tokenizer can't capture that a base64-encoded "dog" is related to a base64-encoded "puppy"

4 ms·

I thought ChatGPT and LLMs in general use vectored word embeddings - which captures that dog and puppy are related (so you can take puppy, subtract dog and add

by helsinkiandrew 3y ago

ben_w 3y ago
That can work for some words, but not all words in all languages divide all of reality up the same way. Colour is a particularly easy way to notice this (ancient Greeks and their wine-dark seas), but also consider: in English "dog" isn't just an animal, but also a way to contemptuously describe a human. Can the word be used that way in Estonian? Or numbers: is 13 unlucky like in English, and is 4 unlucky like in some east asian languages? Or settlements: in the UK a settlement is a "city" if the monarch says so, apparently the Estonian word "linn" is used for both "a city" and "a town".
- helsinkiandrew 3y ago
  I thought some of that would be captured by the embeddings, crudely: ["english" + "dog" + "person"] = ["estonian" + "bad" + "person"]
tgv 3y ago
I had hoped that would be the case, but I tried multi-lingual clustering, and the clusters were neatly assigned per language, even though the meanings were incredibly close (e.g. short phrases literally meaning "cheap" or "unhealthy"). BTW, that was using ada-002.
a2128 3y ago
It doesn't actually have a token for Estonian dog, the word " koer" is tokenized as two tokens, " ko" and "er". In my language it's three tokens. In Russian it's seven(!). So any relations to dog and puppy in other languages would have to happen deeper within the model. Just as the tokenizer can't capture that a base64-encoded "dog" is related to a base64-encoded "puppy"