12 ms·

Finland's ChatGPT equivalent begins to think in Estonian as well

116 points by jruohonen 3y ago

gumballindie 3y ago
“think”. Think of the average person and then think that half of the world is less smart than that. Those people are the target audience for news articles and products that use terms such as “think” and fear mongering when it comes to a”i”.
- IshKebab 3y ago
  This is pointless and frankly baseless pedantry. There's no reason to artificially restrict the word "think" to squashy brains.
  - jraph 3y ago
    I don't like your parent comment but there's merit in avoiding people believing those things actually think and are actually intelligent. In "thinking", there's reasoning, which those things currently don't. They output text statistically and that's what mass media should tell their audience.
    - ben_w 3y ago
      What is "reasoning"? Wikipedia surprised me with this opening paragraph: > Reason is the capacity of applying logic consciously by drawing conclusions from new or existing information, with the aim of seeking the truth, with more than 10 or more than 3 evidence, 99.9% reliability or 87.5% reliability. Apart from the words "consciously" (which has 100 or so different definitions, some of which do and some of which do not apply to AI), and "aim" which can be contentious depending upon if you feel that requires consciousness or can include training rewards… I think the best of them is at the lower of those two mysterious percentages I've never seen discussed before. And that's despite LLMs being wildly bad at, specifically, logic. Though even this specific weakness in general seems like a very easy thing to get around by hybridising it with the computer it has to run on e.g. by telling it how to use a compiler, as logic is the foundation of the arithmetic that the transformers use to do natural language comprehension in the first place.
      - vikramkr 3y ago
        That opening for the Wikipedia article looks problematic - it's grammatically nonsense and has random numbers in it - someone should probably open a talk discussion on what happened Edit: yeah that looks like a recent edit. I thought Wikipedia was supposed to be pretty good about flagging this sort of stuff? Does anyone know how to raise the problem to those super editors that run the site? I assume they won't appreciate a random person undoing an edit https://en.wikipedia.org/w/index.php?title=Reason&diff=prev&oldid=1179925542 https://en.wikipedia.org/w/index.php?title=Reason&diff=prev&...
        ben_w 3y ago
        If I was on desktop right now, I'd put the new bit after the existing citation and add a new {{citation needed}} for the addition. Don't worry too much about diving in, so long as you comment your changes appropriately when committing, that's the point of the place and how it does the quality control in the first place :)
        vikramkr 3y ago
        Leaving a citation needed doesn't sound like the right thing to do for the definition of something as big as reason - esp with the grammatical confusion idk what you'd even edit it to to make sense. Like what is a three evidence or a reliability and what do those reliability numbers mean. And the sentence describing those numbers doesn't make sense - is it 10 or 3 or which of those percentages? Idk maybe it should just be undone
        ben_w 3y ago
        Then be bold with the edits and delete it, just say why in the comment box. The deletion itself is as easy to undo as to perform, and if you say why you're doing it, someone bringing that part back can do a better job next time. :)
      - Sharlin 3y ago
        > And that's despite LLMs being wildly bad at, specifically, logic. I'd say "wildly bad" is wildly exaggerated given that GPT-4 exists, even if you define "logic" as "logical puzzles encoded in natural language" and not "the thing that allows an LLM to reason highly efficiently about what token comes next".
    - Sharlin 3y ago
      What is, then, the qualitative difference between a GOFAI automated reasoning system [1], like Coq, and an LLM? Why does Coq do reasoning but GPT-4 doesn't? Does it only count as reasoning if hand-coded by a human? Or did you have another definition of "reasoning" in mind? Statistical or probabilistic reasoning is still reasoning, and indeed essentially mandatory to make sense of the real world, and is what humans do. Indeed the failure of painstakingly hand-trained symbolic reasoning engines to model reality (as opposed to just simple toy worlds) was the entire reason for the rise of neural networks. The entire point of a neural network is to efficiently compress large amounts of statistical data into a model with as much predictive power as possible. Somewhere within GPT-4's many layers there has to exist something like Bayes networks and other ultra-efficient representations of the vast network of correlations present in the training corpus – otherwise there's no way it could do what it does. [1] https://en.wikipedia.org/wiki/Automated_reasoning https://en.wikipedia.org/wiki/Automated_reasoning
      - jraph 3y ago
        I didn't know "automated reasoning" was an actual technical term. I didn't mean this by "reasoning". To me, Coq does not reason (in the meaning I intended). It checks proofs are well-formed and assists proof writing. But yes, there's some implementation of logic in Coq. It applies (a specific form of) logic. Again, LLMs output probable text. They don't do logic. They do statistics on language. Coq works on the actual objects themselves directly, not on the text that presents them. LLMs usually fail to answer to riddles correctly for this reason. We saw plenty of occurrences of this in recent times. Each time LLMs are benchmarked, people think of trying riddles and it works to some extent because something that looks like these riddles are in the training set but it eventually fails. Humans also make mistakes when solving riddles, but not for the same reasons (the process is not the same, they are not "processing text", the are really processing logic, even if the process is fallible).
        Sharlin 3y ago
        So you just replaced the nebulously defined word "think" with an equally nebulously defined word, "reason"? Not a very convincing argument. You do not know what humans do, no human does, as we don't have such an access to our internal representations. You neither know what LLMs do, no human does, as even though we have full access to their internals, we have little to no idea how to interpret said internals. The most interesting thing about LLMs is the degree that they have learned to model the real world while only being exposed to words. Because those words are highly correlated not just with each other, but with an external entity unknown to the LLM, it's very likely that by learning to model that external entity, the LLM's word-predicting power will increase. LLMs are likely to have converged on a world-model simply because it makes the most sense given the available evidence. This is no different from how human scientists create models of things they cannot directly see based on what they can see. And of course, "seeing" is not directly perceiving the external reality either! Sensory data merely correlates with what's really going on, and the brain has learned to model the world – an unknown external entity – based on the correlations that exist in the sensory language.
  - Y_Y 3y ago
    Is it really baseless? I'd say it depends on your theory of mind and is a serious question which generates interesting discussion. Myself I don't consider "think" an appropriate term, since the super-Markov-chain types of LLM right now seem to miss out on important aspects of cognition, particularly synthesis of two disparate ideas. It's hard to test this since you need to ask it about things which aren't already discussed in its corpus, but even GPT-4 tends to disappoint me when I ask it weird questions that have straightforward answers but probably aren't written about much. As an example, I asked how I could go about obtaining a large quantity of animal skulls cheaply. It directed me to taxidermists and specialists dealing in bones of exotic animals, whereas it could be much easier to go to an abbatoir, or even buy whole animals (since the exotic ones are so expensive).
  - s_dev 3y ago
    Google doesn't 'think' and neither does ChatGPT. I will give you a hard limit on what a machine can't compute -- it will never figure out given a programming script and an input when that program will stop. A human can do it by thinking through the problem -- whatever that process/phenomena of thought actually is it can think outside of itself and that fundamentally seperates it from a program even one as sophisticated as ChatGPT.
    - astrange 3y ago
      The limits on the halting problem are the same for a human and a computer. People aren't hypercomputers.
    - ben_w 3y ago
      > A human can do it by thinking through the problem No, we cannot. Imagine a program that looks for the smallest counterexample to the Collatz conjecture. Nobody knows if it will ever terminate.
    - xcv123 3y ago
      > A human can do it by thinking through the problem No. Humans cannot solve the halting problem. It is unsolvable. Cannot be solved manually or automatically. > It will never figure out given a programming script and an input when that program will stop It can for some programs. Just like humans can for some programs but not all.
    - jiggawatts 3y ago
      Have you verified this hypothesis of yours? In my experience GPT4 especially can definitely "think through" what a program does and arrive at the result. Even the significantly dumber ChatGPT can do this: https://chat.openai.com/share/3ee29c5e-6cb4-485f-b0b4-29e982463687 https://chat.openai.com/share/3ee29c5e-6cb4-485f-b0b4-29e982... People mistakenly think that it's just regurgitating what it has seen on the web, but even if you don't prompt it in English it'll still a) solve the problem correctly, and b) do it in the target language: https://chat.openai.com/share/5b4e6e99-3fd8-4316-811c-2a9dec776f63 https://chat.openai.com/share/5b4e6e99-3fd8-4316-811c-2a9dec... If that's not impressive enough, the ability to round-trip from script-to-poem and then poem-to-script should blow your mind: What might have been the original PowerShell script snippet that inspired this poem? Be as terse as possible, using aliases where possible. In the realm of numbers, a tale begins, Where $x is a 7, with a smile it grins. From one to ten, the journey we thread, Adding 7 to $x, step by step we tread. $x grows and grows, like a tree so tall, With every step, it doesn't fall. Once the journey ends, not a moment we relax, Triple the fun, we do with $x. In this dance of digits and the arithmetic song, With $x as our guide, we whirl along. From start to finish, through each complex, This is the tale of our friend, $x. Here's a terse PowerShell script snippet that inspired the poem: ```powershell $x = 7 1..10 | % {$x += 7} $x *= 3 ```
  - ggm 3y ago
    No. You're wrong. It's important and we need to stomp on random use of language like this, which has consequences. It's not thinking and you are dangerously wrong if you don't understand why. The danger isn't AGI it's misplaced belief in the state of AI. Call this thinking and you skew perceptions.
    - HPsquared 3y ago
      All they need to do is put the word in quotes. ChatGPT Equivalent begins to "think" in Estonian. Any better suggestions? Thinking itself isn't exactly a rigorously-defined word. Do animals think?
  - Lio 3y ago
    Well rather than argue it out with everyone I just went straight to source and asked ChatGPT what it was thinking about: > I don't have thoughts or consciousness like a human. I'm here to provide information and assist with any questions or tasks you have. Is there something specific you'd like to know or discuss? So in its own words it doesn't "think"... ...which just conclusively proves that its worked out how to lie and is about to kill us all! :P
    - IshKebab 3y ago
      That's just what it's been told to say. I can also say it. Doesn't make it true. Do animals think? Have any told you so?
v4dok 3y ago
This is important work. And think about African or southeast Asian languages, they are even more screwed. We need to make sure that AI multilingual to avoid total English domination of culture.
- benatkin 3y ago
  This will be easy, it will go from mostly English to hyperpolyglot quickly.
  - ben_w 3y ago
    The issue raised in the article is that there may not be enough training material in many languages to do this. I find this very plausible.
  - DoingIsLearning 3y ago
    If it's training on available online corpus then it will go quickly mostly for English and Mandarin. Most countries' classic texts and books are still undigitized sitting in Libraries and public archives. Also book publishing market and online publishing are proportional to total population, smaller country means less content.
- tellarin 3y ago
  Project Aya is one such attempt at a multi-lingual model (targeting 101 languages): - https://txt.cohere.com/aya-multilingual/ https://txt.cohere.com/aya-multilingual/ - https://aya.for.ai https://aya.for.ai I'm a contributor to the project and all data and model will be open-sourced. We're looking for contributors in many languages!
- loxdalen 3y ago
  I can speak in my own language with chat gpt without much issue
  - rjtavares 3y ago
    While I can speak in Portuguese without much issues (except being hard for them to stick to European Portuguese), I've nooticed that sometimes it uses a clear translation of an English expression that does not feel natural in Portuguese at all.
dstala 3y ago
Very much applicable to South Asian languages too, but the problem remains same. Not enough content being available to train. And unlike Finland, Government or private funding is a distant dream for many states in India.
qwerty456127 3y ago
I wonder if a GPT or something alike could possibly produce a good conlang easy to understand and learn for people of a specific linguistic group by optimally interpolating between their languages and filtering out idiosyncrasies.
- pmontra 3y ago
  Maybe, but it would take some real effort. This naive attempt of mine is an obvious non starter (it's ChatGPT 3.5). Transcript with some minimal editing to fit it into HN: User: Would you be able to develop an artificial language easy to learn and use for speakers of English and romance languages? Extremely simple grammar. ChatGPT: Sure, I'd focus on minimal grammar, consistent rules, and common vocabulary. It's doable. User: Good. Give a name to that language ChatGPT: How about "LinguaEase"? User: OK. Give me some examples starting with translations of the sentences in this chat so far. ChatGPT "LinguaEase Exempla: > 'Sure, I'd focus on minimal grammar, consistent rules, and common vocabulary.' becomes 'Yep, keep grammar simple, rules steady, vocab shared.' > 'How about "LinguaEase"?' becomes 'What about "LinguaEase"?' > 'OK. Give me some examples starting with translations of the sentences in this chat so far.' becomes 'Alright, show me samples, begin with translations of this chat's sentences.'" I went with English and romance languages because they are the languages I know. It looks all English to me. I would probably have had to be more specific. For example: only present tense. No plural. No gender. Only one set of pronouns. Aim to sentences like Beer two me drink yesterday. Beer ten me buy tomorrow. Then work on vocabulary.
  - d1sxeyes 3y ago
    I agree it would be very difficult. I think that current LLMs would fail here because I don't think they would be able to make a value judgement about what is 'harder' between 'I drank two beers yesterday' and 'Beer two me drink yesterday'. In fact, I'm not sure most linguists would agree. All languages are something of a compromise between accuracy and simplicity. An LLM is currently not equipped to make decisions about the correct balance between accuracy and simplicity. For example, English eschews cases (in most situations) in favour of fixed word order. 'Beer two me drink yesterday' violates that rule (and in fact, manages to use case incorrectly in one of the few places the vestiges of a case system still exist in English). However, a sentence constructed using the same structure as yours highlights a problem. 'Me two dog bite yesterday' Did you bite two dogs or were you bitten by two dogs? How can you tell? English has the subject of the sentence (the thing/person doing the verb) first, then the verb, then the object (the thing having the verb done to it). In languages with more fluid word order, normally a case marker is added to either the determiner or the noun itself: 'Me-o two dog bite yesterday'. Let's choose '-o' as our accusative case ending. Now we can tell that it was you who was bitten by the dog. Next problem: what if instead of 'me' it's 'man'. 'Man-o two dog bite yesterday'. How do we know whether it's two men, or two dogs? Again, English keeps the number with the noun it refers to. Most languages do, but some also put a marker on the number to make it clear. 'Man-o two-o dog bite yesterday'. Now we can tell that two men were bitten by some number of dogs. Next problem is that we don't know how many dogs bit the men. Were two men bitten by the same dog, or different dogs? Were two men surrounded by a pack of dogs? Is it important? Most languages care about this enough to at least mark the plural. What if we don't know exactly when the men were bitten? We could have some generic word like 'past', to make 'man-o two-o dog bite past'. But at that point, is your conlang really any easier than English, or just different?
kunley 3y ago
ChatGPT does not think.
- jiggawatts 3y ago
  Are you sure? How would you even know!? This is like a machine intelligence casually dismissing a brain as not being able to properly think, since meat can't compute.[1] Don't confuse the substrate or the training method with the result. [1] Obligatory reference: https://www.mit.edu/people/dpolicar/writing/prose/text/thinkingMeat.html https://www.mit.edu/people/dpolicar/writing/prose/text/think...
  - diogenes4 3y ago
    Hell i'm not even sure you or I think. This is the problem with referring to subjective perception when we can't be sure we're even referring to the same thing. A "thought" might just be a linguistic quirk of two meats exchanging information.
    - d1sxeyes 3y ago
      Cogito ergo sum... I think more or less the only thing we can be sure of is that we think. However, I don't think we REALLY know what thinking means, and whether the way we think is categorically different from how an LLM thinks.
      - diogenes4 3y ago
        I've always felt that "sum" was sufficient and self-evident. I see no reason to drag "cogito" into the conversation, let alone "ergo".
        d1sxeyes 3y ago
        Self evident perhaps, but sufficient? I think that depends on a wider context of what the question you’re trying to answer is.
  - deleted 3y ago
Kuinox 3y ago
ChatGPT is a single pass for each character. It can't "think" in English then translate it. It will often respond in English instead of the language you are using. Yes, it's better in English than any other languages, because the English dataset is bigger, and that's why an English AI will always be slightly better with the same technology.
- dustypotato 3y ago
  > ChatGPT is a single pass for each character. Not for each token?
  - Kuinox 3y ago
    Yeah, each token, but that's not my point.
- xico 3y ago
  That’s indeed its interface, but there is still a huge training phase during which the model was taught using mostly English, and it explains why some other languages interactions feel like they are some direct translation of English expressions (not to mention higher ways of thinking). Most of the thinking indeed is done during training, and then encoded in the weights which act like a huge cache of thoughts, but there is still a lot of learning on how to use the context when you emit this character (token), the attention layers, which is also a form of thinking that uses whatever new input you are giving to the model and applies a series of transformations (granted, finite in current models) to it.
  - Kuinox 3y ago
    But maybe the dataset for said language is just too small to produce correct content at all, and that's why ChatGPT is bad, not because "ChatGPT is thinking in English". Except when it default to answering in English, ChatGPT sounds like french when I talk to it in french. The bigger the dataset, the best is it's capabilities, training it exclusively with a language don't mean it will be better at this language.
- 3y ago
omneity 3y ago
Slightly off-topic, but figured this might be the best place to connect with like minded people: I’ve been working on NLP tooling for low resource languages, and had quite the success developing an LLM that speaks Moroccan natively [0], started from a super tiny dataset I manually annotated and scaling it to find anything written in Moroccan on the whole web. Currently helping out a research group to extend the same approaches from my project to Indic languages and writing scripts. If anyone here is working on NLP for low resource languages, hit me up! I’d love to exchange ideas, techniques and possibly to collaborate. [0]: https://sawalni.com https://sawalni.com
- vintermann 3y ago
  That's very cool! I may be slightly like-minded. It's been many years and technology has outrun it, but ~5 years ago I built a model to translate between Scandinavian languages and in particular the two written forms of Norwegian. I had the same problem of lacking training data, but I tried an approach of bootstrapping from rule-based translators. There exist decent rule-based translators for the Scandinavian languages. Of course they make mistakes, but I figured that as long as I used the rule-translated text only as source, never as target, it wouldn't matter. Effectively, instead of learning to translate from real Danish to correct Norwegian, it learned to translate from slightly broken machine-Danish to correct Norwegian. And my hope was that it would learn to translate from real Danish to correct Norwegian as a side effect. I then figured that I might as well let it take input in slightly broken machine-Swedish as well, hopefully helping its generalization. Then I could use this translator for the next step, again using the real text as the target, but using output from this translator as input. It worked reasonably well. It certainly beat Google Translate at the time, since that at the time went via English even for closely related languages like the Scandinavian ones. I seriously considered making a start-up and commercializing it, some people had expressed interest (among them newspapers - they used a rule-based translator/grammar checker at the time, I think maybe they still do.) But in the end, it's probably good I didn't. These days I understand they don't even train sequence to sequence models, they just train a general-purpose language model and instruction-tune it to do translation.
  - 3y ago
helsinkiandrew 3y ago
Interestingly it's using the LUMI supercomputer to do this: HPE Cray EX supercomputer, 2560 nodes comprised of a 64 core AMD Trento CPU and four AMD MI250X GPUs (128 GB). With a combined Linpack performance of 375 Pflop/s https://www.lumi-supercomputer.eu/lumis-full-system-architecture-revealed/ https://www.lumi-supercomputer.eu/lumis-full-system-architec... https://en.wikipedia.org/wiki/LUMI https://en.wikipedia.org/wiki/LUMI
- haffi112 3y ago
  Isn't that going to be a problem with no NVIDIA GPUs?
  - thiago_fm 3y ago
    You can run LLMs in your own machine. Do you think a super computer would have issues? CUDA has optimizations, but you don't necessarily need it to do inference at all. Those super computers are extremely powerful, it might not be as energy efficient as H100s, but it does the job.
  - 0xDEF 3y ago
    There has been a lot of recent progress in making PyTorch AMD compatible exactly because many government/university supercomputers are based on AMD GPUs.
  - mikk14 3y ago
    Funnily enough I literally just today launched a big job on LUMI that I have started also on another smaller cluster with nVidia GPUs. Basically, I'm running Llama2-70B to do some zero-shot text classification. The nVidia setup uses 4 A100s, while on LUMI I could access 6 MI250Xs. It is, unfortunately, not an apples-to-apples comparison, because on the nVidia cluster I'm running it via llama-cpp-python and a quantized 34B version, while on LUMI I'm running the official non-quantized full 70B version via the transformers library. Long story short, I'm getting a 7.5x higher throughput from LUMI than on the nVidia cluster (which means each card is 5x faster on LUMI). Edit: The AMD GPUs work fine because one can run Pytorch for ROCm via the pytorch-triton-rocm package.
anileated 3y ago
Describing a glorified autocomplete as “thinking” is wrong on many levels; it devalues human consciousness, it attributes to a software tool qualities that it does not possess, and has implications for the entire AI industry. If we are to believe an LLM is at least sentient and intelligent on par with an octopus, much less a human, then we have a moral imperative to fight for LLMs to be recognized as protected sentient thinking beings with awareness and agency and admit that AI industry is built on systematic abuse of those beings. Of course, no one does it, because it would be ridiculous. It’s time to admit that people who insist that LLMs “think” are simply trying to make ML seem mysterious and magical in an ill thought through (because see above) attempt to attract more VC money.
- looping8 3y ago
  I don't think that admitting an AI is in some way sentient would create a push to protect it. Animal testing is still very popular, despite there being no arguments about them being sentient. The tech industry isn't a moral leader and, based on some things major companies do, isn't even moral.
  - aziaziazi 3y ago
    Moreover, OP “octopus” exemple is a very bad pick : those animals are one of the more sentient we currently know, probably more than humans.
- nercury 3y ago
  Have we forgot that we have attributed "thinking" to all the machines since the beginning?
- GolDDranks 3y ago
  I bet you think that aeroplanes don't fly either. /s
  - TeMPOraL 3y ago
    Of course they don't. Have you seen an aeroplane flapping its wings? Those that do usually are in the process of falling out of the sky. No, what an aeroplane is is a glorified lawn dart - making it bigger and strapping oversized hair dryers powered by gasoline doesn't make it fly. /s, but only so much - that's how the "stochastic parrot"/"glorified autocomplete" responses read to me.
paulhalla 3y ago
I use ChatGPT audio to get answers in a few languages, including Estonian. I find it really amusing that Estonian ChatGPT audio has a thick American accent. Generally the content is solid and the speech is fluid, not halting or robotic or anything, so it legit sounds like an American person that is super fluent in Estonian.
nodoodles 3y ago
The real question is, will it pick up and emulate national archetypes in communication - I expect this model to be introverted and avoid talking to strangers
leshenka 3y ago
Explain me this. As I understand, chatgpt is trained by dumping a huge amounts of text in it. It doesn't have a notion of "language", all it does is continues the series of tokens. Therefore all you need to do in order to train it a new language is to build a corpus of text and feed it to the model. Is this right? And if so, is this what they did here or is it something completely different?
- Gravityloss 3y ago
  Yes, Finnish works fine.
- tuukkah 3y ago
  To begin with, the training material is tokenized and one thing the choice of the optimal tokenizer depends on is the source language(s). Consider that in English, the optimal tokenization of "cats" may be "cat" + "s" (based on the vocabulary and morphology of English) but in another language that may make no sense. The choice of the tokenizer is probably more important in a language which has less training material available.
- otabdeveloper4 3y ago
  Most likely it's just a shitty job of curating the training set. Estonian is related to Finnish and they probably got some of the texts mixed up, because who cares about all those silly languages, amirite?
nologic01 3y ago
Seems reasonable that large, general purpose LLM's are developed as digital public goods and effort is extended to support as many human languages as possible. Alas, as you can easily confirm by checking the number of entries in Wikipedia, most populations have more pressing concerns than digitizing their cultural universes. Which is sad predicament: the equivalent of non-reversible biodiversity loss, but applied to human culture. In theory I suppose one could plug-in gaps in available corpora with translation, but ideally you want to preserve the richness of distinct cultures.
deleted 3y ago
[deleted]
helsinkiandrew 3y ago
I thought ChatGPT and LLMs in general use vectored word embeddings - which captures that dog and puppy are related (so you can take puppy, subtract dog and add cat to get kitten). Wouldn't this also capture that the Estonian token for dog is related to English token for dog etc? So ChatGPT is 'thinking' in some kind of semantic tokens not English.
- ben_w 3y ago
  That can work for some words, but not all words in all languages divide all of reality up the same way. Colour is a particularly easy way to notice this (ancient Greeks and their wine-dark seas), but also consider: in English "dog" isn't just an animal, but also a way to contemptuously describe a human. Can the word be used that way in Estonian? Or numbers: is 13 unlucky like in English, and is 4 unlucky like in some east asian languages? Or settlements: in the UK a settlement is a "city" if the monarch says so, apparently the Estonian word "linn" is used for both "a city" and "a town".
  - helsinkiandrew 3y ago
    I thought some of that would be captured by the embeddings, crudely: ["english" + "dog" + "person"] = ["estonian" + "bad" + "person"]
- tgv 3y ago
  I had hoped that would be the case, but I tried multi-lingual clustering, and the clusters were neatly assigned per language, even though the meanings were incredibly close (e.g. short phrases literally meaning "cheap" or "unhealthy"). BTW, that was using ada-002.
- a2128 3y ago
  It doesn't actually have a token for Estonian dog, the word " koer" is tokenized as two tokens, " ko" and "er". In my language it's three tokens. In Russian it's seven(!). So any relations to dog and puppy in other languages would have to happen deeper within the model. Just as the tokenizer can't capture that a base64-encoded "dog" is related to a base64-encoded "puppy"

116 points by jruohonen 3y ago

“think”. Think of the average person and then think that half of the world is less smart than that. Those people are the target audience for news articles and products that use terms such as “think” and fear mongering when it comes to a”i”.

IshKebab 3y ago
This is pointless and frankly baseless pedantry. There's no reason to artificially restrict the word "think" to squashy brains.
- jraph 3y ago
  I don't like your parent comment but there's merit in avoiding people believing those things actually think and are actually intelligent. In "thinking", there's reasoning, which those things currently don't. They output text statistically and that's what mass media should tell their audience.
  - ben_w 3y ago
    What is "reasoning"? Wikipedia surprised me with this opening paragraph: > Reason is the capacity of applying logic consciously by drawing conclusions from new or existing information, with the aim of seeking the truth, with more than 10 or more than 3 evidence, 99.9% reliability or 87.5% reliability. Apart from the words "consciously" (which has 100 or so different definitions, some of which do and some of which do not apply to AI), and "aim" which can be contentious depending upon if you feel that requires consciousness or can include training rewards… I think the best of them is at the lower of those two mysterious percentages I've never seen discussed before. And that's despite LLMs being wildly bad at, specifically, logic. Though even this specific weakness in general seems like a very easy thing to get around by hybridising it with the computer it has to run on e.g. by telling it how to use a compiler, as logic is the foundation of the arithmetic that the transformers use to do natural language comprehension in the first place.
    - vikramkr 3y ago
      That opening for the Wikipedia article looks problematic - it's grammatically nonsense and has random numbers in it - someone should probably open a talk discussion on what happened Edit: yeah that looks like a recent edit. I thought Wikipedia was supposed to be pretty good about flagging this sort of stuff? Does anyone know how to raise the problem to those super editors that run the site? I assume they won't appreciate a random person undoing an edit https://en.wikipedia.org/w/index.php?title=Reason&diff=prev&oldid=1179925542 https://en.wikipedia.org/w/index.php?title=Reason&diff=prev&...
      - ben_w 3y ago
        If I was on desktop right now, I'd put the new bit after the existing citation and add a new {{citation needed}} for the addition. Don't worry too much about diving in, so long as you comment your changes appropriately when committing, that's the point of the place and how it does the quality control in the first place :)
        vikramkr 3y ago
        Leaving a citation needed doesn't sound like the right thing to do for the definition of something as big as reason - esp with the grammatical confusion idk what you'd even edit it to to make sense. Like what is a three evidence or a reliability and what do those reliability numbers mean. And the sentence describing those numbers doesn't make sense - is it 10 or 3 or which of those percentages? Idk maybe it should just be undone
        ben_w 3y ago
        Then be bold with the edits and delete it, just say why in the comment box. The deletion itself is as easy to undo as to perform, and if you say why you're doing it, someone bringing that part back can do a better job next time. :)
    - Sharlin 3y ago
      > And that's despite LLMs being wildly bad at, specifically, logic. I'd say "wildly bad" is wildly exaggerated given that GPT-4 exists, even if you define "logic" as "logical puzzles encoded in natural language" and not "the thing that allows an LLM to reason highly efficiently about what token comes next".
  - Sharlin 3y ago
    What is, then, the qualitative difference between a GOFAI automated reasoning system [1], like Coq, and an LLM? Why does Coq do reasoning but GPT-4 doesn't? Does it only count as reasoning if hand-coded by a human? Or did you have another definition of "reasoning" in mind? Statistical or probabilistic reasoning is still reasoning, and indeed essentially mandatory to make sense of the real world, and is what humans do. Indeed the failure of painstakingly hand-trained symbolic reasoning engines to model reality (as opposed to just simple toy worlds) was the entire reason for the rise of neural networks. The entire point of a neural network is to efficiently compress large amounts of statistical data into a model with as much predictive power as possible. Somewhere within GPT-4's many layers there has to exist something like Bayes networks and other ultra-efficient representations of the vast network of correlations present in the training corpus – otherwise there's no way it could do what it does. [1] https://en.wikipedia.org/wiki/Automated_reasoning https://en.wikipedia.org/wiki/Automated_reasoning
    - jraph 3y ago
      I didn't know "automated reasoning" was an actual technical term. I didn't mean this by "reasoning". To me, Coq does not reason (in the meaning I intended). It checks proofs are well-formed and assists proof writing. But yes, there's some implementation of logic in Coq. It applies (a specific form of) logic. Again, LLMs output probable text. They don't do logic. They do statistics on language. Coq works on the actual objects themselves directly, not on the text that presents them. LLMs usually fail to answer to riddles correctly for this reason. We saw plenty of occurrences of this in recent times. Each time LLMs are benchmarked, people think of trying riddles and it works to some extent because something that looks like these riddles are in the training set but it eventually fails. Humans also make mistakes when solving riddles, but not for the same reasons (the process is not the same, they are not "processing text", the are really processing logic, even if the process is fallible).
      - Sharlin 3y ago
        So you just replaced the nebulously defined word "think" with an equally nebulously defined word, "reason"? Not a very convincing argument. You do not know what humans do, no human does, as we don't have such an access to our internal representations. You neither know what LLMs do, no human does, as even though we have full access to their internals, we have little to no idea how to interpret said internals. The most interesting thing about LLMs is the degree that they have learned to model the real world while only being exposed to words. Because those words are highly correlated not just with each other, but with an external entity unknown to the LLM, it's very likely that by learning to model that external entity, the LLM's word-predicting power will increase. LLMs are likely to have converged on a world-model simply because it makes the most sense given the available evidence. This is no different from how human scientists create models of things they cannot directly see based on what they can see. And of course, "seeing" is not directly perceiving the external reality either! Sensory data merely correlates with what's really going on, and the brain has learned to model the world – an unknown external entity – based on the correlations that exist in the sensory language.
- Y_Y 3y ago
  Is it really baseless? I'd say it depends on your theory of mind and is a serious question which generates interesting discussion. Myself I don't consider "think" an appropriate term, since the super-Markov-chain types of LLM right now seem to miss out on important aspects of cognition, particularly synthesis of two disparate ideas. It's hard to test this since you need to ask it about things which aren't already discussed in its corpus, but even GPT-4 tends to disappoint me when I ask it weird questions that have straightforward answers but probably aren't written about much. As an example, I asked how I could go about obtaining a large quantity of animal skulls cheaply. It directed me to taxidermists and specialists dealing in bones of exotic animals, whereas it could be much easier to go to an abbatoir, or even buy whole animals (since the exotic ones are so expensive).
- s_dev 3y ago
  Google doesn't 'think' and neither does ChatGPT. I will give you a hard limit on what a machine can't compute -- it will never figure out given a programming script and an input when that program will stop. A human can do it by thinking through the problem -- whatever that process/phenomena of thought actually is it can think outside of itself and that fundamentally seperates it from a program even one as sophisticated as ChatGPT.
  - astrange 3y ago
    The limits on the halting problem are the same for a human and a computer. People aren't hypercomputers.
  - ben_w 3y ago
    > A human can do it by thinking through the problem No, we cannot. Imagine a program that looks for the smallest counterexample to the Collatz conjecture. Nobody knows if it will ever terminate.
  - xcv123 3y ago
    > A human can do it by thinking through the problem No. Humans cannot solve the halting problem. It is unsolvable. Cannot be solved manually or automatically. > It will never figure out given a programming script and an input when that program will stop It can for some programs. Just like humans can for some programs but not all.
  - jiggawatts 3y ago
    Have you verified this hypothesis of yours? In my experience GPT4 especially can definitely "think through" what a program does and arrive at the result. Even the significantly dumber ChatGPT can do this: https://chat.openai.com/share/3ee29c5e-6cb4-485f-b0b4-29e982463687 https://chat.openai.com/share/3ee29c5e-6cb4-485f-b0b4-29e982... People mistakenly think that it's just regurgitating what it has seen on the web, but even if you don't prompt it in English it'll still a) solve the problem correctly, and b) do it in the target language: https://chat.openai.com/share/5b4e6e99-3fd8-4316-811c-2a9dec776f63 https://chat.openai.com/share/5b4e6e99-3fd8-4316-811c-2a9dec... If that's not impressive enough, the ability to round-trip from script-to-poem and then poem-to-script should blow your mind: What might have been the original PowerShell script snippet that inspired this poem? Be as terse as possible, using aliases where possible. In the realm of numbers, a tale begins, Where $x is a 7, with a smile it grins. From one to ten, the journey we thread, Adding 7 to $x, step by step we tread. $x grows and grows, like a tree so tall, With every step, it doesn't fall. Once the journey ends, not a moment we relax, Triple the fun, we do with $x. In this dance of digits and the arithmetic song, With $x as our guide, we whirl along. From start to finish, through each complex, This is the tale of our friend, $x. Here's a terse PowerShell script snippet that inspired the poem: ```powershell $x = 7 1..10 | % {$x += 7} $x *= 3 ```
- ggm 3y ago
  No. You're wrong. It's important and we need to stomp on random use of language like this, which has consequences. It's not thinking and you are dangerously wrong if you don't understand why. The danger isn't AGI it's misplaced belief in the state of AI. Call this thinking and you skew perceptions.
  - HPsquared 3y ago
    All they need to do is put the word in quotes. ChatGPT Equivalent begins to "think" in Estonian. Any better suggestions? Thinking itself isn't exactly a rigorously-defined word. Do animals think?
- Lio 3y ago
  Well rather than argue it out with everyone I just went straight to source and asked ChatGPT what it was thinking about: > I don't have thoughts or consciousness like a human. I'm here to provide information and assist with any questions or tasks you have. Is there something specific you'd like to know or discuss? So in its own words it doesn't "think"... ...which just conclusively proves that its worked out how to lie and is about to kill us all! :P
  - IshKebab 3y ago
    That's just what it's been told to say. I can also say it. Doesn't make it true. Do animals think? Have any told you so?

v4dok 3y ago

This is important work. And think about African or southeast Asian languages, they are even more screwed. We need to make sure that AI multilingual to avoid total English domination of culture.

benatkin 3y ago
This will be easy, it will go from mostly English to hyperpolyglot quickly.
- ben_w 3y ago
  The issue raised in the article is that there may not be enough training material in many languages to do this. I find this very plausible.
- DoingIsLearning 3y ago
  If it's training on available online corpus then it will go quickly mostly for English and Mandarin. Most countries' classic texts and books are still undigitized sitting in Libraries and public archives. Also book publishing market and online publishing are proportional to total population, smaller country means less content.
tellarin 3y ago
Project Aya is one such attempt at a multi-lingual model (targeting 101 languages): - https://txt.cohere.com/aya-multilingual/ https://txt.cohere.com/aya-multilingual/ - https://aya.for.ai https://aya.for.ai I'm a contributor to the project and all data and model will be open-sourced. We're looking for contributors in many languages!
loxdalen 3y ago
I can speak in my own language with chat gpt without much issue
- rjtavares 3y ago
  While I can speak in Portuguese without much issues (except being hard for them to stick to European Portuguese), I've nooticed that sometimes it uses a clear translation of an English expression that does not feel natural in Portuguese at all.

dstala 3y ago

Very much applicable to South Asian languages too, but the problem remains same. Not enough content being available to train. And unlike Finland, Government or private funding is a distant dream for many states in India.

qwerty456127 3y ago

I wonder if a GPT or something alike could possibly produce a good conlang easy to understand and learn for people of a specific linguistic group by optimally interpolating between their languages and filtering out idiosyncrasies.

pmontra 3y ago
Maybe, but it would take some real effort. This naive attempt of mine is an obvious non starter (it's ChatGPT 3.5). Transcript with some minimal editing to fit it into HN: User: Would you be able to develop an artificial language easy to learn and use for speakers of English and romance languages? Extremely simple grammar. ChatGPT: Sure, I'd focus on minimal grammar, consistent rules, and common vocabulary. It's doable. User: Good. Give a name to that language ChatGPT: How about "LinguaEase"? User: OK. Give me some examples starting with translations of the sentences in this chat so far. ChatGPT "LinguaEase Exempla: > 'Sure, I'd focus on minimal grammar, consistent rules, and common vocabulary.' becomes 'Yep, keep grammar simple, rules steady, vocab shared.' > 'How about "LinguaEase"?' becomes 'What about "LinguaEase"?' > 'OK. Give me some examples starting with translations of the sentences in this chat so far.' becomes 'Alright, show me samples, begin with translations of this chat's sentences.'" I went with English and romance languages because they are the languages I know. It looks all English to me. I would probably have had to be more specific. For example: only present tense. No plural. No gender. Only one set of pronouns. Aim to sentences like Beer two me drink yesterday. Beer ten me buy tomorrow. Then work on vocabulary.
- d1sxeyes 3y ago
  I agree it would be very difficult. I think that current LLMs would fail here because I don't think they would be able to make a value judgement about what is 'harder' between 'I drank two beers yesterday' and 'Beer two me drink yesterday'. In fact, I'm not sure most linguists would agree. All languages are something of a compromise between accuracy and simplicity. An LLM is currently not equipped to make decisions about the correct balance between accuracy and simplicity. For example, English eschews cases (in most situations) in favour of fixed word order. 'Beer two me drink yesterday' violates that rule (and in fact, manages to use case incorrectly in one of the few places the vestiges of a case system still exist in English). However, a sentence constructed using the same structure as yours highlights a problem. 'Me two dog bite yesterday' Did you bite two dogs or were you bitten by two dogs? How can you tell? English has the subject of the sentence (the thing/person doing the verb) first, then the verb, then the object (the thing having the verb done to it). In languages with more fluid word order, normally a case marker is added to either the determiner or the noun itself: 'Me-o two dog bite yesterday'. Let's choose '-o' as our accusative case ending. Now we can tell that it was you who was bitten by the dog. Next problem: what if instead of 'me' it's 'man'. 'Man-o two dog bite yesterday'. How do we know whether it's two men, or two dogs? Again, English keeps the number with the noun it refers to. Most languages do, but some also put a marker on the number to make it clear. 'Man-o two-o dog bite yesterday'. Now we can tell that two men were bitten by some number of dogs. Next problem is that we don't know how many dogs bit the men. Were two men bitten by the same dog, or different dogs? Were two men surrounded by a pack of dogs? Is it important? Most languages care about this enough to at least mark the plural. What if we don't know exactly when the men were bitten? We could have some generic word like 'past', to make 'man-o two-o dog bite past'. But at that point, is your conlang really any easier than English, or just different?

kunley 3y ago

ChatGPT does not think.

jiggawatts 3y ago
Are you sure? How would you even know!? This is like a machine intelligence casually dismissing a brain as not being able to properly think, since meat can't compute.[1] Don't confuse the substrate or the training method with the result. [1] Obligatory reference: https://www.mit.edu/people/dpolicar/writing/prose/text/thinkingMeat.html https://www.mit.edu/people/dpolicar/writing/prose/text/think...
- diogenes4 3y ago
  Hell i'm not even sure you or I think. This is the problem with referring to subjective perception when we can't be sure we're even referring to the same thing. A "thought" might just be a linguistic quirk of two meats exchanging information.
  - d1sxeyes 3y ago
    Cogito ergo sum... I think more or less the only thing we can be sure of is that we think. However, I don't think we REALLY know what thinking means, and whether the way we think is categorically different from how an LLM thinks.
    - diogenes4 3y ago
      I've always felt that "sum" was sufficient and self-evident. I see no reason to drag "cogito" into the conversation, let alone "ergo".
      - d1sxeyes 3y ago
        Self evident perhaps, but sufficient? I think that depends on a wider context of what the question you’re trying to answer is.
- deleted 3y ago

Kuinox 3y ago

ChatGPT is a single pass for each character. It can't "think" in English then translate it. It will often respond in English instead of the language you are using. Yes, it's better in English than any other languages, because the English dataset is bigger, and that's why an English AI will always be slightly better with the same technology.

dustypotato 3y ago
> ChatGPT is a single pass for each character. Not for each token?
- Kuinox 3y ago
  Yeah, each token, but that's not my point.
xico 3y ago
That’s indeed its interface, but there is still a huge training phase during which the model was taught using mostly English, and it explains why some other languages interactions feel like they are some direct translation of English expressions (not to mention higher ways of thinking). Most of the thinking indeed is done during training, and then encoded in the weights which act like a huge cache of thoughts, but there is still a lot of learning on how to use the context when you emit this character (token), the attention layers, which is also a form of thinking that uses whatever new input you are giving to the model and applies a series of transformations (granted, finite in current models) to it.
- Kuinox 3y ago
  But maybe the dataset for said language is just too small to produce correct content at all, and that's why ChatGPT is bad, not because "ChatGPT is thinking in English". Except when it default to answering in English, ChatGPT sounds like french when I talk to it in french. The bigger the dataset, the best is it's capabilities, training it exclusively with a language don't mean it will be better at this language.
3y ago

omneity 3y ago

Slightly off-topic, but figured this might be the best place to connect with like minded people: I’ve been working on NLP tooling for low resource languages, and had quite the success developing an LLM that speaks Moroccan natively [0], started from a super tiny dataset I manually annotated and scaling it to find anything written in Moroccan on the whole web. Currently helping out a research group to extend the same approaches from my project to Indic languages and writing scripts. If anyone here is working on NLP for low resource languages, hit me up! I’d love to exchange ideas, techniques and possibly to collaborate. [0]: https://sawalni.com https://sawalni.com

vintermann 3y ago
That's very cool! I may be slightly like-minded. It's been many years and technology has outrun it, but ~5 years ago I built a model to translate between Scandinavian languages and in particular the two written forms of Norwegian. I had the same problem of lacking training data, but I tried an approach of bootstrapping from rule-based translators. There exist decent rule-based translators for the Scandinavian languages. Of course they make mistakes, but I figured that as long as I used the rule-translated text only as source, never as target, it wouldn't matter. Effectively, instead of learning to translate from real Danish to correct Norwegian, it learned to translate from slightly broken machine-Danish to correct Norwegian. And my hope was that it would learn to translate from real Danish to correct Norwegian as a side effect. I then figured that I might as well let it take input in slightly broken machine-Swedish as well, hopefully helping its generalization. Then I could use this translator for the next step, again using the real text as the target, but using output from this translator as input. It worked reasonably well. It certainly beat Google Translate at the time, since that at the time went via English even for closely related languages like the Scandinavian ones. I seriously considered making a start-up and commercializing it, some people had expressed interest (among them newspapers - they used a rule-based translator/grammar checker at the time, I think maybe they still do.) But in the end, it's probably good I didn't. These days I understand they don't even train sequence to sequence models, they just train a general-purpose language model and instruction-tune it to do translation.
- 3y ago

helsinkiandrew 3y ago

Interestingly it's using the LUMI supercomputer to do this: HPE Cray EX supercomputer, 2560 nodes comprised of a 64 core AMD Trento CPU and four AMD MI250X GPUs (128 GB). With a combined Linpack performance of 375 Pflop/s https://www.lumi-supercomputer.eu/lumis-full-system-architecture-revealed/ https://www.lumi-supercomputer.eu/lumis-full-system-architec... https://en.wikipedia.org/wiki/LUMI https://en.wikipedia.org/wiki/LUMI

haffi112 3y ago
Isn't that going to be a problem with no NVIDIA GPUs?
- thiago_fm 3y ago
  You can run LLMs in your own machine. Do you think a super computer would have issues? CUDA has optimizations, but you don't necessarily need it to do inference at all. Those super computers are extremely powerful, it might not be as energy efficient as H100s, but it does the job.
- 0xDEF 3y ago
  There has been a lot of recent progress in making PyTorch AMD compatible exactly because many government/university supercomputers are based on AMD GPUs.
- mikk14 3y ago
  Funnily enough I literally just today launched a big job on LUMI that I have started also on another smaller cluster with nVidia GPUs. Basically, I'm running Llama2-70B to do some zero-shot text classification. The nVidia setup uses 4 A100s, while on LUMI I could access 6 MI250Xs. It is, unfortunately, not an apples-to-apples comparison, because on the nVidia cluster I'm running it via llama-cpp-python and a quantized 34B version, while on LUMI I'm running the official non-quantized full 70B version via the transformers library. Long story short, I'm getting a 7.5x higher throughput from LUMI than on the nVidia cluster (which means each card is 5x faster on LUMI). Edit: The AMD GPUs work fine because one can run Pytorch for ROCm via the pytorch-triton-rocm package.

anileated 3y ago

Describing a glorified autocomplete as “thinking” is wrong on many levels; it devalues human consciousness, it attributes to a software tool qualities that it does not possess, and has implications for the entire AI industry. If we are to believe an LLM is at least sentient and intelligent on par with an octopus, much less a human, then we have a moral imperative to fight for LLMs to be recognized as protected sentient thinking beings with awareness and agency and admit that AI industry is built on systematic abuse of those beings. Of course, no one does it, because it would be ridiculous. It’s time to admit that people who insist that LLMs “think” are simply trying to make ML seem mysterious and magical in an ill thought through (because see above) attempt to attract more VC money.

looping8 3y ago
I don't think that admitting an AI is in some way sentient would create a push to protect it. Animal testing is still very popular, despite there being no arguments about them being sentient. The tech industry isn't a moral leader and, based on some things major companies do, isn't even moral.
- aziaziazi 3y ago
  Moreover, OP “octopus” exemple is a very bad pick : those animals are one of the more sentient we currently know, probably more than humans.
nercury 3y ago
Have we forgot that we have attributed "thinking" to all the machines since the beginning?
GolDDranks 3y ago
I bet you think that aeroplanes don't fly either. /s
- TeMPOraL 3y ago
  Of course they don't. Have you seen an aeroplane flapping its wings? Those that do usually are in the process of falling out of the sky. No, what an aeroplane is is a glorified lawn dart - making it bigger and strapping oversized hair dryers powered by gasoline doesn't make it fly. /s, but only so much - that's how the "stochastic parrot"/"glorified autocomplete" responses read to me.

paulhalla 3y ago

I use ChatGPT audio to get answers in a few languages, including Estonian. I find it really amusing that Estonian ChatGPT audio has a thick American accent. Generally the content is solid and the speech is fluid, not halting or robotic or anything, so it legit sounds like an American person that is super fluent in Estonian.

nodoodles 3y ago

The real question is, will it pick up and emulate national archetypes in communication - I expect this model to be introverted and avoid talking to strangers

leshenka 3y ago

Explain me this. As I understand, chatgpt is trained by dumping a huge amounts of text in it. It doesn't have a notion of "language", all it does is continues the series of tokens. Therefore all you need to do in order to train it a new language is to build a corpus of text and feed it to the model. Is this right? And if so, is this what they did here or is it something completely different?

Gravityloss 3y ago
Yes, Finnish works fine.
tuukkah 3y ago
To begin with, the training material is tokenized and one thing the choice of the optimal tokenizer depends on is the source language(s). Consider that in English, the optimal tokenization of "cats" may be "cat" + "s" (based on the vocabulary and morphology of English) but in another language that may make no sense. The choice of the tokenizer is probably more important in a language which has less training material available.
otabdeveloper4 3y ago
Most likely it's just a shitty job of curating the training set. Estonian is related to Finnish and they probably got some of the texts mixed up, because who cares about all those silly languages, amirite?

nologic01 3y ago

Seems reasonable that large, general purpose LLM's are developed as digital public goods and effort is extended to support as many human languages as possible. Alas, as you can easily confirm by checking the number of entries in Wikipedia, most populations have more pressing concerns than digitizing their cultural universes. Which is sad predicament: the equivalent of non-reversible biodiversity loss, but applied to human culture. In theory I suppose one could plug-in gaps in available corpora with translation, but ideally you want to preserve the richness of distinct cultures.

deleted 3y ago

[deleted]

helsinkiandrew 3y ago

I thought ChatGPT and LLMs in general use vectored word embeddings - which captures that dog and puppy are related (so you can take puppy, subtract dog and add cat to get kitten). Wouldn't this also capture that the Estonian token for dog is related to English token for dog etc? So ChatGPT is 'thinking' in some kind of semantic tokens not English.

ben_w 3y ago
That can work for some words, but not all words in all languages divide all of reality up the same way. Colour is a particularly easy way to notice this (ancient Greeks and their wine-dark seas), but also consider: in English "dog" isn't just an animal, but also a way to contemptuously describe a human. Can the word be used that way in Estonian? Or numbers: is 13 unlucky like in English, and is 4 unlucky like in some east asian languages? Or settlements: in the UK a settlement is a "city" if the monarch says so, apparently the Estonian word "linn" is used for both "a city" and "a town".
- helsinkiandrew 3y ago
  I thought some of that would be captured by the embeddings, crudely: ["english" + "dog" + "person"] = ["estonian" + "bad" + "person"]
tgv 3y ago
I had hoped that would be the case, but I tried multi-lingual clustering, and the clusters were neatly assigned per language, even though the meanings were incredibly close (e.g. short phrases literally meaning "cheap" or "unhealthy"). BTW, that was using ada-002.
a2128 3y ago
It doesn't actually have a token for Estonian dog, the word " koer" is tokenized as two tokens, " ko" and "er". In my language it's three tokens. In Russian it's seven(!). So any relations to dog and puppy in other languages would have to happen deeper within the model. Just as the tokenizer can't capture that a base64-encoded "dog" is related to a base64-encoded "puppy"