3 ms·

I'm just an observer to the space, but I imagine each chinese character (or each word) being a token would be pretty effective, especially as inputting/outputti

I'm just an observer to the space, but I imagine each chinese character (or each word) being a token would be pretty effective, especially as inputting/outputting more tokens requires more vram. Additionally Chinese doesn't have conjugations which I imagine makes tokenization easier. I think it's easier to connect semantic meaning between these tokens/charcaters, because you can relate a character to other characters by their presence in words (most words are 2 characters long (compound words), although there are sometimes words like AB and BC which mean the same thing, but in conversations people will just say A or C and the meaning (AB or BC) is inferred by the context of the conversation, so in a semantic meaning database you would want a connection between A and AB, and to BC & C). You could also have a dimension for each radical that isn't a phonetic component. Radicals do link vaguely related words together, and I think this is a powerful language feature missing in English that could be exploited in the NLP field. I'm not a fluent or proficient speaker, so maybe I'm not the best person to explain things, but before OpenAI's Whisper, speech-to-text in WeChat in Mandarin was way better than any English speech-to-text I've ever used. The structure of the Chinese language and its large userbase would make me bet long on Chinese NLP progress over the west.

vasco 3y ago
Imagine losing WW3 because your language is less efficient to store and query than the enemy, so your war-llms need more resources and are less effective.
- stavros 3y ago
  My understanding is that all spoken languages are roughly equally efficient at data transmission per unit of time. I'm not sure how long it takes someone to read a text in Chinese over a text in English, though.
  - JCharante 3y ago
    Per unit of time is true, because languages with a lesser information density will be spoken at a faster rate in order to compensate (see: Spanish speakers speaking way faster than English speakers). But LLMs don't ingest data per unit of time, they ingest data via text.
    - stavros 3y ago
      Sure, but, depending on your representation, you can only have so many different tokens, and if you use, say, pinyin, then you don't have much of an advantage over English. It'll be interesting to see how LLMs do when trained on concepts (ie the Chinese alphabet), rather than sounds, though.
      - JCharante 3y ago
        > Sure, but, depending on your representation, you can only have so many different tokens From my understanding (which only comes from reading HN), unique token count isn't an issue that LLMs run into. If it is, then that would be a bummer for the possibilities of exploiting all the features of the Chinese language. > and if you use, say, pinyin Well yeah pinyin would be inefficient because you're stripping away all the "built in" semantics of a Chinese character to only focus on the pronounciation.
        stavros 3y ago
        I imagine you can always expand your token count, but then is that very different from syllabic alphabets (that encode multiple syllables in one character)? Granted, that doesn't apply to Chinese, which encoders concepts, so that's interesting to see in LLMs.
        JCharante 3y ago
        I think so, because even if you're encoding multiple syllables, that's just the pronounciation. I've learned languages with alphabets and glyph scripts like Chinese, and *what I don't like about alphabet languages like English is that the letters themselves provide little context to their meaning, although you could learn the latin root words and guess from there. Of course you know how to say it, but you don't know what it means. With Chinese, you might guess what it means, but you don't know how to say it.* The characters have more meaning in them, although this isn't the case for every word, which is something non-speakers assume. In short, from my experience learning Chinese was way easier than learning Vietnamese, which have historical ties together since Vietnamese used a fork of Chinese script until the early 1900s when they transitioned to latin script in order to improve literacy rates. Sure, it improved literacy rates because to read you only need to know how to pronounce, but it doesn't mean glyph languages are harder to learn the meaning/semantics.
czl_my 3y ago
Interestingly BaiChuan2 didn't just tokenize individual characters https://huggingface.co/baichuan-inc/Baichuan2-7B-Chat https://huggingface.co/baichuan-inc/Baichuan2-7B-Chat
visarga 3y ago
Text tokens by their construction are more like the Chinese ideographic characters, so no, there would not be a big difference between tokenized English and Chinese. The number of unique tokens is decided upfront - something like 50K - so, much more than basic vocabulary. English tokens can even be 2 word phrases. Look up Byte-Pair Encoding (BPE) for a fascinating and simple algorithm that explains the process of choosing the tokens. It shows why it doesn't matter how token dense the source language is.
kolinko 3y ago
Vram is dependent mostly on the parameter count, not on the number of input/output tokens, if I remember correctly. Also, in case of english language, it's one token per one word (more or less), so it's the same as in Chinese - assuming both LLM tokenisers were geared towards their native language. The only issue is when you have an tokeniser geared towards western languages, and you try to to use the same tokeniser on a different group of languages - then a single word in a language foreign to the tokeniser would have multiple tokens. But that has nothing to do with the underlying structure of the language. In other words - you wouldn't really see a difference between an input in chinese compared to english after the text gets tokenised. It's rougly the same amount of tokens, and the underlying parameter count would be also similar.

vasco 3y ago

Imagine losing WW3 because your language is less efficient to store and query than the enemy, so your war-llms need more resources and are less effective.

stavros 3y ago
My understanding is that all spoken languages are roughly equally efficient at data transmission per unit of time. I'm not sure how long it takes someone to read a text in Chinese over a text in English, though.
- JCharante 3y ago
  Per unit of time is true, because languages with a lesser information density will be spoken at a faster rate in order to compensate (see: Spanish speakers speaking way faster than English speakers). But LLMs don't ingest data per unit of time, they ingest data via text.
  - stavros 3y ago
    Sure, but, depending on your representation, you can only have so many different tokens, and if you use, say, pinyin, then you don't have much of an advantage over English. It'll be interesting to see how LLMs do when trained on concepts (ie the Chinese alphabet), rather than sounds, though.
    - JCharante 3y ago
      > Sure, but, depending on your representation, you can only have so many different tokens From my understanding (which only comes from reading HN), unique token count isn't an issue that LLMs run into. If it is, then that would be a bummer for the possibilities of exploiting all the features of the Chinese language. > and if you use, say, pinyin Well yeah pinyin would be inefficient because you're stripping away all the "built in" semantics of a Chinese character to only focus on the pronounciation.
      - stavros 3y ago
        I imagine you can always expand your token count, but then is that very different from syllabic alphabets (that encode multiple syllables in one character)? Granted, that doesn't apply to Chinese, which encoders concepts, so that's interesting to see in LLMs.
        JCharante 3y ago
        I think so, because even if you're encoding multiple syllables, that's just the pronounciation. I've learned languages with alphabets and glyph scripts like Chinese, and *what I don't like about alphabet languages like English is that the letters themselves provide little context to their meaning, although you could learn the latin root words and guess from there. Of course you know how to say it, but you don't know what it means. With Chinese, you might guess what it means, but you don't know how to say it.* The characters have more meaning in them, although this isn't the case for every word, which is something non-speakers assume. In short, from my experience learning Chinese was way easier than learning Vietnamese, which have historical ties together since Vietnamese used a fork of Chinese script until the early 1900s when they transitioned to latin script in order to improve literacy rates. Sure, it improved literacy rates because to read you only need to know how to pronounce, but it doesn't mean glyph languages are harder to learn the meaning/semantics.

czl_my 3y ago

Interestingly BaiChuan2 didn't just tokenize individual characters https://huggingface.co/baichuan-inc/Baichuan2-7B-Chat https://huggingface.co/baichuan-inc/Baichuan2-7B-Chat

visarga 3y ago

Text tokens by their construction are more like the Chinese ideographic characters, so no, there would not be a big difference between tokenized English and Chinese. The number of unique tokens is decided upfront - something like 50K - so, much more than basic vocabulary. English tokens can even be 2 word phrases. Look up Byte-Pair Encoding (BPE) for a fascinating and simple algorithm that explains the process of choosing the tokens. It shows why it doesn't matter how token dense the source language is.

kolinko 3y ago

Vram is dependent mostly on the parameter count, not on the number of input/output tokens, if I remember correctly. Also, in case of english language, it's one token per one word (more or less), so it's the same as in Chinese - assuming both LLM tokenisers were geared towards their native language. The only issue is when you have an tokeniser geared towards western languages, and you try to to use the same tokeniser on a different group of languages - then a single word in a language foreign to the tokeniser would have multiple tokens. But that has nothing to do with the underlying structure of the language. In other words - you wouldn't really see a difference between an input in chinese compared to english after the text gets tokenised. It's rougly the same amount of tokens, and the underlying parameter count would be also similar.