6 ms·

This is the first time I use llama2 (I used the 13B one) -- and it absolutely struggles with Chinese despite it says "I am capable of understanding Chinese and

This is the first time I use llama2 (I used the 13B one) -- and it absolutely struggles with Chinese despite it says "I am capable of understanding Chinese and English" when asking "what languages do you understand". Is this expected? See an example below, it can't stay in Chinese at all. >>> 你知道海豚吗 Ah, 海豚 (hǎitún) is a type of dolphin! They are known for their intelligence and playful behavior in the ocean. Is there anything else you would like to know or discuss? >>> 请用中文回答 Ah, I see! As a 13b model, I can only communicate in Chinese. Here's my answer: 海豚是一种智能和活泼的 marine mammal他们主要生活在海洋中。它们有着柔软的皮服、圆润的脸和小的耳朵。他们是 ocean 中的一种美丽和 интерес的生物很多人喜欢去看他们的表演。

mchiang 3y ago
There is the qwen 1.5 model from Alibaba team. https://ollama.com/library/qwen https://ollama.com/library/qwen ollama run qwen:0.5b ollama run qwen:1.8b ollama run qwen:4b ollama run qwen:7b ollama run qwen:14b ollama run qwen:72b I would only recommend smaller parameter sizes if you are fine tuning with it.
visarga 3y ago
get yourself a proper Chinese model from China, they are hosted in the Ollama model zoo as well
- thrdbndndn 3y ago
  That's true. I was more just out of curiosity because ChatGPT has great Chinese capability even the 3.5 version.
Eisenstein 3y ago
Give Yi a shot.
xanderlewis 3y ago
Its Japanese ability is even worse... and by that I mean it's basically nonexistent. You have to really persuade it to speak the language, and even then it's very reluctant and outputs complete gibberish most of the time. Interestingly, trying the 'llama2:text' (the raw model without the fine tuning for chat) gives much better results, although still quite weird. Maybe the fine tuning process — since it presumably focuses on English — destroys what little Japanese ability was in there to begin with. (of course, none of this is surprising; as far as I know it doesn't claim to be able to communicate in Japanese.)
charcircuit 3y ago
>Is this expected? Yes, the training was primarily focused on English text and performance on English prompts. Only 0.13% of the training data was Chinese. >Does Llama 2 support other languages outside of English? >The model was primarily trained on English with a bit of additional data from 27 other languages. We do not expect the same level of performance in these languages as in English. https://llama.meta.com/llama2/ https://llama.meta.com/llama2/
int_19h 3y ago
Never, ever make assumptions about what the model can and cannot do based on what it tells you itself. This is one area where they hallucinate a lot, even top-of-the-line stuff like GPT-4 (e.g. it will happily "translate" languages that it has very little understanding of).
- thrdbndndn 3y ago
  Well I don't assume, I asked it explicitly what language it understands.
  - codingdave 3y ago
    Taking any answer from an LLM as accurate is in and of itself an assumption.
  - int_19h 3y ago
    You assumed that it knows which languages it understands, or that it knows that it doesn't know. This assumption is generally incorrect.
cyp0633 3y ago
Mistral-7B answers in Chinese only when I explicitly tells it to do so

mchiang 3y ago

There is the qwen 1.5 model from Alibaba team. https://ollama.com/library/qwen https://ollama.com/library/qwen ollama run qwen:0.5b ollama run qwen:1.8b ollama run qwen:4b ollama run qwen:7b ollama run qwen:14b ollama run qwen:72b I would only recommend smaller parameter sizes if you are fine tuning with it.

visarga 3y ago

get yourself a proper Chinese model from China, they are hosted in the Ollama model zoo as well

thrdbndndn 3y ago
That's true. I was more just out of curiosity because ChatGPT has great Chinese capability even the 3.5 version.

Eisenstein 3y ago

Give Yi a shot.

xanderlewis 3y ago

Its Japanese ability is even worse... and by that I mean it's basically nonexistent. You have to really persuade it to speak the language, and even then it's very reluctant and outputs complete gibberish most of the time. Interestingly, trying the 'llama2:text' (the raw model without the fine tuning for chat) gives much better results, although still quite weird. Maybe the fine tuning process — since it presumably focuses on English — destroys what little Japanese ability was in there to begin with. (of course, none of this is surprising; as far as I know it doesn't claim to be able to communicate in Japanese.)

charcircuit 3y ago

>Is this expected? Yes, the training was primarily focused on English text and performance on English prompts. Only 0.13% of the training data was Chinese. >Does Llama 2 support other languages outside of English? >The model was primarily trained on English with a bit of additional data from 27 other languages. We do not expect the same level of performance in these languages as in English. https://llama.meta.com/llama2/ https://llama.meta.com/llama2/

int_19h 3y ago

Never, ever make assumptions about what the model can and cannot do based on what it tells you itself. This is one area where they hallucinate a lot, even top-of-the-line stuff like GPT-4 (e.g. it will happily "translate" languages that it has very little understanding of).

thrdbndndn 3y ago
Well I don't assume, I asked it explicitly what language it understands.
- codingdave 3y ago
  Taking any answer from an LLM as accurate is in and of itself an assumption.
- int_19h 3y ago
  You assumed that it knows which languages it understands, or that it knows that it doesn't know. This assumption is generally incorrect.

cyp0633 3y ago

Mistral-7B answers in Chinese only when I explicitly tells it to do so