6 ms·
Why are there so many English-first AI models from China? Are they not interested in serving their own population? Or is it that if they publish Chinese-first m
by lvl155 1y ago
Why are there so many English-first AI models from China? Are they not interested in serving their own population? Or is it that if they publish Chinese-first models it won't get publicity in the West?
- whynotmaybe 1y agoHaven't we reached a situation where English is the de facto language of scientific research, especially AI benchmarks ? It's clearly impossible for me to try anything in Chinese, I'd need a translation.
- xmichael909 1y agoCorrect. Lingua franca for at least the last 75 years, if not longer.
- numpad0 1y agoFor publishing results, yes, but not necessarily for the generation part of it.
- unsupp0rted 1y agoLess and less, it feels like, every year. I wonder if anybody has hard numbers on that.
- enlyth 1y agoI assume a large portion of high quality training material is in English
- sigmoid10 1y agoYou'd be correct. The largest portion of all languages in Common Crawl (aka the "whole open internet" training corpus) is English with 43%. No other language even reaches double digit percentages. The next biggest one is Russian at 6%, followed by German at 5%.
- Svoka 1y agoI wonder where are you getting your data. According to wikipedia russian is #7 https://en.wikipedia.org/wiki/Languages_used_on_the_Internet https://en.wikipedia.org/wiki/Languages_used_on_the_Internet Only place where russian is in top 5 is in Wikipedia views. Russian part of internet steadily goes down, as russian imperialism crumbles.
- div72 1y ago> The largest portion of all languages in Common Crawl https://commoncrawl.github.io/cc-crawl-statistics/plots/languages.html https://commoncrawl.github.io/cc-crawl-statistics/plots/lang...
- Svoka 1y agoThanks! I wonder where this discrepancy comes from
- tough 1y agoprobably under-indexing of non-english sources by these crawlers. would be interesting if yandex opened some data sets!
- simongray 1y agoAnd lots of people write on the web using English as a second language, which both reduces the presence of their native language and increases the presence of English.
- tough 1y agoyep not a native english speaker here and yet my online footprint is mostly english due to software pushing me to learn it
- numpad0 1y ago
- chvid 1y agoAll LLMs are trained on the same basic blob of data - mostly in English, mostly pirated books and stuff.
- eru 1y agoThat's wrong. Many LLMs are trained on synthetic data produced by other LLMs. (Indirectly, they may be trained on pirated books. Sure. But not directly.)
- loufe 1y agoLikely the case for established model makers, but barring illegal use of outputs from other companies' models, a "first generation" model would still need this as a basis, no?
- eru 1y agoWhy illegal? The more open models (or at least open-weight models) should allow using their outputs. Details depend on license. But yes, 'first generation' models would be trained on human text almost by definition. My comment was only to contradict the claim that 'all LLMs' are trained from stolen text, by noting that some LLMs aren't trained (directly) on human text at all.
- bilbo0s 1y agoThe mandarin language models obviously exist, but what would you do with them if they provided access to them? And what knowledge would be in them? What is the body of knowledge encoded in Mandarin? What does that look like? Sad reality is that not many outside of China have the facility with Mandarin to use those models. Even non-native Mandarin speakers who claim to be "fluent", are often messing up intended meaning in text. Or making literal translations that wind up making no sense. Inside of China, llm use will be Mandarin based. Outside, it seems to me English is the natural choice. Irony of Irony, probably the best way for a non Mandarin speaking layman to test a Mandarin based model would be to use another LLM to translate prompts to Mandarin. It's a sad future we're looking at. Or a brilliant one. Time will tell.
- johnla 1y agoFor it to be brilliant, AI needs to be a benevolent tool all the time. It would take just a few malignant actors to turn our world upside. I suspect it'll follow the same Internet and social media path. Great at first, grow markets, bring us together and then take a turn.
- horacemorace 1y agoYou’re right of course. That’s why these open source / weight releases are so critically important.
- eru 1y ago> Even non-native Mandarin speakers who claim to be "fluent", are often messing up intended meaning in text. Or making literal translations that wind up making no sense. Happens with English as well, but non-native speakers of English still benefit from these models.
- mensetmanusman 1y agoEnglish won. The Chinese youth struggle to write their own calligraphy characters they can read now. Typing favors English.
- rahimnathwani 1y agoIt's easy and fast to type Chinese sentences using a keyboard.
- throwaway519 1y agoThe pendulum already turned back. The current generation under 20 grew up with touchscreens. That obseletes input with pinyin; many don't care if the device has no keyboard.
- thenthenthen 1y agoInput is so interesting in China, basically a sorta t9 but just single letters and picking the right characters, with common/frequently used characters first, using pinyin. For example to say “ How are you?” You just type “nhm” (Ni Hao Ma) and 你好吗 shows up as suggestion/autofill. You can make surprisingly long sentences using this method.
- olalonde 1y ago> That obseletes input with pinyin Uh? Pinyin input is by far the most popular input technique in China. I rarely see anyone using handwriting input. That being said, it has nothing to do with English winning. It's just a Chinese input technique that uses the latin alphabet. English fluency in China is not very common, especially spoken English.
- eru 1y agoJust like German is written with almost the same alphabet as English, but that doesn't give you English fluency.
- syndeo 1y agoMy father-in-law here in China uses handwriting input, but everyone else I've seen here uses Pinyin, and it's totally fast and natural for them. And very true about the English. With some exceptions (of course), folks here maybe know a handful of words at best, and even then, pronunciation is usually pretty rough. People here really aren't using it; they are perfectly comfortable with their Chinese, and why wouldn't they be? Anyone saying otherwise clearly hasn't been here to see it firsthand.
- 34679 1y agoNearly everyone in the urban areas of China spoke some English when I visited way back in 1995. It's a bilingual society.
- rahimnathwani 1y agoI lived in Beijing and Shanghai for 9 years (2010-2019) and this is NOT my impression at all.
- crazygringo 1y agoThis is not true. I was in Beijing around then and never met a single person who spoke English if they hadn't learned it for professional reasons (they worked in tourism, international business, etc.). It could not have been further from a bilingual society.
- gcy 1y agoI suppose you probably were visiting some university districts/CBDs where people likely to have received higher education. Elsewhere, aside from basic "hello"/"how are you", locals in general are not able to communicate in English.
- syndeo 1y agoNot sure which part you were in, but this is just not true in my experience. I've been to Beijing, Shenzhen, Guangzhou, and some others, and Mandarin really is a must if you want to even have a chance of communicating. I can't imagine how I'd function here if I only had English. I've not yet been to Shanghai, and while I would expect the English-speaking percentage to be a bit higher, it would still likely only be in the single-digits by my estimation.
- choutianxius 1y agoOne reason is that there is no "good" search engine in China. The most popular one, Baidu, is like garbage compared to Google search. The most useful training data in Chinese would likely be from the social media and video sharing platforms, which I guess is much more difficult to crawl and clean up.
- thoroughburro 1y agoA few thousand years of literature ain’t nothing…
- fwipsy 1y agoGiven premodern population sizes and literacy rates, historical texts probably don't exist in anything like the quantity that internet posts do. Even if they did, the information may not be relevant to the modern world.
- kccqzy 1y agoPeanuts compared to the discourse available on the internet. The literature that survived thousands of years are cream of the crop; you won't find lots of random unimportant dialog between people thousands of years ago, but you find that on Reddit.
- littlestymaar 1y ago> The most popular one, Baidu, is like garbage compared to Google search It must be very bad when you see the walking turd that Google search has become over the years…
- spacebanana7 1y agoI wonder whether English text having fewer characters provides an advantage somehow.
- jmole 1y agonot really, since tokenization combines multiple characters
- paulsutter 1y agoI don’t see any indication that it’s English-first?
- yyhhsj0521 1y agoChinese internet mostly consists of a few closed gardens tightly controlled by big corps. Crawlers simply don't work when each company employs an army of engineers to guard their data. Many of the most popular websites are also app only. It's impossible to get the corpus necessary to train a good LLM.
- bredren 1y agoDo we have estimates on the corpus that is available? This model's repo describes "multiple strategies to generate massive diverse synthetic reasoning data." FWIW, AI 2027 forecasts heavy emphasis on synthetic data creation. Is the lack of existing corpus just an extra hurdle for Hanzi-first models that are also leading the pack in benchmarks?
- AlexCoventry 1y agoDeepSeek claims they had 12% more Chinese tokens than English, in their training corpus for DeepSeek V2, FWIW. https://arxiv.org/pdf/2405.04434#page=12 https://arxiv.org/pdf/2405.04434#page=12 > Our tokenized pretraining corpus contains 8.1T tokens, where Chinese tokens are approximately 12% more than English ones.
- Leary 1y agoThey are not "English-first". Deepseek-R1, for example, reasons in Chinese when you ask it a question in Chinese.
- overfeed 1y agoWhy are so many American models multi-lingual, supporting hundreds of languages not commonly spoken in the United States? Could it be that being multilingual results in a larger pool of human knowledge on the technical side compared to training on just a single language or 2. And on the business side, supporting more languages results in a larger TAM (total addressable market). Using english-language dataset for training LLMs is the default, not the other way like you insinuate.
- achierius 1y agoThat's clearly a different question. It'd be possible for these models to be Mandarin-first while still supporting other languages, like American models are English-first while doing the same, but that's not what's happening.
- overfeed 1y ago> That's clearly a different question. It'd be possible for these models to be Mandarin-first while still supporting other languages What would a hypothetical "Mandarin-first" model look like to you? I challenge the notion that the current models are "English-first" - that is an unsubstantiated opinion not supported by fact. I bet, dollars to donuts, these models are SoTA in Mandarin as well. When framed that way, asking "Why are they marketed as English-speaking models outside of China" or "Why are they really good at English" are simply not interesting questions - they have obvious answers.
- yorwba 1y ago> What would a hypothetical "Mandarin-first" model look like to you? Given a language-agnostic prompt like "12 + 89", any explanatory text it outputs could be expected to be in Mandarin most of the time. According to this test, Xiaomi's MiMo-7B-RL is an English-first model.
- overfeed 1y ago"12 + 89" uses the latin alphabet and is in no way language-agnostic in this context. I expect borrowed constructs to appear relatively more frequently in the language they were borrowed from. Now I'm curious how Mistral models would respond to a "language-agnostic" phrases like "Rendezvous" or "coup d'etat"
- Havoc 1y agoI was under the impression that we just see the English stuff given that we're using English news channels.
- throwup238 1y agoCommonCrawl [1] is the biggest and most easily accessible legally acquired crawling dataset around, collecting data since 2008. Pretty much everyone uses this as their base dataset for training foundation LLMs and since it's mostly English, all models perform well in English. [1] https://commoncrawl.org/ https://commoncrawl.org/
- lwansbrough 1y agoI’m going to go with: to ensure it is not disadvantaged in benchmarks
- julianozen 1y agoOne thing I thought was interesting about this paper [1] on understanding LLMs was how the models associate words/concepts in different languages with each other in what they call Multilingual Circuits. So the example they give: English: The opposite of "small" is " → big French: Le contraire de "petit" est " → grand Chinese: "小"的反义词是" → 大 Cool graphic for the above [2] So while English is the lingua franca of the interenet and represents the largest corpus of data, the primary models being built are able to use an English dataset to build associations across languages. This might create significantly stronger AI and reasoning even for languages and regions that lack the data, tech and resources to build local models [1] https://www.anthropic.com/research/tracing-thoughts-language-model https://www.anthropic.com/research/tracing-thoughts-language... [2] https://www.anthropic.com/_next/image?url=https%3A%2F%2Fwww-cdn.anthropic.com%2Fimages%2F4zrzovbb%2Fwebsite%2Fe0e156ea6c912a385d66ed562187fced8c392a58-1650x750.png&w=3840&q=75 https://www.anthropic.com/_next/image?url=https%3A%2F%2Fwww-...
- revskill 1y agoChinese is hard.
- Barrin92 1y ago>Or is it that if they publish Chinese-first models it won't get publicity in the West? This is a large part of it. Kai-Fu Lee's company (https://www.01.ai/ https://www.01.ai/) has been publishing open source Chinese language/market focused models pretty early, but the entire conversation around Chinese tech just isn't available to you if you don't speak Chinese, in particular these days given that good English language reporting on the Chinese tech sector just seems very scarce.