7 ms·
AI translates Hokkien, an unwritten language, for the first time
- gilgoomesh 4y agoIt is fascinating that there are such major languages (or language groups) without standard written representations. I understand the concept of formal literary languages but in this case, having literary Chinese as your "written language" feels like an anachronism. Like treating Latin as literary language in medieval Europe: ultimately disenfranchising to common people and something that needed to be set aside.
- SECProto 4y ago> Like treating Latin as literary language That comparison is a stretch, as Latin and English are both written alphabetically, while classic and the various modern chinese dialects/languages are written logographically. It's a delightfully complex topic! The headline is a bit misleading, as wikipedia has an entire article on written Hokkien - as you note, it isn't standardized. https://en.wikipedia.org/wiki/Written_Hokkien https://en.wikipedia.org/wiki/Written_Hokkien
- davidjohnstone 4y agoThe comparison between Latin in Europe in the early modern period and literary Chinese in the Sinosphere for much of history is pretty good (and often made). Latin and literary Chinese are very different languages, but had similar roles in society by being the standard written language in a world where everybody spoke different languages (that were heavily influenced by, or even derived from, but different to the written form).
- SECProto 4y agoFair enough, there are similarities. However, alphabetically written languages are the same as their spoken - learn how to pronounce a hundred letter combinations and exceptions, and as long as you know the spoken language you're good. However with latin, pronouncing it wouldn't help with comprehension. With logographically represented languages, the characters have more to do with meaning than with pronunciation. It's part of why Chinese characters were used for completely different spoken languages (Korean, Japanese)
- tokinonagare 4y agoThe AI community would be taken more seriously if they didn't over-exaggerate or even straight up lie about their achievements like it is done in that article title. Hokkien, specifically the version mentioned in the article (Taiwanese Southern Min) has two main writing systems: Chinese characters and Latin letters based romanization. Heck, that's actually more than most languages in the world! The Tâi-lô[1] system, a slight modification of Pe̍h-ōe-jī was recently standardized by the government. On the other hand, Chinese characters can be used to write Taiwanese, not just Classical Chinese or Standard Chinese (Mandarin), save for a few words that are often written using Latin or bopomofo letters. It's true that a large number of speakers have not been taught either of those writing systems, but it does not mean they don't exist. I've already met someone which taught me few words in Hokkien (the infamous chia̍h-pá--bōe 食飽未) and write it in Chinese characters for me. I'm working with various Sinitic languages, and Taiwanese is far from being the worst when it comes to available resources in written form (for my use cases dictionnaries). [1] https://en.wikipedia.org/wiki/T%C3%A2i-u%C3%A2n_L%C3%B4-m%C3%A1-j%C4%AB_Phing-im_Hong-%C3%A0n https://en.wikipedia.org/wiki/T%C3%A2i-u%C3%A2n_L%C3%B4-m%C3...
- vore 4y agoIf most people aren't able to use those systems, I think it's about as good as unwritten, especially if the target audience is your everyday speaker who wouldn't be familiar.
- tokinonagare 4y agoI disagree. It makes a big difference: creating a writing system is not a simple as it seems and it's a task that require quite a lot of work, especially when there is huge dialectal variations in-between dialects. When a system already exist it means there is already a bunch of tools available such as dictionaries, input method[1], a corpus of texts even if small or medium, including text books. That can be used both for AI and teaching. This make the task way less daunting than doing everything from scratch. In case of Taiwanese, there is a small but existing corpus of literature, as a 5 seconds Google/Wikipedia search shows[2][3]. [1] https://aclanthology.org/O16-1026.pdf https://aclanthology.org/O16-1026.pdf [2] https://zh.wikipedia.org/wiki/%E8%87%BA%E8%AA%9E%E7%99%BD%E8%A9%B1%E6%96%87#%E4%BB%A3%E8%A1%A8%E4%BD%9C%E5%93%81 https://zh.wikipedia.org/wiki/%E8%87%BA%E8%AA%9E%E7%99%BD%E8... [3] https://zh.wikipedia.org/wiki/%E5%8F%B0%E6%96%87%E5%B0%8F%E8%AA%AA https://zh.wikipedia.org/wiki/%E5%8F%B0%E6%96%87%E5%B0%8F%E8...
- contingencies 4y agoOne of the amusing 20th century historical anecdotes which I am constantly pointing out to mainland Chinese is that the lauded literary figure Lu Xun of Shaoxing actually advocated for the replacement of written Chinese with an alphabetic/romanized writing system based on phonetic transposition. China then invented pinyin, which is a relatively well conceived and elegant system, but refused to support the replacement of characters and instead advocated for "simplified" characters (which are approximately as hard as the originals, but provide a convenient cultural gap). While Lu Xun is now taught in all mainland Chinese schools, they conveniently leave out the part where he advocated for leaving characters in the past. His primary argument was the years wasted on rote learning that Chinese students require as a precondition to literacy.
- kmeisthax 4y agoJapanese tried the same thing - dropping kanji[0] and kana[1] entirely in favor of romaji[2]. It failed primarily for the same reason that English spelling reform fails, which is that we the horrifically mangled and confusing orthography to distinguish words spelled the same way. [0] Characters derived from Chinese. Each character has both an embedded meaning and one or more readings. They look like this: 漢字 [1] Characters that spell out whole syllables. The glyphs are typographically derived from kanji, but very simplified and without meanings. Due to medieval Japanese gender politics, there are two of them. They look like this: ひらがな and カタカナ. [2] The characters you are using right now.
- contingencies 4y agoJapan is the only culture I know of with three or four concurrent systems in use. China could be said to have two. Taiwan could be said to have three, if you could bopomofo.
- samus 4y agoJapanese could get rid of either Hiragana or Katakana if they really wanted, since both of them are fully functional syllabaries. But in Taiwan, the choice between Bopomofo, Pinyin, and their local variant of Pinyin has political significance because it is taken to be reflective of their stance to the Mainland.
- ekianjo 4y ago> Like treating Latin as literary language in medieval Europe: ultimately disenfranchising to common people not in latin countries which had languages and dialects not too far from Latin in the first place
- hadrianpaulo 4y agoThis is amazing work from Meta and Peng-Jen Chen. As a person who grew up having Hokkien as my first language, I've feared that Hokkien might go extinct over time. Having a formal English and Mandarin Chinese education had pretty much started to erase my Hokkien knowledge. I'm glad that AI technologies can be used to preserve this language.
- tokinonagare 4y agoThey won't. It will only help accelerate the move towards concentration to national languages (Standard Chinese in Taiwan and China, French in France, etc.) Why would anyone put the effort to learn a "small" language when it can be translated automatically for understanding? The way to preserve a language is putting humans in the loop. Creating content in that language; interestingly more and more shows are produced in Taiwan in or using non-Mandarin languages as a political way to mark a difference with the big neighbour. And having government support, notably at school (at young age) by allowing partial or total teaching in the language to be preserved.
- lettergram 4y agoThe better translation technology is the less likely it is I’ll ever learn a language. To put it another way, in 10 years when I visit China I could probably have an in-depth conversation without ever knowing Chinese. I’d the Chinese can come to a university and never need to learn English, they won’t. Your parents and community will teach out to speak and that’ll be the end of it. It’ll be closer to the Tower of Babel scenario. Every community will have their own language and dialect and the AI will just adapt. If a solar flare takes out electronics we won’t be able to understand each other. Alternatively, the AI will teach us to speak or an implant will teach us similar to the matrix. Those are the scenarios in the next 25-30 years I see.
- version_five 4y ago> 10 years when I visit China I could probably have an in-depth conversation without ever knowing Chinese Doubt it. When you learn a language you learn a lot more than the translation, you learn thinking and connotations that require a broader understanding that never get captured regardless of the translation quality, because they require knowing history and seeing previous uses in context that can't be captured in the translation. "Good" translation will do for language what wikipedia has done for knowledge. Everyone will get a superficial understanding, some will pretend that reading off a definitions is the same as actually knowing something, and the world will get a lot more shallow, empty conversations.
- throwawayyy2022 4y agoAs a native Hokkien speaker, I am shocked to find the AI does it so well. However, a significant part of Hokkien is from classic Chinese which has written symbols, and the other part is from some local dialect in Fujian.
- jeromegv 4y agoWe don’t know how much of that video is real and how much was pre-recorded (as they knew what the speaker would say). During that same presentation, we also know that Zuckerberg lied about the metaverse and how characters are able to move their legs (it was filmed).
- polygamous_bat 4y agoThe model is published here: https://huggingface.co/spaces/facebook/Hokkien_Translation https://huggingface.co/spaces/facebook/Hokkien_Translation We can try it out right now. Given all of this, I am much more reason to believe this more than Zuck's metaverse claims.
- smoldesu 4y agoWhat is Facebook's motivation to lie about small-batch AI experiments?
- Kilenaitor 4y agoHe did not lie about the legs. He said they were coming soon. No different than any other marketing video about a sill-in-development, upcoming feature.
- dkaoster 4y agoI had the privilege of taking a 10-week course on Hokkien when I lived in Taiwan, and while it is true that Hokkien is unwritten, it is also true that there are some classic poems that are really only intelligible when read in Hokkien. The evolution of Hokkien and how it relates to classic Chinese is utterly fascinating to me. I feel like my understanding of Chinese as a whole became a lot deeper after learning Hokkien. People say that Hokkien is a dying / dead language, but the way I see it is that it won't die out completely in my lifetime so it's still worth learning.
- e63f67dd-065b 4y agoCalling Hokkien unwritten is a bit of a stretch. I grew up watching Hokkien shows from Taiwan, and the subtitles have always used plain chinese characters. I'm not sure about Taiwan, but in Singapore and Malaysia, where the chinese community speaks a slightly different dialect, everybody can read using chinese characters just fine, even the elderly who can't really speak mandarin chinese.
- danieltanfh95 4y agoJournalists need to hype something up
- larryliu 4y agoAgree very much. Hokkien is 65 million people's language, more than Italy's population. There are Hokkien-speaking people in Hong Kong and China's Guangdong province. They write Hokkien in Chinese characters mixed with pinyin for these without a corresponding character.
- amilios 4y agoIt's written, it just doesn't have a standardized written form, it's written informally in standard Chinese character but it's modified from the spoken form in order to conform better to standard Mandarin forms, AFAIK. From Wikipedia: "if "pure, unadulterated spoken vernacular Taiwanese" were written exclusively in Chinese characters, with minimal use of Mandarin phrases, over 25% of morphemes would have no character, about 25% would have arbitrarily selected (yet more or less conventionally accepted) characters that are homophones or near-homophones, 10% would be written using characters exclusive to Hokkien, and 40% would be written with characters that have the correct sound and meaning. However, in more colloquial styles of Taiwanese Hokkien, the proportion of morphemes written with conventionally accepted characters would drop even lower than 40%."
- noobermin 4y agoThis might just be a difference between Taiwanese Hokkien and SG/Malaysia Hokkien. As with everything in Chinese dialects it varies from place to place, community to community, etc.
- armchairhacker 4y agoIs this an area where linguistics and the IPA [1] can help? You could translate the spoken language into IPA, and then write a model which translates the IPA into English. Or does the IPA not actually capture all of Hokkien's nuance? [1] https://en.wikipedia.org/wiki/International_Phonetic_Alphabet https://en.wikipedia.org/wiki/International_Phonetic_Alphabe...
- samus 4y agoNo need to use the IPA. There exist romanization systems for Hokkien: https://en.m.wikipedia.org/wiki/Pe%CC%8Dh-%C5%8De-j%C4%AB https://en.m.wikipedia.org/wiki/Pe%CC%8Dh-%C5%8De-j%C4%AB
- satisfice 4y agoWhat’s the point of having different languages if people are just going to translate between them?
- Loocid 4y agoCulture, history, the impossible task of getting billions of people to agree on a standard.
- evolve2k 4y agoYes! Also innovation. There are many instances of certain languages recognising a concept for which there is no direct translation in other languages. The spread of concepts across languages has provided meaningful benefit over thousands of years. People who are multi-lingual also speak of how after immersing in one language or another they start to dream and think in a new language and this adjusts their thought patterns. The affect of our dominate language on us is not neutral and different languages bring different benefits and framing.
- divbzero 4y agoLast year there was news of Apple hiring Hokkien speakers to train Siri [1] but those efforts have not panned out yet. [1]: https://international.thenewslens.com/article/146603 https://international.thenewslens.com/article/146603
- nuker 4y agoAI or ML? I think AI is when it started reasoning, as HAL9000, we are far away from that. Thats why cars cannot self-drive yet.
- tommiegannert 4y agoRemember to account for ten years of inflation. AI -> General AI ML -> AI Anthing-not-deep-neural-networks ML -> ML
- xisthesqrtof9 4y agoSomeone needs to put this on a headset and translate irl. I will buy it.