4 ms·

I spent months last year building algorithms to segment Chinese sentences into individual words (Chinese has no spaces) and disambiguate the meaning of the word

I spent months last year building algorithms to segment Chinese sentences into individual words (Chinese has no spaces) and disambiguate the meaning of the words (Chinese words often have many different translations). ChatGPT solved this problem practically as well as I have zero-shot. Here's a very difficult example that's not exactly correct, but better than what I could produce: > In the Chinese sentence "这次公司裁员裁不到我头上." with the translation "I won't be fired during this round of layoffs.", list all the words separately with their individual word translations 这次 - this time, 公司 - company, 裁员 - layoffs, 裁不到 - won't be fired, 我 - I, 头上 - during (literal translation: on top of the head) I knew this day would be coming but since nothing was available at the time I had to try an implement a hodge-podge of algorithms to get the job done. But honestly, if I can integrate ChatGPT instead I'm kinda glad! Thousands of lines of code gone, a much simplified pipeline, and I can focus on higher level tasks instead. What's important is the problem I'm trying to solve. But it does show that a lot of NLP and CV tasks will be as simple as explaining what you want, and the result will mostly be good enough. But there will still be value in solving more unique problems, or knowing the best way to mold problems such as they fit the foundational models available, and knowing how to build on top of them. To keep your sanity, my advice is have some problems in mind that you're passionate about solving as well so that you diversity your mental investment.

throwaway1183 4y ago
I was extremely passionate about text-to-image, but now that it is solved, I am experiencing a crisis. I have lost what I have been pursuing for a long time and I am now unsure of my skills. I fear that I will not be able to find a job because I only have internship experience and no real job experience. I was hopeful before pursuing this Master's degree, but now I am not so sure.
- martindbp 4y ago
  My intention was more picking a problem like "solve climate change" rather than "solve image segmentation". For me that's the space of human learning, language learning and Chinese. But for my day job it's more general AI. So, keep one eye on the tech for its own sake, but the other eye on what real-world problems you actually want to solve with it.
throwaway675309 4y ago
Wait what? When I used to live overseas nearly a decade ago I was able to leverage a fairly accurate Chinese tokenizer [I don't remember the name] but there are definitely options out there that existed for far longer than chat GPT. This one in particular I've had decent success with for 繁體漢字 and you could use a one to one dictionary to convert simplified to traditional: https://github.com/APCLab/jieba-tw https://github.com/APCLab/jieba-tw Hell, with the example you just provided you could probably just look for longest words found in CC-CEDICT.
- martindbp 4y ago
  That is indeed the 80% solution, but it's really not that simple to get good results. First of all, Jieba as a segmenter has a very high error rate, so I switched to transformer based one which is better, but still quite bad (it's a hard problem). You can mitigate some of this by joining long words that are in CEDICT (but do not cross other word borders!), but this often backfires. As for CEDICT entries, very often the entry translation does not exist in the sentence translation. Sometimes it exists with a morphological change like "fire -> fired", sometimes more difficult "go -> went". Sometimes it's a synonym which you may be able to find in a thesaurus, maybe not. The thesaurus is also incomplete or has errors. If you use a synonym you often get false positives from completely unrelated hanzi's in the sentence. Sometimes the CEDICT word has 8 different entries and some of them are rare meanings with low frequency words that match many sentences but really shouldn't, how do you detect that? Entries are not ordered by importance. Weight by specificity? Ok, but each entry has an average of 5 words, what if one entry matches 4 high frequency (low information) words but another matches 1 low frequency word? What if multiple words in the sentence has overlapping target words in the translation, which is which? And now you have words with different pinyin readings, so multiple completely different entries in CEDICT. So you try to first classify the pinyin (or the other way around, use translation correlation to classify pinyin?). But if the initial segmentation is bad, you've got garbage in, garbage out anyway. Then you have lots of two-character Chinese words where one is regularly omitted, like in my example if you look up 裁 in CEDICT you get "cut", but it's referring to 裁员 which is more like layoff or fired. "Cut" may be acceptable here, but there are many cases where if you do this you get completely nonsensical translations. Now you need to match names as well. What if names are not capitalized? If they are capitalized, should you match them in order they're mentioned if there are multiple names in the sentence? Ok, so most Chinese names will be in some form of romanization, so you look for something that looks like the Pinyin of the word. Oh, but sometimes it's Wade-Giles isn't it? Oh, and because the translation you have comes from machine translation, now sometimes you get Korean or Japanese names for some reason, try and match those too? In order to get more variety of word translations in context, I worked out something similar, where I give a machine translator the text "你好吗, 1. 你 2. 好 3. 吗", and often it will give in-context translations that are better, but often fails miserably as well. Now you've got to try to combine these outputs with CEDICT, and perhaps multiple sentence translations (human + machine?) to get the best possible result. I also tried joint optimization over segmentation, pinyin and word disambiguation, by maximizing overlap and low frequency mappings (high information), but never managed to get it working well enough. This is just a third of the problems and edge-cases you hit when trying to do this, I hope I've illustrated how much of a mess this becomes. A simple solution might work well enough for personal use as in your case, but I'm running through the scripts of TV-shows and displaying intelligent subtitles, so every error is quite jarring to the end-user. So being able to replace it with a ChatGPT prompt is quite the improvement! Because honestly, doing all this manual algorithm work can be an extremely frustrating process :)

throwaway1183 4y ago

I was extremely passionate about text-to-image, but now that it is solved, I am experiencing a crisis. I have lost what I have been pursuing for a long time and I am now unsure of my skills. I fear that I will not be able to find a job because I only have internship experience and no real job experience. I was hopeful before pursuing this Master's degree, but now I am not so sure.

martindbp 4y ago
My intention was more picking a problem like "solve climate change" rather than "solve image segmentation". For me that's the space of human learning, language learning and Chinese. But for my day job it's more general AI. So, keep one eye on the tech for its own sake, but the other eye on what real-world problems you actually want to solve with it.

throwaway675309 4y ago

Wait what? When I used to live overseas nearly a decade ago I was able to leverage a fairly accurate Chinese tokenizer [I don't remember the name] but there are definitely options out there that existed for far longer than chat GPT. This one in particular I've had decent success with for 繁體漢字 and you could use a one to one dictionary to convert simplified to traditional: https://github.com/APCLab/jieba-tw https://github.com/APCLab/jieba-tw Hell, with the example you just provided you could probably just look for longest words found in CC-CEDICT.

martindbp 4y ago
That is indeed the 80% solution, but it's really not that simple to get good results. First of all, Jieba as a segmenter has a very high error rate, so I switched to transformer based one which is better, but still quite bad (it's a hard problem). You can mitigate some of this by joining long words that are in CEDICT (but do not cross other word borders!), but this often backfires. As for CEDICT entries, very often the entry translation does not exist in the sentence translation. Sometimes it exists with a morphological change like "fire -> fired", sometimes more difficult "go -> went". Sometimes it's a synonym which you may be able to find in a thesaurus, maybe not. The thesaurus is also incomplete or has errors. If you use a synonym you often get false positives from completely unrelated hanzi's in the sentence. Sometimes the CEDICT word has 8 different entries and some of them are rare meanings with low frequency words that match many sentences but really shouldn't, how do you detect that? Entries are not ordered by importance. Weight by specificity? Ok, but each entry has an average of 5 words, what if one entry matches 4 high frequency (low information) words but another matches 1 low frequency word? What if multiple words in the sentence has overlapping target words in the translation, which is which? And now you have words with different pinyin readings, so multiple completely different entries in CEDICT. So you try to first classify the pinyin (or the other way around, use translation correlation to classify pinyin?). But if the initial segmentation is bad, you've got garbage in, garbage out anyway. Then you have lots of two-character Chinese words where one is regularly omitted, like in my example if you look up 裁 in CEDICT you get "cut", but it's referring to 裁员 which is more like layoff or fired. "Cut" may be acceptable here, but there are many cases where if you do this you get completely nonsensical translations. Now you need to match names as well. What if names are not capitalized? If they are capitalized, should you match them in order they're mentioned if there are multiple names in the sentence? Ok, so most Chinese names will be in some form of romanization, so you look for something that looks like the Pinyin of the word. Oh, but sometimes it's Wade-Giles isn't it? Oh, and because the translation you have comes from machine translation, now sometimes you get Korean or Japanese names for some reason, try and match those too? In order to get more variety of word translations in context, I worked out something similar, where I give a machine translator the text "你好吗, 1. 你 2. 好 3. 吗", and often it will give in-context translations that are better, but often fails miserably as well. Now you've got to try to combine these outputs with CEDICT, and perhaps multiple sentence translations (human + machine?) to get the best possible result. I also tried joint optimization over segmentation, pinyin and word disambiguation, by maximizing overlap and low frequency mappings (high information), but never managed to get it working well enough. This is just a third of the problems and edge-cases you hit when trying to do this, I hope I've illustrated how much of a mess this becomes. A simple solution might work well enough for personal use as in your case, but I'm running through the scripts of TV-shows and displaying intelligent subtitles, so every error is quite jarring to the end-user. So being able to replace it with a ChatGPT prompt is quite the improvement! Because honestly, doing all this manual algorithm work can be an extremely frustrating process :)