12 ms·
Finnish breaks natural language processors
- hibbelig 8y agoWhy can't the NLP split the compounds? I think this problem of splitting happens in multiple languages -- the submission mentions German, and then you have Chinese which doesn't have spaces, so you have to split words. (I understand that splitting Japanese into words is simpler.)
- htns 8y agoAnother complication not mentioned in the tweets is consonant gradation, in which the stem changes in dozens of ways when suffixed (examples: http://users.jyu.fi/~pamakine/kieli/suomi/vaihtelu/astevaihtelu.html http://users.jyu.fi/~pamakine/kieli/suomi/vaihtelu/astevaiht...), which makes Finnish a lot less cleanly agglutinative than most other languages. It's not like it can't be done, but it's yet another step (find the likely split points, then try to undo the consonant gradation) and I'd imagine casual foreign researchers aren't going to bother.
- hpaavola 8y agoFinn here. There are countless memes about oddities of our language. The one I like the most is "kuusi palaa". It can mean; the spruce is on fire the spruce returns the number six is on fire the number six returns six of them are on fire six of them return your moon is on fire your moon returns six pieces Good luck for all implementing NLP :)
- arka2147483647 8y agoThis is a specificially constructed corner case, not a general finnish thing: - kuu = moon - kuusi = spruce - palaa = returns - pala = piece From those you can see how one can intrepent the compound word (with or without suffix) however you wish. Eg: - kuu+si = your moon (where 'si' is the suffix which means 'your') - pala+a = pieces (where 'a' suffix is the plural)
- Whatitat90 8y agoI wonder what oddities lie in Linus Torvalds' "perkeleen vittupää" (https://lkml.org/lkml/2013/7/13/132 https://lkml.org/lkml/2013/7/13/132).
- stevekemp 8y agoI like the "dog example" too, which highlights the different forms/cases: https://linustechtips.com/main/topic/72936-english-swedish-german-and-finnish-decline-dog/ https://linustechtips.com/main/topic/72936-english-swedish-g... I'm a recent immigrant to Finland, and while the regular grammar makes some things simple, the cases and suffixes are enough to drive me crazy! The simple things are easy to handle if you're reading a textbook, but remembering them in real-time when speaking is very hard.
- BerislavLopac 8y agoNo wonder Tolkien used it as a basis for Elvish.
- uasm 8y ago> "kuusi palaa" Can we get a bit more detail here? Are those variations on pronunciation? How does a native Finn able to infer the correct meaning?
- gorbachev 8y agoNo, there is no variation on pronunciation. They sound exactly the same when spoken. The only way to infer what's meant is from context.
- simonh 8y agoMy favourite homophone example is a 92 character poem in Chinese[0], where every syllable is a slight variation on Shi. [0]https://en.wikipedia.org/wiki/Lion-Eating_Poet_in_the_Stone_Den https://en.wikipedia.org/wiki/Lion-Eating_Poet_in_the_Stone_... Interestingly when I showed the actual text to my wife, who is Chinese, she didn't understand what was funny about it until I asked her to read it to me, when she burst out laughing. That's when I found out she doesn't sub-vocalise, where I do all the time.
- apendleton 8y agoThese aren't really homophones though, right? Because they differ in tone?
- yorwba 8y agoThere are only 4 different tones, so by the pigeonhole principle there must be lots of homophones. On the other hand, most of the characters used to be pronounced differently when they were still commonly used as individual words, so the poem only works because it mixes modern pronunciation with classical grammar.
- emilfihlman 8y agoFinnish is pronounced extremely regularly. The way it's written is the way it's pronounced (in almost all cases).
- appsonify 8y agoOr counting in Finnish :D https://www.youtube.com/watch?v=AuNs1v_jtfs https://www.youtube.com/watch?v=AuNs1v_jtfs
- toolslive 8y agoWhat about Hungarian ? https://en.wikipedia.org/wiki/Finno-Ugric_languages https://en.wikipedia.org/wiki/Finno-Ugric_languages
- ahoka 8y agoIt's hard to spellcheck too, but there's hunspell which is pretty good and is used by a lot of open source products: https://en.m.wikipedia.org/wiki/Hunspell https://en.m.wikipedia.org/wiki/Hunspell
- miki123211 8y agoPolish has some of those problems too, at least the ones with conjugations. The words mom (appearing in your contact list), mom (as in call mom) and mom (as in send money to mom) are different. That applies to proper names too, and it isn't regular. It's not easy to figure oujt who the user refers to if you just have the contacts list and the command. Polish had problems way before nlp, for example "Annie has sent you a message" and "John has sent you a message" would be translated differently (because male/female), same for "2 minutes ago" and "5 minutes ago". Also programmatically formulating natural messages like "Dear John" is nigon impossible.
- szatkus 8y agoI started learning Finnish and it looks much worse. Twice as much cases, consonant gradation and puhekieli. Everything else seems on similar difficulty level as in Polish (maybe except genders, but it's pretty regular). But I may not have a proper perspective as Polish is my native language and I don't work with NLP (although I created a small tool to determine a word based on its inflected form with data from Wiktionary).
- gtrfreewifi 8y agoPuhekiäli voi olla vähä vaikiaa, mutta sitten on viälä eriksensä murtehet.
- ajuc 8y agoAll Slavic languages are quite quirky in this regard. Most native speakers don't realize it, but when you try to write correct message parametrized with numbers in Polish, you need 3 or 4 special cases: - 1 - 2-4, 22-24, 32-34, ... - 5-21, 25-31, 35-41, ... If you say "out of X" you also need to handle numbers 100-199, 100 000-199 000, 100 000 000-199 000 000, ... separately (because 100 = sto, and if number begins with "s" - "out of X" is "ze" instead if "z") - this is in combination with previous 3 cases, so "199 ouf of 199" is different from "99 out of 99", and different from "122 out of 122". So "Copied X file(s) out of Y" would be: - "Skopiowano 1 plik" - "Skopiowano 2 pliki z 3" - "Skopiowano 5 plików z 5" - "Skopiowano 1 plik ze 100" - "Skopiowano 2 pliki ze 100" - "Skopiowano 5 plików ze 100". Almost no software handles this correctly :) But spell checkers work OK (they just ignore relationships between words usually).
- pakitan 8y agoSo, does that mean that if OpenAI's text generator turns out to be as "dangerous" as it's being marketed, we can all just switch to Finnish as lingua franca and Skynet won't have a chance? :)
- breakingcups 8y agoThat implies we can all "just" switch. I have a feeling Skynet will have a better chance than I do!
- sergioisidoro 8y agoI've been trying to learn and "switch" to Finnish for 5 years. I can confirm this.
- DocG 8y agoThere was a joke that Estonian is more closer to Japanese than any other European language. As to not say its related to Japanese, but how much it differs from European ones. No future tense, no genders, but 14 cases. This illustrates nicely where finno-ugric languages(Finnish, Hungarian, Estonian) reside compared to other European languages. https://images.mentalfloss.com/sites/default/files/196.jpg https://images.mentalfloss.com/sites/default/files/196.jpg Benefit is that people are somewhat safe for scammers, as no translator does a good job. Downside is, none of the translators work and its way easier to just translate to english for example.
- a_imho 8y agoWhat is the benefit of genders anyway? Especially when gender neutral alternatives are available. It never made much sense to me, they introduce a lot of complexity for very little. Worse, nowadays you have all the problems of misaddressing and offending folks.
- iainmerrick 8y agoThere was an interesting/amusing post here the other day about learning German and how it relates to programming concepts. They suggested that gendered words work as a kind of parity bit for error correction.
- wolfgke 8y ago> There was an interesting/amusing post here the other day about learning German and how it relates to programming concepts. They suggested that gendered words work as a kind of parity bit for error correction. Slightly off topic (not about grammatical genus, but about German's case system): I often compare German's case system to a type system in a programming language: the verb expects the sentence's object in the suitable case. If you use a wrong type, it gives a compile error. In most cases, the case (type) that you have to use makes sense, but there are rare cases where you simply have to learn the case. The latter is like having to use a (programming) library that was designed by some other programmer with an interface that you would design differently. Just like you have to write a computer program that gives no compile error (e.g. because of typing), in German, you also have to formulate your sentences in a way that gives no compile error to its case (type) system.
- plq 8y agoThese are common problems for languages with complex morphology. Finnish, Japanese, Turkish and Hungarian are closely related in this sense and share a lot of NLP research. While implementing spell checkers for these languages need a bit more effort than just compiling a list of words, it's far from an unsolved problem. AFAIK Zemberek[1] is a Turkish spellchecker that implements two-level finite state morphology. Also, a small terminology fix: Verbs are "conjugated" but names/objects are "declinated". [1]: https://github.com/ahmetaa/zemberek-nlp https://github.com/ahmetaa/zemberek-nlp
- szatkus 8y agoWhat's the problem with Japanese? It's highly regular language, so it should be easy to tokenize. AFAIR the whole language has maybe few irregular verbs, compared to few hundreds in English.
- fluffycat 8y agoTurkish is also highly regular and has only one irregular verb. The main problem arises from morphological complexity. Because of these well defined rules that change the rich set of suffixes (>100 different), analysis of a lot of words ends up with many possible parses, it is hard to resolve this ambiguity. I can't speak Japanese, but if it is also morphologically rich, it should face similar problems.
- szatkus 8y agoIn Japanese I think it's pretty straightforward to trace the original word. But there is a fair share of homophones, so if your input is speech-to-text you already have many possibilities without much grammar.
- isani 8y agoJapanese is usually written without spaces. Words and sentences just run into each other. When writing in hiragana (syllabic characters), word boundaries are often ambiguous. Englishwouldbemuchhardertoparseifwrittenlikethis.
- mettamage 8y agoNatural Language Processor broken by Finnish language. :D Okay, I'll see myself out now.
- raverbashing 8y agoMy suspicion is that most NLP techniques were invented by native speakers of "easy" languages Though romance languages/germanic languages (and even English ancestors) have their quirks they're not at Finno-Ugric levels (Not that English has no quirks - especially in pronunciation - but it is "easy" to deal with most of the weird exceptions)
- aasasd 8y agoI still haven't figured out how Germans use swipe keyboards on smartphones with the monster compound sentence-words. If they do.
- hibbelig 8y agoOn iOS, the keyboard just suggests compounds. This works for those you use often. Another option is to choose a prefix, then backspace (over the space), then add to the prefix. Example compound word: Schleifmaschinenverleih (word boundaries: Schleif'maschinen'verleih) You begin with "Schl" which might suggest "Schleifer", then you delete "er", add "m", and if you're lucky, you get "Schleifmaschine". "Schleifen" means to grind. "Maschine" means machine. So "Schleifmaschine" is grinding machine, and the disappearance of "en" is a grammatical artifact. "Verleih" means rental (as in a company that does rentals).
- zorked 8y agoIt just means phones ship with larger dictionaries. And I would guess this isn't even the proper way to handle German. The long words are just sequences of words not split by spaces when writing. In a parallel universe the German writing system would use spaces and the language wouldn't need to change a bit. Words are features of writing systems, not of languages.
- perlgeek 8y agoI often just write separate words. Not correct, strictly speaking, but understandable. I still prefer regular keyboards so (desktop and laptop) -- not sure if that's related.
- Geimfari 8y agoI just write the words separately and remove the spaces afterwards. An "I'm about to write a compound word"-button would be an easy solution, but it shouldn't be hard for the keyboard to automatically join the words after you've finished your sentence, it's mostly rule based.
- Agentlien 8y agoThere's the same problem in Swedish. I usually swipe the words separately or, if they're short, I type them manually and then go back to swiping. The decision to type them usually comes when I've tried a few times to swipe it before realising it's just not in the dictionary. It's actually really frustrating. I absolutely love swiping, but it doesn't feel like it works very well with Swedish.
- stonewhite 8y agoTurkish being a member of ural-altaic language family, just like Finnish, suffers from similar problems. Google Translate improved much over the years yet it still generates laughable text at best. Stemmers used to create show-stopper word stems (don't really know the current situation). Although, it imported a lot of technical terms from various european languages, making technical texts seem to be more legible due to lack of composited words in these contexts. similarly from tweets: yiyecek = food yiyecek miydi = will he/she eat that (2nd word is not a separate word, just a conjunction that is written separately) göz = eye gözcü = scout gözlük = glasses gözlükçü = glass salesman gözcülük = scouting gözlükçüler = glass salesmen gözlükçüydüler = they were glass salesmen gözlükçü müydüler = were they glass salesmen? also an all time classic: çekoslavakyalılaştıramadıklarımızdan mısınız = are you one of those people whom we tried unsuccessfully to assimilate into a Czechoslovakian citizen?
- Lasokki 8y agoUral-Altaic language family is today considered an obsolete concept [1] and the families are considered unrelated [1] https://en.wikipedia.org/wiki/Ural%E2%80%93Altaic_languages https://en.wikipedia.org/wiki/Ural%E2%80%93Altaic_languages
- plq 8y agoThe models / software created for one language mostly fit others. Eg. Two-level finite state morphology was invented for Finnish and was very successfully adopted to parse Turkish words. So the opinions of Linguists aside, the languages that make up the ural-altaic family are not that far apart from each other.
- mda 8y agoYes,but both languages share some important structural similarities. Rich morphology, extreme agglutination and vowel harmony.
- celticninja 8y agoIn a similar vein, submitted on HN this morning: https://linustechtips.com/main/topic/72936-english-swedish-german-and-finnish-decline-dog/ https://linustechtips.com/main/topic/72936-english-swedish-g...
- Lorkki 8y ago> tietokone = knowledge machine (literally) = computer This is somewhat beside the point, but although in everyday speech "tieto" usually means knowledge in the sense of knowing a fact or skill, it can also be taken as "information", which seems more likely to have been the original intent of the term. Nevertheless, pretty much every elementary computing course in Finnish begins by stating that "the computer doesn't actually know anything". :)
- deleted 8y ago[deleted]
- Griceraae50100 8y agoReally interesting! I was wondering a few months ago how autocompletion on phones works for agglutinating languages like Finnish, but I didn't have a Finn to ask =/ If you wouldn't mind enlightening me, does it even exist for Finnish? If so, how does it work?
- chousuke 8y agoWriting Finnish on a phone is terrible until it learns all of your most commonly used words and their most common forms, at which point out becomes merely bad. Whenever I write Finnish using my phone I basically have to work around the keyboard not doing what I want. I'll still swipe to form words, but I have to go back to "fix" them because the system usually can't even handle simple compounds.
- sershe 8y agoI can tell you based on Russian that it works poorly compared to English, esp. for verbs. I'd expect to be even worse for Turkish (which I used to know on beginner level), or Finnish. When the word you are typing becomes obvious, it would offer 3 (3 is the default on my keyboard app) random forms that do not appear to take context into account, or if they do they're often wrong. You can either just finish typing, or choose one of those, erase the ending/suffixes and fix them.
- jeromebaek 8y agoThis is interesting. Especially the bit about compounding taking the meaning from figurative to literal. Modern NLP embeddings still assume a canonical meaning for each word and therefore cannot (immediately, anyway) distinguish between a figurative or literal meaning. Not that such efforts don't exist, but they all seem to be missing something fundamental - namely, is the "canonical" meaning, the vector, supposed to be literal, or figurative? This is a question with potentially no right answer, yet we all assume that the answer is "literal". At the same time, I do wonder if part of the problem with NLP algorithms and Finnish isn't so much its complexity but the fact that there id very little Finnish data to train on.
- gio2j 8y agoMorphology of Finnish language has of course been studied, and spell checkers for it exist. E.g. open-source: https://voikko.puimula.org/ https://voikko.puimula.org/ (and its online version https://oikofix.com/contact?lang=en https://oikofix.com/contact?lang=en, where you can also play with the analysis/parsing capabilities) Of course, the more complicated morphology makes it less straightforward than English where simple word list based approaches are sufficient.
- jasonhansel 8y agoLatin has similar issues. Thankfully we have Whittaker's Words and the Perseus Word Study tool to help. I'm surprised that Latin is (apparently) better off than Finnish in this respect.
- frosted-flakes 8y agoIt's interesting that Nokia's auto-completion worked well for Finnish, but modern solutions don't work. > Joose Rajamäki 🇫🇮🇪🇺 @joose_rajamaeki Feb 20 > Yes, autocompletion [on phones] exists. But I hardly ever manage to compose a message where it wouldn't encounter new words. Also, it doesn't know the inflections, so it needs to encounter each word in each possible form before suggesting them. > Joose Rajamäki 🇫🇮🇪🇺 @joose_rajamaeki Feb 20 > Old Nokia phones were good. They let you indicate that the word root was finished and you wanted to add agglutination and/or inflections.
- b0bby_tabl3s 8y agoHere is the best dose of Finnish I can recommend: Go to YouTube and search for Korpiklaani. Finnish Folk Metal! Your life will be better. :)
- dandare 8y agoI don't understand case #4, can someone explain please?
- wodenokoto 8y agoSeems like they are splitting their tokens wrong. Spaces are just a suggestion. But seriously, the problems outlined reminded me very much of Japanese and korean, just turned up to 11. Japanese has endless amounts of homonyms and at least in theory words can go on and on and on with added conjugations. They have a lot of compound nouns, often build from abbreviations of the words compounded. The author mentions that these compounds are a problem in Finnish NLP because the explode the size of the vocabulary. Written Japanese does not contain any word boundaries at all. They split on morphemes for NLP tasks, which helps against exploding vocabularies, but also disseminate the parts into their own meaning-unit. For written Japanese you have characters which can guide you in meaning for a lot of compound words, but they blow up your character space. That is not the case for Finnish, but I can't really decide if that is an advantage or disadvantage. Also, a spellchecker and a chatbot would rely on completely different techniques, so you can have one without the other. Japanese doesn't even have spellcheckers, but they have a ton of other tools that millions of people rely on everyday for writing faster and better text, like kana-kanji conversion and word suggestions.