6 ms·
How to understand half of Harry Potter book in any language (+ source code)
- nathell 15y agoReinventing Zipf's law, huh?
- legierski 15y agoOh, never heard about it, thanks for the tip!
- nathell 15y agoIf you haven't, and are interested in this kind of thing, I strongly recommend Manning and Schütze's "Foundations of Statistical Natural Language Processing". This is _the_ book if you want a gentle yet thorough introduction to statistical NLP.
- korussian 15y agoThat's a fantastic idea. I'm struggling to learn Korean, and it's tough because few of the words are recognizable to me. I have a base of English/French/Russian, so that doesn't help (much). I would love to try to put my grammar/flash cards aside and go the Harry Potter route. I can't code. What could you do to help me, an average user, do this with Korean?
- legierski 15y agoI don't know much about Korean, but if I were you I would just get a book and start reading it, no matter what. And check out somewhere online the most popular words in this language.
- klbarry 15y agoHe would have to learn the Korean alphabet first (there are some highly reviewed books on Amazon which I cannot attest to). I've heard it's not super difficult to learn, though.
- korussian 15y agoI learned the Korean alphabet (Hangul). It's by far the easiest/most logical natural language alphabet to learn, and takes just a few hours to get reasonably proficient. It rocks and I read it well... the trouble is I don't know what I'm reading. Korean is opaque, and though I have a book of the 6000 most used words in Korean, I don't find it very helpful when picking up a children's book, for example.
- TeMPOraL 15y ago> I can't code. I guess there are many people who would benefit from this idea and, like you, can't code. It sounds like a good idea for an Internet service - a list of words to check before reading book X in language Y.
- cgag 15y agoI don't think being able to code or do any sort of analysis is particularly valuable. When you read a 50,000+ word book, you're going to see the top 50 words enough times to get them down pretty well. I know I've picked up a lot of words like wand and owl from my copy of la piedra filosofal. I think just go get copies of books you know well, or childrens books (harry potter, the little prince) and start reading with a dictionary or wordreference near by. You could also pick up a frequency dictionary if you just want to build a base level of vocabulary.
- protractor 15y agoTo plug a friend of mines startup: www.talktomeinkorean.com
- neop 15y agoI'm also learning Korean and I'm looking for a book to try and read. Unfortunately I haven't found any suggestions. From what I've heard children's books are actually harder to read because they use a lot of words that only children use. Harry Potter might be a good idea, but it seems to me like it would have a lot of fantasy related vocabulary which won't be very useful. A story set in modern day real world would probably be better.
- hkolek 15y agoI like the approach but I think it's a big mistake to not strip stop words. He should focus on nouns and verbs imo. The top words he lists are all stopwords/grammatical particles "de", "que", "la", "y" etc. I don't think knowing those words will help to understand anything. I think if you understand only the grammatical particles in a sentence it won't help at all to understand the meaning of the sentence. On the other hand if you know the verbs and nouns but not the grammatical particles you can at least infer some meaning or what it's about.
- legierski 15y agoYou can always print out, let's say, 50 top words and not bother with "de", "que", etc.
- aohtsab 15y agoWhen I was starting out in German, I started listening to the audiobooks and reading the German text. Knowing the small words (prepositions and other stopwords) isn't of much use on its own, but having a general feel for the plot and knowing the context of what 'should be happening' in the book helped give me a much broader understanding (and richer vocabulary!) than merely ploughing through the German 101 textbook. tl;dr: Applaud the idea, but it's misguided. You need to focus on understanding words in greater context to derive any meaning.
- vidarh 15y agoMy final year I forced my way through Faust by using an old Danish translation side by side with the German version to bring my grade up. I'm Norwegian, so the old Danish translation gave me a "halfway house" - modern Danish is very understandable for a Norwegian; older Danish gets quite a bit closer to German. I still missed a lot of what was going on, but it brought me up a full grade in a semester. Of course I'd probably had done just as well with less effort had I chosen something easier - I was being pretentious about it.
- raphman 15y agoNice. Additional stemming would probably provide better data, however.
- thisisstupid 15y agoThis is kind of stupid... You should look into statistical machine translation. And Zipf's law. And any introductory book about natural language processing, as this kind of thing is pretty embarrassing.
- goblin89 15y agoI like HP series in this regard. Vocabulary complexity slightly increases with each book, which helps to progressively learn a language.
- deleted 15y ago[deleted]
- mseebach 15y agoSo, the tl;dr is that this guy discovered that using a dictionary is a good way of learning words in a different language. In more detail, there's an assertion and a proposed solution - and nothing to even remotely back up the assertion? Show me a page of Harry Potter in Spanish translated in this manner - I somewhat doubt it will make much sense.
- ajuc 15y agoThat method of learning foreign language was adviced by David Snopek. American that learned Polish that way in 1 year (and speaks fluently - this is very hard for non-native Polish speakers, most people have problems with producing correct sentences at all after one year). His post about this: http://www.linguatrek.com/blog/2010/12/harry-potter-the-book-that-taught-me-polish http://www.linguatrek.com/blog/2010/12/harry-potter-the-book... You can guess surprisingly big part of a book basing just on the context and a few words you already know. That's how kids learn their native language. It's a good method, it worked for many people (including me:)) and I don't understand why are you arguing it isn't.
- mseebach 15y agoI'm not arguing it's a bad way to learn a language, although I can't see how it will help you pronounce, but that's a different discussion. Quite the opposite, I'm in deep agreement with the article's assertion that focussing on grammar is counter-productive. 20 common nouns, 20 common verbs, "please" and "thank you", and you're actually able to have a basic conversation. I'm saying that this guy starts off OK, then goes on to devise some code to do that, then declares success apparently without running the code and reasoning about the output. He says that 50 words are 50% - but his script stops at 20? And of those, he lists ten, which are, as others have pointed out, grammar artefacts and stopwords - completely without consequence to getting to read the book in question.
- pm215 15y agoThere's some similar statistics for Japanese novels here: http://pomax.nihongoresources.com/index.php?entry=1223045359 http://pomax.nihongoresources.com/index.php?entry=1223045359 which I think show that the problem is not at the "most common" end of the distribution but at the "least common" end. The jump between '80% understanding' and '90% understanding' requires knowing an extra 5341 words, 90% to 95% needs another 7495, and so on. Basically the long tail is really nasty, and even 90% understanding is still not knowing one word in ten...
- cskau 15y agoThank you for sharing this. I'm doing something related at the moment, so it's nice to see previous work like this.
- acslater00 15y agoTL;DR Nearly half of the word occurrences in Harry Potter are prepositions, so if you learn a small number of them you can claim that you "understand half of Harry Potter". For example, you can absorb sparking dialogue like the following: "Harry and to at to I to with Voldemort or what to and I do for, Hermoine!!"
- jenius 15y agoYeah really, this is absolutely dumb. How did this get to the front page of hacker news? Knowing those words will literally get you nowhere if you are trying to learn spanish. This is like trying to learn a javascript and finding that in reality it's not that hard because half the language is open and close parens, semicolons, and the word "function". Fluent in javascript!
- thisisstupid 15y agoThe logical implication is that a lot of people on HN aren't that bright.
- skore 15y ago"If you read this far, you should follow me on Twitter." I remember there was an article about this a while back which concluded that using the words "you should follow me" gets the highest conversion of readers to twitter followers. This guy gets a front page post and is at 208 followers. Maybe we can finally lay it to rest as yet another instance of the useless lifehacker circlejerk cargo cultist behavior and agree that the only thing that makes people follow your feed is good content? Please?
- kiiski 15y agoOne should also remember that not all languages use so many prepositions. In Finnish, for example, we usually change the ending of a word instead of using prepositions. For example, the previous paragraph in Finnish: "Pitää myös muistaa, että kaikissa kielissä ei käytetä yhtä paljon prepositioita. Suomessa, esimerkiksi, muutamme yleensä sanan loppua preposition käyttämisen sijasta"
- pooriaazimi 15y ago> ... as I’ve already read all 7 books in 2 languages before - I could pick up a lot just from context... That's what I've always said to my friends, but they insist on learning grammar, which of course they'll forget after a few hours/days. Three years ago I could hardly read anything non-technical in English, now I can understand Lolita - which is a hard novel to read for a non-native. All thanks to audiobooks. I found that audiobooks are the most fantastic way of learning a new language... I couldn't have possibly read Lolita or Silmarillion before - They're just too hard for someone who is trying to learn a new language. Long sentences full of new/invented words - It's easy to lose the thread. But listening to a skillful reader reading them aloud, you can understand even the most complicated words and sentences just from the context, and the reader's tone and emphasize... If you want to learn a new language, do yourself a favor and listen to some audiobooks. Pick a book you've already read in your native language (preferably more than once) and you'll be amazed how easy it is to understand and learn new words (you must have a basic understanding of that language of course).
- klbarry 15y agoThat is amazing advice that I will take up later this year. Thank you.
- krelian 15y agoAudiobooks are great for learning the spoken part of a language but I would guess that most of us "technical people" need and want to learn things from the bottom up. That's the only way we feel that we really know something. It's true that grammar can be very complicated and will be easily forgotten without practice. A good practice is reading a book and trying to see not only if you can understand what is happening but also analyze each sentence and see if you can understand which grammar rules are used in its construction and why.
- jakobe 15y agoThat may be why technical people might have a hard time learning languages.
- pooriaazimi 15y ago
- rivalis 15y agoNlp folks call those "stopwords," because they don't contribute much to statistical understanding of text. That is, in most nlp applications, those words are removed to leave more meaningful text behind. How did this make front page?
- krelian 15y ago>did you know that out of 5 most popular languages in the world, 3 of them are relatively easy to acquire? They are: English, Spanish and Russian, and my plan is to be fluent in English and Spanish and be able to get by with Russian by the end of 2013! Who’s with me?!) I'll grant that Spanish is relatively easy but Russian is considered one of the most difficult languages to learn. It's hard for me to judge the difficulty of English but I wouldn't say it is an easy language.
- apendleton 15y agoRussian is probably somewhere in between. How hard a language is to acquire depends a lot on what language or languages you already speak, but for native English speakers, Russian is considered difficult, but probably not "one of the most difficult." The Defense Language Institute, which provides instructions for native-English-speaking US military translators, classifies Russian as a category 3 language (of four categories), which makes it harder than French or Spanish (category 1), or German (category 2), and on the same level as Farsi or Hindi, but easier than the a reasonably-sized swath of category 4 languages that includes Mandarin, Japanese, and Arabic.
- tolliator 15y agoAs a native russian speaker, I can say with utmost certainty that Russian is NOT the easiest language to learn. In fact, I would argue that it is somewhere on the upper scale of difficulty. I have been living in North America only 10 years - and I can't even teach Russian to my own kids - we had to get a tutor.
- wrs 15y agoLinguality (http://www.linguality.com/ http://www.linguality.com/) prints French and Italian novels with the original text on the right pages and a page-specific mini-dictionary on the left pages. No need to keep stopping to look up words in a dictionary. Unfortunately there are only a few Linguality books. Can you do this with an e-reader?