6 ms·
The Long Tail of the English Language
- jasode 12y agohttp://en.wikipedia.org/wiki/Zipf%27s_law http://en.wikipedia.org/wiki/Zipf%27s_law EDIT ADD: Zip's Law was not mentioned directly in the blog post but the reply clarified that the API returns a zipf score. However, a word's zipf ranking is dependent on the corpus used. The Wordapi "About" page[1] says most data came from Princeton WordNet but a sibling comment says it came from a subtitles compilation. If the project could clarify the data sources, it would be helpful. [1]https://www.wordsapi.com/about https://www.wordsapi.com/about
- impostervt 12y agoThe "frequency" score returned by Words API is the a Zipf score for the word. Ranges from ~1.6 to ~7.6. Regarding your update - I'll update the About page.
- japaget 12y agoThe source material for this frequency count comes from Open Subtitles [http://www.opensubtitles.org/en/search http://www.opensubtitles.org/en/search]. Hence the frequencies here apply to spoken English, not written English. In written English the three most common words are "the", "of" and "and", whereas here they are "you", "I", and "the".
- CapitalistCartr 12y agoI wonder how other languages compare to English. I know English is far from pure. "The problem with defending the purity of the English language is that English is about as pure as a cribhouse whore. We don't just borrow words; on occasion, English has pursued other languages down alleyways to beat them unconscious and rifle their pockets for new vocabulary." -- James Nicoll
- impostervt 12y agoSubtlex has done word frequency counts in a number of languages: Dutch - http://crr.ugent.be/programs-data/subtitle-frequencies/subtlex-nl http://crr.ugent.be/programs-data/subtitle-frequencies/subtl... Chinese - http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2880003/ http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2880003/ Greek - http://www.bcbl.eu/subtlex-gr/ http://www.bcbl.eu/subtlex-gr/ There's a few others (Polish, French, etc) but I can't find the links for some reason.
- sp332 12y agoAn interesting and large list of "English" words from other languages: https://en.wikipedia.org/wiki/Lists_of_English_words_by_country_or_language_of_origin https://en.wikipedia.org/wiki/Lists_of_English_words_by_coun...
- hellrich 12y agoWhat do you mean by "pure"? Lack of loanwoards? Lack of any linguistic changes (e.g., sound, meaning) for existing words? Lack of any innovation, i.e. no new words for new concepts/objects? You could try to find some of these things in languages spoken by rather isolated groups. Yet I don't think one should call such a language "pure", implying some kind or (moral) superiority.
- laichzeit0 12y agoCompared to Classical Latin for instance, I can sort of intuitively see how a "purity" comparison might make some sense. The grammar is more rich and precise, suffixes and prefixes attached to primitive roots are more regular in terms of meaning. Meaning is very decoupled from word order, which allows much richer possibilities for rhyme and prose. If the restored pronunciation is anything to go by, then the pronunciation is very regular. Then again I'm not a linguist. I would say it's like arguing whether Python is more "pure" than Perl. They're both Turing complete, they express very similar concepts but I'd wager most people would concede that Python feels "purer".
- deciplex 12y ago
- davedriesmans 12y agoVery nice product! As a sidenote: anyway knows a service that provides human associations? Eg What do you associate with "Hacker" as a person "Computer", "Night", "Internet"
- impostervt 12y agoIt seems like a good idea, but to automate it you'd need to maybe scrap websites and create a count of words that appear in the same sentence. Or you could get really crazy and start comparing subject/object relationships, etc.
- rspeer 12y agoI work on ConceptNet, which does this. http://conceptnet5.media.mit.edu http://conceptnet5.media.mit.edu
- mrfusion 12y agoSo I'm wondering, if you just learned the 200 most popular words, you might get pretty far in learning a new language, no?
- impostervt 12y agoProbably depends on the language. A three-year old supposedly knows about 1,000 English words. You could probably at least get around.
- maaku 12y ago200? no. Try 2,000. That should be representable of a barely usable level of the language. If you are what might be described as fluent, you're probably at >20,000. Take this headline: JACUZZIS FOUND EFFECTIVE IN TREATING PHLEBITIS "EFFECTIVE" and "IN" are the only two words found in the first 2,000 words sorted by frequency, although "FOUND" is close. So with only 200 words you'd understand: WNPHMMVF SBHAQ RSSRPGVIR IN GERNGVAT CUYROVGVF 2,000 words would give you: WNPHMMVF FOUND EFFECTIVE IN GERNGVAT CUYROVGVF And some grammar knowledge would tell you WNPHMMVF(N) FOUND EFFECTIVE IN GERNGVAT(V) CUYROVGVF(N) Which is enough to know that you only really need to look up "TREATING" in the dictionary to understand the gist of the sentence But it'd hardly pass as a fluent understanding...
- Rangi42 12y ago"Some grammar knowledge" probably also tells you the structure of the verb, namely TREAT+ING(V,present progressive). "TREAT" is the 1,118th most common word, so that gives you: WNPHMMVF FOUND EFFECTIVE IN TREATING CUYROVGVF And then some contextual knowledge of what sort of things get treated tells you that CUYROVGVF is a disease or injury, and WNPHMMVF is a medicine or therapy. So knowing 2,000 words and some grammar isn't quite fluent, but it's around the level of a middle-school child who has to ask what lots of nouns are.
- yongjik 12y agoThis is actually a very nice example of the importance of grammar. A typical novice English learner, even if they can look up all the words from the dictionary, would go on like this: "Hmm, Jacuzzis is clearly a noun, so it must be the subject. So Jacuzzis found "effective"... wait, that doesn't make sense. "Effective" is not a company, how does one found "effective"? Oh, it must be the past tense of "find". OK, so these Jacuzzis found... effective? How does one find "effective"? And why was effective hiding in "treating phlebitis"? OK, whatever, maybe I could decipher this if I understand "treating phlebitis"... So it is a kind of disease that is... treating someone (or something)? How could a disease treat anyone? This sentence makes no sense!!" If you think "Duh, that's ridiculous, nobody thinks like that", then you obviously never met a struggling English learner.
- dzdt 12y agoFor a corpus much more complete on the long tail, try google books ngram viewer : https://books.google.com/ngrams https://books.google.com/ngrams. That uses full data from google's book-scanning endeavors, millions of books compared to millions of words in the originally linked article.
- madaxe_again 12y agoI love to play with this, and see how thought waxes and wanes - a good one is to stick in every soviet premier since Lenin, or 20th c. US presidents - really tells you rather a bit about the mindshare that these individuals had.
- drpgq 12y agoI'm guessing "icing" is more popular in Canada.
- cpwright 12y agoOnly if you're thinking of icing on roads (or in hockey), but not if you're thinking of icing on top of a cake.
- uberalex 12y ago'I' doesn't seem to work. It is very common. https://books.google.com/ngrams/graph?content=I%2C+you&year_start=1800&year_end=2000&corpus=15&smoothing=3&share=&direct_url=t1%3B%2CI%3B%2Cc0%3B.t1%3B%2Cyou%3B%2Cc0 https://books.google.com/ngrams/graph?content=I%2C+you&year_...
- copsarebastards 12y agoTry lower-casing it. EDIT: This looks like a case of a common programming antipattern: you don't care about the casing for comparison purposes, so instead of implementing a case-insensitive compare, you downcase the strings and call it a day. But that's inherently a loss of data, and not having that data will eventually come back to bite you.
- impostervt 12y agoShould be fixed now.
- te_platt 12y agoIf you have the right kind of friends you can play "Who knows the most obscure word?". Everyone picks a word from memory, check each word's position (We used to use google ngrams, but this would work well.), whoever gets the least common word wins that round, repeat until it's not fun anymore. That's how I learned defenestrate, obsequious, and a few other words.
- pdpi 12y agoFirst word I tried wasn't even present in the list (equivocate). How do you score that?
- te_platt 12y agoEquivocate came up in https://books.google.com/ngrams https://books.google.com/ngrams. Looks like the linked tool isn't quite up for it after all.
- sdsykes 12y ago"Kajagoogoo" is the 96,714th most common word in the English language.
- HeyLaughingBoy 12y agoThat's cuz it's so shy...
- Terr_ 12y agoI love "sesquipedalian", because it's so self-referential.
- vorg 12y agoI love "non-self-referential" because if I use it enough times with all the correct stresses and pauses, I can build a computer.
- justaman 12y agoTIL "fucking" is the 214th most popular word in the English language.
- madaxe_again 12y agoNow I understand somewhat better why I often end up ceasing midway through discourse, as my colloquists often end up interjecting requesting definition of whatever neologism has just emanated from lexicon into conversation.... a random sampling of words that I have just today been asked to define either do not feature in this list at all, or are 90,000+ in terms of use. Brobdingnagian, deleterious, autodidact, loquaciousness... are these really such strange vernacular? I suppose this is what happens when you spend your formative years buried in literature - it probably doesn't help that I mispronounce all sorts, as books make poor elocutionists. I do worry that as we further and further consolidate our vocabulary that we lose the breadth and depth of thought that nuanced words provide... so did Orwell...
- aristus 12y ago1. Never use a metaphor, simile, or other figure of speech which you are used to seeing in print. 2. Never use a long word where a short one will do. 3. If it is possible to cut a word out, always cut it out. 4. Never use the passive where you can use the active. 5. Never use a foreign phrase, a scientific word, or a jargon word if you can think of an everyday English equivalent. 6. Break any of these rules sooner than say anything outright barbarous. -- Orwell, "Politics and the English Language"
- madaxe_again 12y agoI was referring more to newspeak, as Orwell's thinking progressed over the intervening five years, but the above are perfectly valid editorial rules. The thing is, note "will do" in (2). There are many cases where a short word really won't do, lest you lose nuance and meaning.
- delluminatus 12y agoAh, another person who talks like a book. I also get that quite regularly. I have a bit of a personal rule, though, where I never use a more complicated word when a simpler one would work. After all, conversation is at heart a means of communication. Although we might get a bit of a thrill from talking over people's heads, it doesn't really mean much of anything. And all it takes is one misuse -- for example, calling Brobdingnagian a neologism -- and we tear ourselves down more than we could hope to build ourselves up.
- jloughry 12y agoThe most striking thing about Randall Munroe's "Up Goer Five" comic [1] was that the word "computer" is on the list, but "thousand" isn't. "(Explained using only the ten hundred words people use the most often)" [1] http://www.xkcd.com/1133/ http://www.xkcd.com/1133/
- dullcrisp 12y agoWell to be fair, thousand is a number, so analysis of written text will find it a lot rarer than it is actually used/spoken.
- yellowstuff 12y agoI poked around in the neighborhood of "Tremulous" and saw a lot of obvious data errors: "ladiesand","manklnd", "confldentlal", "howdare", "monment". Other words don't seem particularly rare: "productively", "areolas", "combusts", "lazier".
- jboggan 12y agoIt seems like their long-tail data is full of misspellings. Try typing in "vacillate" and looking at the other words on the graph in relative frequency, for example.
- eginhard 12y agoNo corpus is ever "clean". Depending on the type of corpus there might be many misspellings, so obviously they will occur in the graph alongside other low-frequency words.