3 ms·
I've recently come across a word frequency set of Chinese subtitles [0] and the word distribution doesn't seem to follow Zipf's Law / The power distribution, wh
by wodenokoto 10y ago
I've recently come across a word frequency set of Chinese subtitles [0] and the word distribution doesn't seem to follow Zipf's Law / The power distribution, which seems counter to all my textbooks on statistical linguistics.
This is not discussed in the associated paper either, though the paper concludes that the distribution is a good representation of common Chinese words (i.e., it isn't skewed). Any clues to why that is?
[0] http://crr.ugent.be/programs-data/subtitle-frequencies/subtlex-ch http://crr.ugent.be/programs-data/subtitle-frequencies/subtl...
- rahimnathwani 10y ago"though the paper concludes that the distribution is a good representation of common Chinese words" I don't know why the word distribution doesn't follow a power law, but I can tell you that it isn't a good representation of common Chinese words. I downloaded the data set a while back, intending to use it for improving my vocabulary. I found some words (like the Chinese word for 'vampire') had too high a frequency. The lists are based on American TV shows' Chinese fansubs. I guess vampires and zombies are common themes.
- wodenokoto 10y agoI completely agree that the lists looks odd in many regards, but they test it on humans, and find correlations with frequency in the list and how quickly native speakers can recognise the word. I don't consider myself an expert in word-frequencies, but the article makes a convincing argument that these frequencies are representative of natural Chinese and a better or just as good measure as their baseline corpus.