2 ms·
I could believe this for romance languages. I'd need somebody to explain how this applies in ideograms and non-western languages, and what the outliers are, vow
by ggm 4y ago
I could believe this for romance languages. I'd need somebody to explain how this applies in ideograms and non-western languages, and what the outliers are, vowels vs consonant.
- thaumasiotes 4y ago1. Non-western languages. There are no differences; this is not a category of any interest. Urdu is a non-western language, except of course that it is, in a global sense, very closely related to French. (Why do you think we say "Indo-European"?) 2. Ideograms. The modern language that comes closest to using written ideograms is Japanese. But the language that comes closest to what you're probably thinking of is Mandarin Chinese, where every character is fairly rare. You have exactly the same options that you do with any other language: you could render the text into an alphabet of interest, or you can do your analysis on the symbols the text is originally written in. That second approach will make some questions much easier; if you sample a letter and get န်, you can be certain you're not reading French. It will make analyzing Chinese harder because character frequencies are so low. (see https://lingua.mtsu.edu/chinese-computing/statistics/char/list.php?Which=MO https://lingua.mtsu.edu/chinese-computing/statistics/char/li... - the most common character in written Chinese is 的 at 4.1% of all characters; the second most common accounts for an additional 1.6% of all characters. The top 152 characters collectively account for 50% of all text.) This will mean that codebreaking requires longer texts.† It won't have much effect on the problem of identifying the language by sampling unenciphered monograms from it; that stays easy, because the commonly used characters are highly informative even if e.g. you're comparing written Mandarin to written Cantonese. You'll probably need more samples (for an equally reliable judgment) than you would for a script that didn't have so many unique characters. † This problem is significant. Specific content words will all be rare, and we can't really expect a message, even a very long message, to discuss every possible topic at the same frequency that the corpus of written Chinese as a whole does. Even if we could expect that, the amount of ciphertext you'd need to reliably distinguish the 3100th-most common character in this corpus (酱 [noun, 'sauce'] representing 0.09% of all characters) from the 2494th-most common (娶 [verb, 'to marry a woman'] representing 0.19% of all characters) would be quite large.
- deleted 4y ago[deleted]