4 ms·

Just last year, there was also this publication: Sander Lestrade: Unzipping Zipf’s law https://journals.plos.org/plosone/article?id=10.1371/journal.pone.018198

Just last year, there was also this publication: Sander Lestrade: Unzipping Zipf’s law https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0181987 https://journals.plos.org/plosone/article?id=10.1371/journal... "In spite of decades of theorizing, the origins of Zipf’s law remain elusive. I propose that a Zipfian distribution straightforwardly follows from the interaction of syntax (word classes differing in class size) and semantics (words having to be sufficiently specific to be distinctive and sufficiently general to be reusable). These factors are independently motivated and well-established ingredients of a natural-language system. Using a computational model, it is shown that neither of these ingredients suffices to produce a Zipfian distribution on its own and that the results deviate from the Zipfian ideal only in the same way as natural language itself does."

wodenokoto 4y ago
I remember reading that article when it was on the front page of HN, and felt underwhelmed by the conclusion. Reading the quoted conclusions again, I almost fail to see what the discovery is. Natural language follows a slightly modified version of Zipf's Law / Power distribution, but from what I remember from college, so does randomly distributed writing (if you draw randomly from [a-z ] and split on spaces, you get a similar distribution) EDIT: From Chris Mannings book "Statistical Foundation of NLP" (MIT-1999) > As a final remark on Zipf’s law, we note that there is a debate on how surprising and interesting Zipf’s law and ‘power laws’ in general are as a description of natural phenomena. It has been argued that randomly generated text exhibits Zipf’s law (Li 1992). >To show this, we construct a generator that randomly characters from the 26 letters of the alphabet and the blank (that is, each of these 27 symbols has an equal chance of being generated next). Simplifying slightly, the probability of a word of length n being generated is the probability of generating a non-blank character n times and the blank after that. One can show that the words generated by such a generator obey a power law of the form Mandelbrot suggested. The key insights are that there are 26 times more words of length n 1 than length n, and (ii) that there is a constant ratio by which words of length are more frequent than words of length n + 1. These two opposing trends combine into the regularity of Mandelbrot’s law. See exercise 1.4. Where "Mandelbrot’s law" is the above "slightly modified version of Zipf's Law / Power distribution" I mentioned, and was published in 1954

wodenokoto 4y ago

I remember reading that article when it was on the front page of HN, and felt underwhelmed by the conclusion. Reading the quoted conclusions again, I almost fail to see what the discovery is. Natural language follows a slightly modified version of Zipf's Law / Power distribution, but from what I remember from college, so does randomly distributed writing (if you draw randomly from [a-z ] and split on spaces, you get a similar distribution) EDIT: From Chris Mannings book "Statistical Foundation of NLP" (MIT-1999) > As a final remark on Zipf’s law, we note that there is a debate on how surprising and interesting Zipf’s law and ‘power laws’ in general are as a description of natural phenomena. It has been argued that randomly generated text exhibits Zipf’s law (Li 1992). >To show this, we construct a generator that randomly characters from the 26 letters of the alphabet and the blank (that is, each of these 27 symbols has an equal chance of being generated next). Simplifying slightly, the probability of a word of length n being generated is the probability of generating a non-blank character n times and the blank after that. One can show that the words generated by such a generator obey a power law of the form Mandelbrot suggested. The key insights are that there are 26 times more words of length n 1 than length n, and (ii) that there is a constant ratio by which words of length are more frequent than words of length n + 1. These two opposing trends combine into the regularity of Mandelbrot’s law. See exercise 1.4. Where "Mandelbrot’s law" is the above "slightly modified version of Zipf's Law / Power distribution" I mentioned, and was published in 1954