11 ms·
Zipf’s Law Arises Naturally When There Are Underlying, Unobserved Variables
- generj 10y agoWarning: I'm not a statistician, just an economist with some training in it. My quick understanding of this paper is that lurking variables can cause Zipf's law. If enough variables are controlled for, in some domains Zipf's Law goes away? For example, they included parts of speech for natural language and for some parts of speech Zipf's Law wasn't existent but for others it was. This makes intuitive sense to me. They also find a way to estimate how much of a Zipf's Law effect is caused by a particular variable. Seems like a nice test to run in Stata or R.
- kem 10y agoYour questions were similar to mine. I perused the article and hope to read it very carefully later today, but I wondered how often you wouldn't expect unobserved variables to be playing some role, especially in observational data. It's an interesting paper, but when I think of rigorous evaluation of its ideas I get a little confused. I'm probably missing something though.
- alok-g 10y agoI would appreciate a short and intuitive explanation of it. :-) Thanks.
- chrismealy 10y agoDo all probability distributions have known processes that generate them? Like how you can make a normal distribution with a Galton box? Is there something like that with power law distributions? It seems like the deal with power laws is that the events aren't independent (an event becomes more likely the more it occurs, or the rich get richer).
- antognini 10y agoMy background is in astronomy, where power laws are ubiquitous. (There's a joke that when you get some data, you fit it to a power law. If that's not a good enough fit, you fit it to a broken power law.) The reason they're so common is that a power law is generally the distribution you get anytime you have a scale-invariant process.
- klodolph 10y agoA Galton box generates a binomial distribution. Most probability distributions are designed to model some kind of process. The time between clicks of a geiger counter is exponential, the number of clicks in a second is poisson. The amount of time your geiger counter lasts before it breaks and you have to buy a new one is weibull. That's the whole reason why we invented probability distributions in the first place: to model processes.
- dekhn 10y agoThe Galton box generates a binomial distribution which when taken to the limit of n->infinity, produces a normal distribution.
- klodolph 10y agoThis is incorrect, you need to take a limit as n->infinity while dividing by a factor of √n, otherwise the limit does not exist. But Galton boxes in practice are not very large, so they aren't very good approximations of normal distributions.
- AlexCoventry 10y agoWhat do you mean by all probability distributions? Any measurable function on a measurable space with Lebesgue integral 1 can be considered a probability distribution, so the answer to the technical interpretation of your question is no.
- thomasahle 10y ago
- argonaut 10y agoTime for a funny quote: "Power laws... have been termed 'the signature of human activity'... They are certainly the product of one particular kind of human activity: looking for power laws." http://people.seas.harvard.edu/~babis/T797003/Readings/fabrikant02powerlaw.pdf http://people.seas.harvard.edu/~babis/T797003/Readings/fabri...
- deleted 10y ago[deleted]
- trhway 10y agowhich is the same as Pareto distribution which does seem "natural" and "intuitively easy explainable" ... at least to me.
- wodenokoto 10y agoI've recently come across a word frequency set of Chinese subtitles [0] and the word distribution doesn't seem to follow Zipf's Law / The power distribution, which seems counter to all my textbooks on statistical linguistics. This is not discussed in the associated paper either, though the paper concludes that the distribution is a good representation of common Chinese words (i.e., it isn't skewed). Any clues to why that is? [0] http://crr.ugent.be/programs-data/subtitle-frequencies/subtlex-ch http://crr.ugent.be/programs-data/subtitle-frequencies/subtl...
- rahimnathwani 10y ago"though the paper concludes that the distribution is a good representation of common Chinese words" I don't know why the word distribution doesn't follow a power law, but I can tell you that it isn't a good representation of common Chinese words. I downloaded the data set a while back, intending to use it for improving my vocabulary. I found some words (like the Chinese word for 'vampire') had too high a frequency. The lists are based on American TV shows' Chinese fansubs. I guess vampires and zombies are common themes.
- wodenokoto 10y agoI completely agree that the lists looks odd in many regards, but they test it on humans, and find correlations with frequency in the list and how quickly native speakers can recognise the word. I don't consider myself an expert in word-frequencies, but the article makes a convincing argument that these frequencies are representative of natural Chinese and a better or just as good measure as their baseline corpus.
- mhneu 10y agoSimilar idea, applied to different data. http://journals.aps.org/prl/abstract/10.1103/PhysRevLett.113.068102 http://journals.aps.org/prl/abstract/10.1103/PhysRevLett.113... (Can also see the preprint at arxiv) The joint probability distribution of states of many degrees of freedom in biological systems, such as firing patterns in neural networks or antibody sequence compositions, often follows Zipf’s law, where a power law is observed on a rank-frequency plot. This behavior has been shown to imply that these systems reside near a unique critical point where the extensive parts of the entropy and energy are exactly equal. Here, we show analytically, and via numerical simulations, that Zipf-like probability distributions arise naturally if there is a fluctuating unobserved variable (or variables) that affects the system, such as a common input stimulus that causes individual neurons to fire at time-varying rates. In statistics and machine learning, these are called latent-variable or mixture models. We show that Zipf’s law arises generically for large systems, without fine-tuning parameters to a point. Our work gives insight into the ubiquity of Zipf’s law in a wide range of systems.
- laurencea 10y agoAuthor here, happy to answer questions!
- ikeboy 10y agoWhy did it take two years from submission to publication: is that normal?
- canjobear 10y agoThere are many ways to derive Zipf's Law. Does yours make any novel predictions about systems that exhivit this behavior?
- CephalopodMD 10y agoIf you read the introduction, it does. In particular, it explains how normalizing for parts of speech (all nouns, verbs, etc) zipf's law does not emerge. This method predicts that phenomenon as parts of speech being a latent variable, which they do not believe other models can do.
- imagist 10y agoWhat does "rank" mean in this context?
- deleted 10y ago[deleted]
- jonquark 10y agoNumber in the list. So the first item in the list occurs roughly twice as often as the second item and roughly 9 times as often as the ninth item. Disclaimer: I'm not an expert, I looked at the Wikipedia page in response to this story. https://en.wikipedia.org/wiki/Zipf's_law https://en.wikipedia.org/wiki/Zipf's_law
- webmaven 10y agoThe Ordinal of the position in the sorted list of observed values.
- imagist 10y agoSo basically if you order by frequency, order correlates with frequency? Isn't that just two ways of saying the same thing? Isn't it obvious? I don't get why we would need a law for that so I feel like I'm missing some detail.
- jjaredsimpson 10y agoNot just correlates. rank * frequency = constant
- imagist 10y agoAh, that is the missing piece!
- deleted 10y ago[deleted]
- johnlbevan2 10y ago"when observations are ranked from most to least frequent, the frequency of an observation is inversely proportional to its rank" Is this just a complex way of saying "when you put items in ascending order, that order's the inverse of the same items listed in descending order", or have I missed something?
- secretasiandan 10y agoFrom wikipedia: https://en.wikipedia.org/wiki/Zipf's_law https://en.wikipedia.org/wiki/Zipf's_law Thus the most frequent word will occur approximately twice as often as the second most frequent word, three times as often as the third most frequent word, etc.: the rank-frequency distribution is an inverse relation.
- johnlbevan2 10y agoAh; I'd interpreted inversely proportional to mean "y = -mx" rather than "y = m / x", forgetting my GCSE maths terms :S. Thanks for the pointer.
- jjaredsimpson 10y agothe product of rank and frequency is constant. equivalently a plot of rank v frequency is a hyperbola, which implies a log-log plot is linear.
- dmux 10y agoVSauce has a Youtube video that discusses Zipf's: https://www.youtube.com/watch?v=fCn8zs912OE https://www.youtube.com/watch?v=fCn8zs912OE