6 ms·
In case anyone else was curious, Bedford's Law is apparently just a fancy name for the phenomenon where 1's appear much more frequently than 9's in any random s
by sillysaurusx 7y ago
In case anyone else was curious, Bedford's Law is apparently just a fancy name for the phenomenon where 1's appear much more frequently than 9's in any random sample of numbers. I think Bernie Madoff got caught faking the books due to the distributions not being shaped like Bedford's Law.
- eindiran 7y agoThat's correct, but I think its "Benford's Law" rather than "Bedford's Law". https://en.wikipedia.org/wiki/Benford%27s_law https://en.wikipedia.org/wiki/Benford%27s_law
- OnlineGladiator 7y agoWhat on Earth‽‽‽ How have I never heard of this before, and much more importantly - why is it true (I skimmed the wikipedia page and I didn't really see anything for 'why' but mostly for 'what')? So it takes the data and draws it on a log scale to visualize it, but doesn't explain why the data naturally transposes to a log scale (we are talking about random data sets in nature). If I think of it as a village with 1,000 people is more likely than a village with 2,000 people is more likely than a village with 3,000 people (and this continues in both directions), the only real conclusion I can come to is that nature prefers smaller numbers of things. But that wouldn't extend to something like accounting books (or at least the jump is not immediately obvious to me). EDIT: I guess I chose a bad example (although my point still is useful as a thought exercise), since this was used as an exception to the rule: > Benford's law is violated by the populations of all places with population at least 2500 from five US states according to the 1960 and 1970 censuses, where only 19% began with digit 1 but 20% began with digit 2, for the simple reason that the truncation at 2500 introduces statistical bias.
- ouid 7y agodid you type the first "‽" and then copy/paste the rest? basically the reason for this is that for any probability distribution over natural numbers that can reasonably be called the average probability distribution of natural numbers, I have to assign smaller probabilities to larger numbers. There's a lot of ways in which this is formally true, but it's certainly true practically. Among the k digit numbers, the ones starting with 2 are larger than the numbers starting with 1, so they have to be at most equally likely, but there are the same number of each, so the probability of picking any number starting with two has to be less than or equal to the probability of picking any number starting with 1.
- whoopdedo 7y ago> did you type the first "‽" and then copy/paste the rest? Typing that is easy with a composing keyboard. AltGr+!,? outputs ‽ (Or AltGr+?,! is ⸘) Apologies for the off-topic tangent.
- ouid 7y agoTo be fair, I started it.
- sillysaurusx 7y agoI wondered the same thing. Thank you for voicing the question. One-off tangents like that are surprisingly fun, and informative.
- MaulingMonkey 7y agoFor accounting, you have plenty of small things, and fewer large things. A lot more small transactions than large transactions (how frequently do you buy a meal or groceries? vs how frequently do you buy a car or house?) A lot more small companies (every independent bookstore, electrician, etc.) than large companies (how many >$1 billion market cap companies). A lot more small salaries (entry level workers) vs large salaries (CEOs.) A lot more small stock trades (minor portfolio adjustments) than large stock trades (liquidating your entire inventory as you enter/exit a market.)
- kirrent 7y agoIt's because the ranges of distributions you sample from don't necessarily line up nicely with our number systems. For example, if you're measuring the distance between you and nearby stars in light years does it seem likely that the distances fit in the range of 0-99.99 light years perfectly? No, it's far more likely to overflow into 100s or even 200s of light years. Hopefully that gives you an intuitive sense why Benford's law gives a highest probability for 1 as the leading number but 2 is the next most common and so on. You can also see the weaker application by the same argument to the following digits.
- learnstats2 7y agoI'd wager that Benford's law doesn't apply to distances to stars.
- pizza 7y agoSee figure 1b. :) https://digitalcommons.calpoly.edu/cgi/viewcontent.cgi?article=1087&context=rgp_rsr https://digitalcommons.calpoly.edu/cgi/viewcontent.cgi?artic...
- kirrent 7y agoIn fact, star distances follow Benford's law very well through multiple digits. https://arxiv.org/abs/1401.5794 https://arxiv.org/abs/1401.5794
- singingboyo 7y agoAmong other things, the first digit is easier to change (percentage-wise) when a number starts with a large first digit. StackExchange math has a decent analogy: https://math.stackexchange.com/questions/781/why-does-benfords-law-or-zipfs-law-hold https://math.stackexchange.com/questions/781/why-does-benfor...
- baddox 7y agoFor a village to grow from 1,000 people to 2,000 people, it has to double in size. But to grow from 9,000 people to 10,000 people, it only needs to grow by about 11%.
- OnlineGladiator 7y agoI need to think about it more, but I think this is the general direction my intuition is moving. Thank you!
- Someone 7y ago”So it takes the data and draws it on a log scale to visualize it, but doesn't explain why the data naturally transposes to a log scale (we are talking about random data sets in nature)” Random data sets in nature, measured in some arbitrary unit chosen by mankind. If one assumes expressing lengths in inches, feet, meters, parsecs, or any other arbitrary units will not change the distribution of initial digits, Benford’s law follows. For example, when converting a set of lengths measured in yards to one measured in feet, the numbers originally beginning with a ‘1’, i.e. in the half-open interval [1,2) will start with a ‘3’, ‘4’, or ‘5’ i.e. lie in the half-open interval [3,6) in the new table. That can’t be true if initial digits are uniformly distributed.
- SamReidHughes 7y agoAlso, if that isn't convincing, number base. Even if we carefully defined the units such that Planck's constant is 6.6e-34 in base 10, it would still follow Benford's law (especially because that's a constant having a big exponent) for other number bases.
- sillysaurusx 7y agoYour shock is warranted. It's actually a very deep question, called the Zipf mystery: https://www.youtube.com/watch?v=fCn8zs912OE https://www.youtube.com/watch?v=fCn8zs912OE No one really knows why it's true. It's also true across physical constants, not just artificial quantities made up by humans. The same type of distribution pops up in language, in nature, in numbers, popularity of opening chess moves, everywhere.
- saagarjha 7y agoZipf’s Law is related, but not quite the same thing: https://en.wikipedia.org/wiki/Zipf's_law https://en.wikipedia.org/wiki/Zipf's_law
- ColinWright 7y ago> No one really knows why it's true. Any distribution of numbers that is "scale free" follows Benford's Law. This is extremely well understood. The distribution of the areas of lakes, heights of trees, the proportion of the answers that start with the digit "1" should be the same no matter what units you use. You can measure heights in barleycorns, metres, inches, or cubits, the proportion of answers that start with the digit "1" should remain constant across those different units. Grind through the sums and Benford's Law pops out. And Benford's Law is not Zipf's Law. Zipf's Law is also well-understood when you look at coding theory, and how we would expect language (and mutable codes in general) to change so that more frequently used "atoms" are "shorter. So I don't really know what you're claiming here, but on the surface of it, with things I've studied, your comment seems mostly wrong.
- emmelaich 7y agoAt first I thought Zipf's had nothing to do with Benford's. But then .. https://terrytao.wordpress.com/2009/07/03/benfords-law-zipfs-law-and-the-pareto-distribution/ https://terrytao.wordpress.com/2009/07/03/benfords-law-zipfs...
- xelxebar 7y ago> No one really knows why it's true. This is what is so bizzare. We do know why it's true; the mystery is just pumped up in the press. I have only skimmed the paper, but apparently Belevitch [0] showed (in 1959!) that Zipf's law and Mandelbrot's Law [1] are essentially just first-order and second-order approximations, respectively, for a large class of probability distributions, so no wonder it is so ubiquitous. [0] Belevitch, V. (1959). On the statistical laws of linguistic distributions. Annales de la Societe Scientifique de Bruxelles, 73(3), 301–326. [1] https://en.wikipedia.org/wiki/Zipf%E2%80%93Mandelbrot_law https://en.wikipedia.org/wiki/Zipf%E2%80%93Mandelbrot_law
- learnstats2 7y agoWhy is it true? It relates to exponential growth. Start with any number and repeatedly add any percentage, e.g. 10%. Repeated multiplication will get you through the 2000s quicker than it got you through the 1000s.
- Yajirobe 7y agoWhat about what the other commenter mentioned - physical constants? There is no room for exponential growth there (or any sort of growth, for that matter).
- learnstats2 7y agoWhat is meant by physical constants and in what context? E.g. the boiling point of water starts with 1. That's not an aspect of Benford's law: it's intentionally set.
- kevinventullo 7y agoI think of it as less a statement about general data sets, and more a statement about base 10 representations. In binary, every number begins with 1!
- saagarjha 7y agoNot zero.
- kevinventullo 7y agoFair enough :)
- daveFNbuck 7y agoIt works if you assume that the units are arbitrary. Suppose instead of people you're measuring heights. If there's going to be a distribution of leading numbers in random data, it has to be the same in feet and in yards. To convert from yards to feet, you multiply by 3. Anything starting with 4 or 5 will always start with 1 after this multiplication, as will lots of things starting with 3 or 6. So starting with a 1 has to be as likely as starting with 3, 4, 5, 6. The same sort of thing has to work if you multiply by any number. Benford's law gives you the distribution where this works. The fact that this is scale-invariant means that it's not about preferring smaller numbers of things. It's just that the number of items in the range 10,000-19,999 is as large as the number of items in the range 0-9,999. Slightly bigger things also tend to start with ones. This all just assumes that the distribution of leading digits doesn't change if you scale your data, which I find intuitive enough to be satisfying.
- Someone 7y agoThe first digit is a 1 much more frequently, the second digit less so, the third even less so, etc. The more digits the numbers in the sample have, on average, the smaller the difference in frequency, taken over all digits, gets. So, in general, one cannot state that 1's appear much more frequently than 9's. Also, that’s not in any random sample of numbers; you need to sample from a distribution with a certain property. The distributions of many samples of real-world phenomena happen to have that property.
- jsweojtj 7y agoIt's not true for any random sample of numbers. Spin up a Jupyter notebook and paste the following: %matplotlib inline import matplotlib.pyplot as plt import numpy as np import seaborn as sns sns.set_context('poster', font_scale=1.3) n_values = 10000 lower_bound = 1 upper_bound = 100000 nums = np.array( [int(str(x)[0]) for x in np.random.randint(lower_bound, upper_bound, size=n_values)] ) fig, ax = plt.subplots(figsize=(12, 8)) sns.distplot(nums, kde=False, bins=list(range(1, 11)), ax=ax) ax.set_xticks(list(range(1, 11))) fig.tight_layout() And you'll see it's essentially a uniform distribution of leading digits.
- doubleunplussed 7y agoReal world random data often doesn't have a hard maximum. Try plotting a histogram of digits of -log(rand()). (I think this gives an exponential distribution with mean 1)
- jsweojtj 7y agoI'm glad that you agree that the original post was incorrect in stating that: > Bedford's Law is apparently just a fancy name for the phenomenon where 1's appear much more frequently than 9's in any random sample of numbers. because, as my example shows, it's false for the case of random numbers sampled uniformly across five orders of magnitude. You're also right that it's possible to construct a distribution (such as taking -log(rand())) where this behavior is observed.
- jagged-chisel 7y ago> ... an observation about the frequency distribution of leading digits in many real-life sets of numerical data. That's about leading digits only.
- whack 7y ago> in any random sample of numbers To be more precise: > Benford's law tends to apply most accurately to data that span several orders of magnitude. As a rule of thumb, the more orders of magnitude that the data evenly covers, the more accurately Benford's law applies. For instance, one can expect that Benford's law would apply to a list of numbers representing the populations of UK settlements. But if a "settlement" is defined as a village with population between 300 and 999, then Benford's law will not apply https://en.wikipedia.org/wiki/Benford%27s_law#Explanations https://en.wikipedia.org/wiki/Benford%27s_law#Explanations
- jsweojtj 7y agoEven this explanation is incorrect. Evenly sampled data across many orders of magnitude give an even distribution of leading numbers. See my comment for code here: https://news.ycombinator.com/item?id=21541264 https://news.ycombinator.com/item?id=21541264 There are many distributions which do lead to the distribution of leading digits that follow Benford's law, just not uniform. Nor "uniform over several orders of magnitude".
- deleted 7y ago[deleted]
- whack 7y ago> Thus, real-world distributions that span several orders of magnitude rather uniformly They mean uniformly on a log distribution. Ie, there are a uniform number of elements between 10^x and 10^(x+1), for many different values of x. This is best shown by the graph here: https://en.wikipedia.org/wiki/Benford%27s_law#Overview https://en.wikipedia.org/wiki/Benford%27s_law#Overview