4 ms·

That's correct, but I think its "Benford's Law" rather than "Bedford's Law". https://en.wikipedia.org/wiki/Benford%27s_law https://en.wikipedia.org/wiki/Benfor

That's correct, but I think its "Benford's Law" rather than "Bedford's Law". https://en.wikipedia.org/wiki/Benford%27s_law https://en.wikipedia.org/wiki/Benford%27s_law

OnlineGladiator 7y ago
What on Earth‽‽‽ How have I never heard of this before, and much more importantly - why is it true (I skimmed the wikipedia page and I didn't really see anything for 'why' but mostly for 'what')? So it takes the data and draws it on a log scale to visualize it, but doesn't explain why the data naturally transposes to a log scale (we are talking about random data sets in nature). If I think of it as a village with 1,000 people is more likely than a village with 2,000 people is more likely than a village with 3,000 people (and this continues in both directions), the only real conclusion I can come to is that nature prefers smaller numbers of things. But that wouldn't extend to something like accounting books (or at least the jump is not immediately obvious to me). EDIT: I guess I chose a bad example (although my point still is useful as a thought exercise), since this was used as an exception to the rule: > Benford's law is violated by the populations of all places with population at least 2500 from five US states according to the 1960 and 1970 censuses, where only 19% began with digit 1 but 20% began with digit 2, for the simple reason that the truncation at 2500 introduces statistical bias.
- ouid 7y ago
  did you type the first "‽" and then copy/paste the rest? basically the reason for this is that for any probability distribution over natural numbers that can reasonably be called the average probability distribution of natural numbers, I have to assign smaller probabilities to larger numbers. There's a lot of ways in which this is formally true, but it's certainly true practically. Among the k digit numbers, the ones starting with 2 are larger than the numbers starting with 1, so they have to be at most equally likely, but there are the same number of each, so the probability of picking any number starting with two has to be less than or equal to the probability of picking any number starting with 1.
  - whoopdedo 7y ago
    > did you type the first "‽" and then copy/paste the rest? Typing that is easy with a composing keyboard. AltGr+!,? outputs ‽ (Or AltGr+?,! is ⸘) Apologies for the off-topic tangent.
    - ouid 7y ago
      To be fair, I started it.
      - sillysaurusx 7y ago
        I wondered the same thing. Thank you for voicing the question. One-off tangents like that are surprisingly fun, and informative.
- MaulingMonkey 7y ago
  For accounting, you have plenty of small things, and fewer large things. A lot more small transactions than large transactions (how frequently do you buy a meal or groceries? vs how frequently do you buy a car or house?) A lot more small companies (every independent bookstore, electrician, etc.) than large companies (how many >$1 billion market cap companies). A lot more small salaries (entry level workers) vs large salaries (CEOs.) A lot more small stock trades (minor portfolio adjustments) than large stock trades (liquidating your entire inventory as you enter/exit a market.)
- kirrent 7y ago
  It's because the ranges of distributions you sample from don't necessarily line up nicely with our number systems. For example, if you're measuring the distance between you and nearby stars in light years does it seem likely that the distances fit in the range of 0-99.99 light years perfectly? No, it's far more likely to overflow into 100s or even 200s of light years. Hopefully that gives you an intuitive sense why Benford's law gives a highest probability for 1 as the leading number but 2 is the next most common and so on. You can also see the weaker application by the same argument to the following digits.
  - learnstats2 7y ago
    I'd wager that Benford's law doesn't apply to distances to stars.
    - pizza 7y ago
      See figure 1b. :) https://digitalcommons.calpoly.edu/cgi/viewcontent.cgi?article=1087&context=rgp_rsr https://digitalcommons.calpoly.edu/cgi/viewcontent.cgi?artic...
    - kirrent 7y ago
      In fact, star distances follow Benford's law very well through multiple digits. https://arxiv.org/abs/1401.5794 https://arxiv.org/abs/1401.5794
- singingboyo 7y ago
  Among other things, the first digit is easier to change (percentage-wise) when a number starts with a large first digit. StackExchange math has a decent analogy: https://math.stackexchange.com/questions/781/why-does-benfords-law-or-zipfs-law-hold https://math.stackexchange.com/questions/781/why-does-benfor...
- baddox 7y ago
  For a village to grow from 1,000 people to 2,000 people, it has to double in size. But to grow from 9,000 people to 10,000 people, it only needs to grow by about 11%.
  - OnlineGladiator 7y ago
    I need to think about it more, but I think this is the general direction my intuition is moving. Thank you!
- Someone 7y ago
  ”So it takes the data and draws it on a log scale to visualize it, but doesn't explain why the data naturally transposes to a log scale (we are talking about random data sets in nature)” Random data sets in nature, measured in some arbitrary unit chosen by mankind. If one assumes expressing lengths in inches, feet, meters, parsecs, or any other arbitrary units will not change the distribution of initial digits, Benford’s law follows. For example, when converting a set of lengths measured in yards to one measured in feet, the numbers originally beginning with a ‘1’, i.e. in the half-open interval [1,2) will start with a ‘3’, ‘4’, or ‘5’ i.e. lie in the half-open interval [3,6) in the new table. That can’t be true if initial digits are uniformly distributed.
  - SamReidHughes 7y ago
    Also, if that isn't convincing, number base. Even if we carefully defined the units such that Planck's constant is 6.6e-34 in base 10, it would still follow Benford's law (especially because that's a constant having a big exponent) for other number bases.
- sillysaurusx 7y ago
  Your shock is warranted. It's actually a very deep question, called the Zipf mystery: https://www.youtube.com/watch?v=fCn8zs912OE https://www.youtube.com/watch?v=fCn8zs912OE No one really knows why it's true. It's also true across physical constants, not just artificial quantities made up by humans. The same type of distribution pops up in language, in nature, in numbers, popularity of opening chess moves, everywhere.
  - saagarjha 7y ago
    Zipf’s Law is related, but not quite the same thing: https://en.wikipedia.org/wiki/Zipf's_law https://en.wikipedia.org/wiki/Zipf's_law
  - ColinWright 7y ago
    > No one really knows why it's true. Any distribution of numbers that is "scale free" follows Benford's Law. This is extremely well understood. The distribution of the areas of lakes, heights of trees, the proportion of the answers that start with the digit "1" should be the same no matter what units you use. You can measure heights in barleycorns, metres, inches, or cubits, the proportion of answers that start with the digit "1" should remain constant across those different units. Grind through the sums and Benford's Law pops out. And Benford's Law is not Zipf's Law. Zipf's Law is also well-understood when you look at coding theory, and how we would expect language (and mutable codes in general) to change so that more frequently used "atoms" are "shorter. So I don't really know what you're claiming here, but on the surface of it, with things I've studied, your comment seems mostly wrong.
  - emmelaich 7y ago
    At first I thought Zipf's had nothing to do with Benford's. But then .. https://terrytao.wordpress.com/2009/07/03/benfords-law-zipfs-law-and-the-pareto-distribution/ https://terrytao.wordpress.com/2009/07/03/benfords-law-zipfs...
  - xelxebar 7y ago
    > No one really knows why it's true. This is what is so bizzare. We do know why it's true; the mystery is just pumped up in the press. I have only skimmed the paper, but apparently Belevitch [0] showed (in 1959!) that Zipf's law and Mandelbrot's Law [1] are essentially just first-order and second-order approximations, respectively, for a large class of probability distributions, so no wonder it is so ubiquitous. [0] Belevitch, V. (1959). On the statistical laws of linguistic distributions. Annales de la Societe Scientifique de Bruxelles, 73(3), 301–326. [1] https://en.wikipedia.org/wiki/Zipf%E2%80%93Mandelbrot_law https://en.wikipedia.org/wiki/Zipf%E2%80%93Mandelbrot_law
- learnstats2 7y ago
  Why is it true? It relates to exponential growth. Start with any number and repeatedly add any percentage, e.g. 10%. Repeated multiplication will get you through the 2000s quicker than it got you through the 1000s.
  - Yajirobe 7y ago
    What about what the other commenter mentioned - physical constants? There is no room for exponential growth there (or any sort of growth, for that matter).
    - learnstats2 7y ago
      What is meant by physical constants and in what context? E.g. the boiling point of water starts with 1. That's not an aspect of Benford's law: it's intentionally set.
- kevinventullo 7y ago
  I think of it as less a statement about general data sets, and more a statement about base 10 representations. In binary, every number begins with 1!
  - saagarjha 7y ago
    Not zero.
    - kevinventullo 7y ago
      Fair enough :)
- daveFNbuck 7y ago
  It works if you assume that the units are arbitrary. Suppose instead of people you're measuring heights. If there's going to be a distribution of leading numbers in random data, it has to be the same in feet and in yards. To convert from yards to feet, you multiply by 3. Anything starting with 4 or 5 will always start with 1 after this multiplication, as will lots of things starting with 3 or 6. So starting with a 1 has to be as likely as starting with 3, 4, 5, 6. The same sort of thing has to work if you multiply by any number. Benford's law gives you the distribution where this works. The fact that this is scale-invariant means that it's not about preferring smaller numbers of things. It's just that the number of items in the range 10,000-19,999 is as large as the number of items in the range 0-9,999. Slightly bigger things also tend to start with ones. This all just assumes that the distribution of leading digits doesn't change if you scale your data, which I find intuitive enough to be satisfying.

That's correct, but I think its "Benford's Law" rather than "Bedford's Law". https://en.wikipedia.org/wiki/Benford%27s_law https://en.wikipedia.org/wiki/Benford%27s_law

OnlineGladiator 7y ago

What on Earth‽‽‽ How have I never heard of this before, and much more importantly - why is it true (I skimmed the wikipedia page and I didn't really see anything for 'why' but mostly for 'what')? So it takes the data and draws it on a log scale to visualize it, but doesn't explain why the data naturally transposes to a log scale (we are talking about random data sets in nature). If I think of it as a village with 1,000 people is more likely than a village with 2,000 people is more likely than a village with 3,000 people (and this continues in both directions), the only real conclusion I can come to is that nature prefers smaller numbers of things. But that wouldn't extend to something like accounting books (or at least the jump is not immediately obvious to me). EDIT: I guess I chose a bad example (although my point still is useful as a thought exercise), since this was used as an exception to the rule: > Benford's law is violated by the populations of all places with population at least 2500 from five US states according to the 1960 and 1970 censuses, where only 19% began with digit 1 but 20% began with digit 2, for the simple reason that the truncation at 2500 introduces statistical bias.

ouid 7y ago
did you type the first "‽" and then copy/paste the rest? basically the reason for this is that for any probability distribution over natural numbers that can reasonably be called the average probability distribution of natural numbers, I have to assign smaller probabilities to larger numbers. There's a lot of ways in which this is formally true, but it's certainly true practically. Among the k digit numbers, the ones starting with 2 are larger than the numbers starting with 1, so they have to be at most equally likely, but there are the same number of each, so the probability of picking any number starting with two has to be less than or equal to the probability of picking any number starting with 1.
- whoopdedo 7y ago
  > did you type the first "‽" and then copy/paste the rest? Typing that is easy with a composing keyboard. AltGr+!,? outputs ‽ (Or AltGr+?,! is ⸘) Apologies for the off-topic tangent.
  - ouid 7y ago
    To be fair, I started it.
    - sillysaurusx 7y ago
      I wondered the same thing. Thank you for voicing the question. One-off tangents like that are surprisingly fun, and informative.
MaulingMonkey 7y ago
For accounting, you have plenty of small things, and fewer large things. A lot more small transactions than large transactions (how frequently do you buy a meal or groceries? vs how frequently do you buy a car or house?) A lot more small companies (every independent bookstore, electrician, etc.) than large companies (how many >$1 billion market cap companies). A lot more small salaries (entry level workers) vs large salaries (CEOs.) A lot more small stock trades (minor portfolio adjustments) than large stock trades (liquidating your entire inventory as you enter/exit a market.)
kirrent 7y ago
It's because the ranges of distributions you sample from don't necessarily line up nicely with our number systems. For example, if you're measuring the distance between you and nearby stars in light years does it seem likely that the distances fit in the range of 0-99.99 light years perfectly? No, it's far more likely to overflow into 100s or even 200s of light years. Hopefully that gives you an intuitive sense why Benford's law gives a highest probability for 1 as the leading number but 2 is the next most common and so on. You can also see the weaker application by the same argument to the following digits.
- learnstats2 7y ago
  I'd wager that Benford's law doesn't apply to distances to stars.
  - pizza 7y ago
    See figure 1b. :) https://digitalcommons.calpoly.edu/cgi/viewcontent.cgi?article=1087&context=rgp_rsr https://digitalcommons.calpoly.edu/cgi/viewcontent.cgi?artic...
  - kirrent 7y ago
    In fact, star distances follow Benford's law very well through multiple digits. https://arxiv.org/abs/1401.5794 https://arxiv.org/abs/1401.5794
singingboyo 7y ago
Among other things, the first digit is easier to change (percentage-wise) when a number starts with a large first digit. StackExchange math has a decent analogy: https://math.stackexchange.com/questions/781/why-does-benfords-law-or-zipfs-law-hold https://math.stackexchange.com/questions/781/why-does-benfor...
baddox 7y ago
For a village to grow from 1,000 people to 2,000 people, it has to double in size. But to grow from 9,000 people to 10,000 people, it only needs to grow by about 11%.
- OnlineGladiator 7y ago
  I need to think about it more, but I think this is the general direction my intuition is moving. Thank you!
Someone 7y ago
”So it takes the data and draws it on a log scale to visualize it, but doesn't explain why the data naturally transposes to a log scale (we are talking about random data sets in nature)” Random data sets in nature, measured in some arbitrary unit chosen by mankind. If one assumes expressing lengths in inches, feet, meters, parsecs, or any other arbitrary units will not change the distribution of initial digits, Benford’s law follows. For example, when converting a set of lengths measured in yards to one measured in feet, the numbers originally beginning with a ‘1’, i.e. in the half-open interval [1,2) will start with a ‘3’, ‘4’, or ‘5’ i.e. lie in the half-open interval [3,6) in the new table. That can’t be true if initial digits are uniformly distributed.
- SamReidHughes 7y ago
  Also, if that isn't convincing, number base. Even if we carefully defined the units such that Planck's constant is 6.6e-34 in base 10, it would still follow Benford's law (especially because that's a constant having a big exponent) for other number bases.
sillysaurusx 7y ago
Your shock is warranted. It's actually a very deep question, called the Zipf mystery: https://www.youtube.com/watch?v=fCn8zs912OE https://www.youtube.com/watch?v=fCn8zs912OE No one really knows why it's true. It's also true across physical constants, not just artificial quantities made up by humans. The same type of distribution pops up in language, in nature, in numbers, popularity of opening chess moves, everywhere.
- saagarjha 7y ago
  Zipf’s Law is related, but not quite the same thing: https://en.wikipedia.org/wiki/Zipf's_law https://en.wikipedia.org/wiki/Zipf's_law
- ColinWright 7y ago
  > No one really knows why it's true. Any distribution of numbers that is "scale free" follows Benford's Law. This is extremely well understood. The distribution of the areas of lakes, heights of trees, the proportion of the answers that start with the digit "1" should be the same no matter what units you use. You can measure heights in barleycorns, metres, inches, or cubits, the proportion of answers that start with the digit "1" should remain constant across those different units. Grind through the sums and Benford's Law pops out. And Benford's Law is not Zipf's Law. Zipf's Law is also well-understood when you look at coding theory, and how we would expect language (and mutable codes in general) to change so that more frequently used "atoms" are "shorter. So I don't really know what you're claiming here, but on the surface of it, with things I've studied, your comment seems mostly wrong.
- emmelaich 7y ago
  At first I thought Zipf's had nothing to do with Benford's. But then .. https://terrytao.wordpress.com/2009/07/03/benfords-law-zipfs-law-and-the-pareto-distribution/ https://terrytao.wordpress.com/2009/07/03/benfords-law-zipfs...
- xelxebar 7y ago
  > No one really knows why it's true. This is what is so bizzare. We do know why it's true; the mystery is just pumped up in the press. I have only skimmed the paper, but apparently Belevitch [0] showed (in 1959!) that Zipf's law and Mandelbrot's Law [1] are essentially just first-order and second-order approximations, respectively, for a large class of probability distributions, so no wonder it is so ubiquitous. [0] Belevitch, V. (1959). On the statistical laws of linguistic distributions. Annales de la Societe Scientifique de Bruxelles, 73(3), 301–326. [1] https://en.wikipedia.org/wiki/Zipf%E2%80%93Mandelbrot_law https://en.wikipedia.org/wiki/Zipf%E2%80%93Mandelbrot_law
learnstats2 7y ago
Why is it true? It relates to exponential growth. Start with any number and repeatedly add any percentage, e.g. 10%. Repeated multiplication will get you through the 2000s quicker than it got you through the 1000s.
- Yajirobe 7y ago
  What about what the other commenter mentioned - physical constants? There is no room for exponential growth there (or any sort of growth, for that matter).
  - learnstats2 7y ago
    What is meant by physical constants and in what context? E.g. the boiling point of water starts with 1. That's not an aspect of Benford's law: it's intentionally set.
kevinventullo 7y ago
I think of it as less a statement about general data sets, and more a statement about base 10 representations. In binary, every number begins with 1!
- saagarjha 7y ago
  Not zero.
  - kevinventullo 7y ago
    Fair enough :)
daveFNbuck 7y ago
It works if you assume that the units are arbitrary. Suppose instead of people you're measuring heights. If there's going to be a distribution of leading numbers in random data, it has to be the same in feet and in yards. To convert from yards to feet, you multiply by 3. Anything starting with 4 or 5 will always start with 1 after this multiplication, as will lots of things starting with 3 or 6. So starting with a 1 has to be as likely as starting with 3, 4, 5, 6. The same sort of thing has to work if you multiply by any number. Benford's law gives you the distribution where this works. The fact that this is scale-invariant means that it's not about preferring smaller numbers of things. It's just that the number of items in the range 10,000-19,999 is as large as the number of items in the range 0-9,999. Slightly bigger things also tend to start with ones. This all just assumes that the distribution of leading digits doesn't change if you scale your data, which I find intuitive enough to be satisfying.