7 ms·

Look out for No. 1

102 points by yread 15y ago

pigbucket 15y ago
Good to be aware of this if you plan to falsify your tax returns, according to a '98 NYT article (from which this post takes its neat title) http://www.nytimes.com/1998/08/04/science/following-benford-s-law-or-looking-out-for-no-1.html?pagewanted=all&src=pm http://www.nytimes.com/1998/08/04/science/following-benford-...
amirmc 15y ago
Previous discussion of Benford's Law on HN http://news.ycombinator.com/item?id=2701342 http://news.ycombinator.com/item?id=2701342
mdda 15y ago
"Nobody seems sure why so much data has the Benford distribution. " : Have a look at http://terrytao.wordpress.com/2009/07/03/benfords-law-zipfs-law-and-the-pareto-distribution/ http://terrytao.wordpress.com/2009/07/03/benfords-law-zipfs-... (cited in the previous discussion on HN) for an excellent exposition of the theory, and an explanation of the 'power' of the Benford Law.
- bermanoid 15y ago
  That Terry Tao article is excellent, as usual (well, at least for math dorks - what's unusual about this one is that you don't need a very high level of dorkery to grok it 100%) and the "executive summary" reason that Benford's law applies so broadly is easy to find: "More generally, it is not hard to show that if X obeys the continuous Benford’s law, and one multiplies X by some positive multiplier Y which is independent of the first digit of X (and, a fortiori, is independent of the fractional part of log_10(X)), one obtains another quantity X' = XY which also obeys the continuous Benford’s law." In other words, multiplicative combinations of (independent) quantities will inherit Benford's law as long as any one of the quantities obeys it on its own, so it just takes one sub-factor that grows exponentially in order for an entire distribution to follow the law.
- ohashi 15y ago
  That line especially bothered me, it makes it seem like it's some divine or magical rule. If that were the case, I would hate for it to be used as if it were infallible with no actual understanding.
chailatte 15y ago
I don't think the issue at hand is that fraud detections are unsophisticated. The issue at hand is that the fraud detections are in cahoot with the fraud themselves. For example: Enron, Worldcom and Accenture (past) Greece, Italy, Spain and Goldman Sachs, Fitch, Moody, S&P (present) Chinese local banks, Chinese companies and SEC (future) The problem with hackers is that they always try to find mathematical solutions to moral problems. What the world needs right now is a moral solution. A beheading for those that have fouled. Unfortunately, the hackers, the workers - ones that actually have the power to demand changes - are willingly allowing themselves to be exploited by those that lie, and not demanding any moral judgements. They're not interested in changing the politics, they claim. They just want to keep producing interesting things, they exclaim. They don't realize they're the ones keeping the lie going. They're the ones propping up this decrepit shell of a society. Until one day the shell collapses on their kids, anyways.
- william42 15y ago
  The problem is that revolutions tend to be more harmful than their predecessors: see China in the 60s, for example.
  - meric 15y ago
    I think China in the 10's is a good counterexample.
  - 0x12 15y ago
    That's definitely not a given. Revolutions are a time of instability and the outcome can be worse or better. It mostly depends on what precipitated the revolution in the first place, how educated the people in the country where the revolution takes place are and how involved they plan on being once the violence dies down. A revolution is a time of transition and instability without any guarantees towards the outcome.
- ntoshev 15y ago
  What is a moral solution, can you give a historical example?
borgar 15y ago
No link in the article so here is the report cited: http://onlinelibrary.wiley.com/doi/10.1111/j.1468-0475.2011.00542.x/full http://onlinelibrary.wiley.com/doi/10.1111/j.1468-0475.2011....
alecco 15y ago
Last month Khan and Vi Hart made some very interesting videos on Benford's Law: http://www.youtube.com/watch?v=6KmeGpjeLZ0 http://www.youtube.com/watch?v=6KmeGpjeLZ0 http://www.youtube.com/watch?v=SZUDoEdjTzg http://www.youtube.com/watch?v=SZUDoEdjTzg
RockofStrength 15y ago
I'm impressed that Madoff was smart enough to mimic Benford distribution in his monthly returns (that's what the article led me to infer). I actually made an r/math post about Benford's Law that I thought was pretty interesting, although it didn't get much attention or many comments (admittedly bad signs). Any peer review (or su-peer-ior review) is appreciated. Here's a snippet: "...generation of the geometric layout for Benford first digits is a similar process to creating a Sierpinski Triangle by drawing a triangle, and randomly plotting midpoint dots between vertices and sides recursively. In the Benford case the plotted dots are instead individual numbers, which share a geometric connection (e.g. two galaxies colliding will on average be a doubling, so galaxy size should possess Benford distribution)." http://www.reddit.com/r/math/comments/gkj2i/benfords_law_explained_through_a_tree_of_first/ http://www.reddit.com/r/math/comments/gkj2i/benfords_law_exp...

102 points by yread 15y ago

Good to be aware of this if you plan to falsify your tax returns, according to a '98 NYT article (from which this post takes its neat title) http://www.nytimes.com/1998/08/04/science/following-benford-s-law-or-looking-out-for-no-1.html?pagewanted=all&src=pm http://www.nytimes.com/1998/08/04/science/following-benford-...

amirmc 15y ago

Previous discussion of Benford's Law on HN http://news.ycombinator.com/item?id=2701342 http://news.ycombinator.com/item?id=2701342

mdda 15y ago

"Nobody seems sure why so much data has the Benford distribution. " : Have a look at http://terrytao.wordpress.com/2009/07/03/benfords-law-zipfs-law-and-the-pareto-distribution/ http://terrytao.wordpress.com/2009/07/03/benfords-law-zipfs-... (cited in the previous discussion on HN) for an excellent exposition of the theory, and an explanation of the 'power' of the Benford Law.

bermanoid 15y ago
That Terry Tao article is excellent, as usual (well, at least for math dorks - what's unusual about this one is that you don't need a very high level of dorkery to grok it 100%) and the "executive summary" reason that Benford's law applies so broadly is easy to find: "More generally, it is not hard to show that if X obeys the continuous Benford’s law, and one multiplies X by some positive multiplier Y which is independent of the first digit of X (and, a fortiori, is independent of the fractional part of log_10(X)), one obtains another quantity X' = XY which also obeys the continuous Benford’s law." In other words, multiplicative combinations of (independent) quantities will inherit Benford's law as long as any one of the quantities obeys it on its own, so it just takes one sub-factor that grows exponentially in order for an entire distribution to follow the law.
ohashi 15y ago
That line especially bothered me, it makes it seem like it's some divine or magical rule. If that were the case, I would hate for it to be used as if it were infallible with no actual understanding.

chailatte 15y ago

I don't think the issue at hand is that fraud detections are unsophisticated. The issue at hand is that the fraud detections are in cahoot with the fraud themselves. For example: Enron, Worldcom and Accenture (past) Greece, Italy, Spain and Goldman Sachs, Fitch, Moody, S&P (present) Chinese local banks, Chinese companies and SEC (future) The problem with hackers is that they always try to find mathematical solutions to moral problems. What the world needs right now is a moral solution. A beheading for those that have fouled. Unfortunately, the hackers, the workers - ones that actually have the power to demand changes - are willingly allowing themselves to be exploited by those that lie, and not demanding any moral judgements. They're not interested in changing the politics, they claim. They just want to keep producing interesting things, they exclaim. They don't realize they're the ones keeping the lie going. They're the ones propping up this decrepit shell of a society. Until one day the shell collapses on their kids, anyways.

william42 15y ago
The problem is that revolutions tend to be more harmful than their predecessors: see China in the 60s, for example.
- meric 15y ago
  I think China in the 10's is a good counterexample.
- 0x12 15y ago
  That's definitely not a given. Revolutions are a time of instability and the outcome can be worse or better. It mostly depends on what precipitated the revolution in the first place, how educated the people in the country where the revolution takes place are and how involved they plan on being once the violence dies down. A revolution is a time of transition and instability without any guarantees towards the outcome.
ntoshev 15y ago
What is a moral solution, can you give a historical example?

borgar 15y ago

No link in the article so here is the report cited: http://onlinelibrary.wiley.com/doi/10.1111/j.1468-0475.2011.00542.x/full http://onlinelibrary.wiley.com/doi/10.1111/j.1468-0475.2011....

alecco 15y ago

Last month Khan and Vi Hart made some very interesting videos on Benford's Law: http://www.youtube.com/watch?v=6KmeGpjeLZ0 http://www.youtube.com/watch?v=6KmeGpjeLZ0 http://www.youtube.com/watch?v=SZUDoEdjTzg http://www.youtube.com/watch?v=SZUDoEdjTzg

RockofStrength 15y ago

I'm impressed that Madoff was smart enough to mimic Benford distribution in his monthly returns (that's what the article led me to infer). I actually made an r/math post about Benford's Law that I thought was pretty interesting, although it didn't get much attention or many comments (admittedly bad signs). Any peer review (or su-peer-ior review) is appreciated. Here's a snippet: "...generation of the geometric layout for Benford first digits is a similar process to creating a Sierpinski Triangle by drawing a triangle, and randomly plotting midpoint dots between vertices and sides recursively. In the Benford case the plotted dots are instead individual numbers, which share a geometric connection (e.g. two galaxies colliding will on average be a doubling, so galaxy size should possess Benford distribution)." http://www.reddit.com/r/math/comments/gkj2i/benfords_law_explained_through_a_tree_of_first/ http://www.reddit.com/r/math/comments/gkj2i/benfords_law_exp...