3 ms·

It can be fooled, and genuine data can sometimes not follow Benford’s law for a variety of reasons, but it’s a good tool in the toolbox for quick examinations.

by fwlr 4y ago

It can be fooled, and genuine data can sometimes not follow Benford’s law for a variety of reasons, but it’s a good tool in the toolbox for quick examinations. There is another technique (which I can’t find a write-up of, unfortunately) that’s similarly cheap to apply (although limited to cruder fraud than Benford’s law testing can detect): you can calculate backwards from a given percentage to see if it’s actually possible to get that percentage with the number of data points, test subjects, etc. If you have 71 observations and a reported rate of 57%, the test immediately tells you that’s been faked: it’s possible to get 56.3% with 40/71 or 57.7% with 41/71. Neither rounds to 57%. Even tests this crude will detect fraudulent papers, gut level of sophistication of fraud is surprisingly low.

rscho 4y ago
It won't. Many researchers can't round correctly. Because they're doing it by hand. On Excel.
- version_five 4y ago
  Yeah, incorrect (or more likely misunderstood) rounding is not a smoking gun, and is a pretty silly thing to try and use to "catch" somebody. I've actually regularly seen administrators with nothing to add to a discussion call numbers into question because they can't understand how rounded values may not add up exactly to a column total. For that reason, it's relatively common practice in consulting reports to fudge the numbers so the rounded values do add to the colum totals (there are a few other fudges like this to make numbers easier to understand). It's only actually dishonest or material if you then try and explain your conclusions based on the 0.01 you tweaked.
  - IanCal 4y ago
    The point more generally is figures that can't be reached with the denominator. Rounding errors can only account for some of that, but there's plenty of cases where it can't.
- fwlr 4y ago
  I wish I could have found the writeup, it did a much better job explaining it than I did. Including the rounding thing obfuscated things; they had cases where the paper rounded to one or even two decimal points and that number wasn’t achievable with the denominators they had. To re-work my original example, they had real cases where 71 observations would have reported percentages of 57.1%, which couldn’t be achieved with any kind of rounding.

by fwlr 4y ago

rscho 4y ago

It won't. Many researchers can't round correctly. Because they're doing it by hand. On Excel.

version_five 4y ago
Yeah, incorrect (or more likely misunderstood) rounding is not a smoking gun, and is a pretty silly thing to try and use to "catch" somebody. I've actually regularly seen administrators with nothing to add to a discussion call numbers into question because they can't understand how rounded values may not add up exactly to a column total. For that reason, it's relatively common practice in consulting reports to fudge the numbers so the rounded values do add to the colum totals (there are a few other fudges like this to make numbers easier to understand). It's only actually dishonest or material if you then try and explain your conclusions based on the 0.01 you tweaked.
- IanCal 4y ago
  The point more generally is figures that can't be reached with the denominator. Rounding errors can only account for some of that, but there's plenty of cases where it can't.
fwlr 4y ago
I wish I could have found the writeup, it did a much better job explaining it than I did. Including the rounding thing obfuscated things; they had cases where the paper rounded to one or even two decimal points and that number wasn’t achievable with the denominators they had. To re-work my original example, they had real cases where 71 observations would have reported percentages of 57.1%, which couldn’t be achieved with any kind of rounding.