4 ms·
I have mixed feelings. On the one hand, yes, people working with data should be more thoughtful about how many digits they report. But I’m not a big fan of the
by CrazyStat 3y ago
I have mixed feelings. On the one hand, yes, people working with data should be more thoughtful about how many digits they report. But I’m not a big fan of the proposed solution (report everything to a small fixed number of significant digits, like two).
Massive computer experiments are common and (often) cheap. If I’m comparing two classification models on a large dataset a difference between 97% and 96.8% accuracy might well be both statistically and practically significant.
An error I’ve seen professional statisticians make is asserting that averaging samples (or performing more complex statistical analyses like regression) justifies increasing the number of significant digits. For example, if I have a hundred observations measured with two significant digits of precision, the mean of those observations is asserted to be accurate to three significant digits, the standard deviation of the mean being 1/10th that of each observation. This is a natural conclusion if you treat the significant digits as accounting only for independent random errors, but in fact significant digits are also used to account for systematic errors in measuring equipment. A scale that measures down to 0.1g, for example, may be miscalibrated by 0.03g. This makes reporting the mean of 100 samples down to 0.01g obviously nonsensical.
In short, how many digits to report deserves more thought than either most data scientists or the author give it.
- YetAnotherNick 3y ago> 97% and 96.8% accuracy might well be both statistically and practically significant. I don't think author ever objected to this.
- CrazyStat 3y ago> Which answer do you give? Whatever your software tells you (e.g., 87.14234%) or a number made of a small and fixed number of significant digits (e.g., 87%). > The latter is the right answer in almost all instances. and > Most human beings are happy with a 1% error margin. While he doesn't say it as a rule outright, the author repeatedly uses a "small and fixed" two significant digits. He later says this: > You must choose the number of significant digits deliberately. which I agree with, but is at odds with the "small and fixed" dictum with which he leads off the post.
- YetAnotherNick 3y agoFixed doesn't mean 2. If it is statstically and practically significant. It could be 3 or 4.
- CrazyStat 3y agoIf you're adjusting the number of significant figures you report based on what is practically and statistically significant then it's no longer fixed.
- YetAnotherNick 3y agoIt doesn't need to be same for all the task, just fixed for a particular task. The creators of benchmark or the community could come up with digits of significance that should be the standard, which is based on expected error range. For imagenet it could be 4, for ARC it could be 2.