3 ms·
I don’t understand this heuristic and I think it might be a bit garbled. Any idea what the author meant? How do you get 1000? > A good rule of thumb is that yo
by skybrian 3y ago
I don’t understand this heuristic and I think it might be a bit garbled. Any idea what the author meant? How do you get 1000?
> A good rule of thumb is that you’ll want ten data points per significant digit of your evaluation metric. So if you want to distinguish 91% accuracy from 90% accuracy, you’ll want to have at least 1000 data points annotated. You don’t want to be running experiments where your accuracy figure says a 1% improvement, but actually you went from 94/103 to 96/103.
- akprasad 3y agoMy guess is that this should be something like "If you have n significant digits in your evaluation metric, you should have at least 10^(n+1) data points."
- wrs 3y agoAvoiding the term “significant digits” completely: Distinguishing 91 vs 90 is a difference of 1 on a 0-100 scale. 100x10=1000. If you wanted to distinguish 91.0 vs 90.9, that’s 1 on a 0-1000 scale, so you’d want 10,000 points.