3 ms·

The fundamental problem with academic metrics is that if you're solving a _real_ problem (a rarity in academia, BTW), they are merely a proxy for how well you'v

by m0zg 7y ago

The fundamental problem with academic metrics is that if you're solving a _real_ problem (a rarity in academia, BTW), they are merely a proxy for how well you've solved the problem, and often not a good one. Case in point: say you're building a berry picking robot using computer vision. As a part of that you'll probably use an object detection system which lets your robot see the berries and know where they are. Commonly you will use a combination of losses to optimize the system, and a combination of mean average precision metrics to evaluate how good it is. But here's the issue: even the evaluation mAP (let alone loss) does not tell you how good the robot will be at picking berries. Moreover, there's no point of reference for you to tell if e.g. mAP of 80% is "good enough". And "goodness of the robot" while it can be defined as a real world metric (e.g. robot successfully picked 90% of ripe berries and only destroyed 0.5% of the plants), usually can't be formulated as a direct optimization objective. So you end up futzing with the metrics that are easier to work with, hoping and praying that your result is good in the end. If that doesn't sound scary enough, think of autonomous driving systems. :-)

XuMiao 7y ago
Most of the time, the surrogate metrics align with the unknown metrics when they are small, e.g. 80%. They can start to diverge at 90% for example. When I see people report a .5% improvement over a 98% accuracy and claim the statistical significance, I highly doubt about the real significance. The long tail is always longer than we thought. On the hand, there is no coincidence in the world. If some datapoints happen to be outliers that the metrics miscalculated, it might be that the problem isn't setup right. For example, missed some information in the data. In-person interview helps not just because people have a better judgement. It's because we can find out more details about that sample. The current AI/ML method lacks an active information seeking process. They don't have the freedom to explore the world beyond the defined state/reward space.

by m0zg 7y ago

XuMiao 7y ago

Most of the time, the surrogate metrics align with the unknown metrics when they are small, e.g. 80%. They can start to diverge at 90% for example. When I see people report a .5% improvement over a 98% accuracy and claim the statistical significance, I highly doubt about the real significance. The long tail is always longer than we thought. On the hand, there is no coincidence in the world. If some datapoints happen to be outliers that the metrics miscalculated, it might be that the problem isn't setup right. For example, missed some information in the data. In-person interview helps not just because people have a better judgement. It's because we can find out more details about that sample. The current AI/ML method lacks an active information seeking process. They don't have the freedom to explore the world beyond the defined state/reward space.