4 ms·
You're spot on - the calculations in the post are totally wrong. For instance, a Dice coefficient is itself the average of hundreds of thousands of observations
by jph00 7y ago
You're spot on - the calculations in the post are totally wrong. For instance, a Dice coefficient is itself the average of hundreds of thousands of observations (each pixel in an image), so you can't just treat the coefficient as a single data point. However you also can't just take a standard error, since the points are highly spatially correlated.
The post also doesn't draw on any actual reported experiences of either competitors or hosts of competitions. Generally in post competition wrap-ups there's a lot of information sharing and very significant development of understanding that comes about.
To take the imagenet example that's claimed in the post as being a likely example of over-fitting: each year's imagenet results led to very significant advances in our understanding of image classification, which are today very widely used in industry and research.
- AstralStorm 7y agoHave you actually tried to use one of those networks on a decent real dataset? They're making common category errors. ImageNet roulette is an interesting fun example of what happens if you feed an overfitted network with real images. Such as misclassifying children. :)
- bermanoid 7y agoImageNet Roulette deliberately uses a terrible categorization scheme that has long been acknowledged as so poor as to not admit meaningful results in order to make the highly political point that ML should never be applied to people. There's a reason most people scrub that whole piece of the taxonomy before training. Good Resnet models trained on ImageNet (the good parts, not just people) tend to result in state of the art results for almost every transfer-learning domain they're applied to.