3 ms·
I don't know that I've seen a paper that says this explicitly, I'm just saying it based on experience. You may have seen papers where they obscure parts of the
by version_five 4y ago
I don't know that I've seen a paper that says this explicitly, I'm just saying it based on experience. You may have seen papers where they obscure parts of the image and find that the classifier is basing its decision on the background, I think that's well known. Likewise I can't cite a paper of the top of my head, but the usual augmentations like random cropping, flipping, color jitter, etc are pretty well known to practitioners as way of preventing oveffitting. I see additional prediction targets as an extension of that, because they likewise incentivise the model to learn the "right" features because they make it harder to latch on to some spurious pattern in the data. And I've had success with it practically, which is why I made my original comment. Ymmv of course
- zone411 4y agoI haven't seen a paper on this either, but I think that if it does work, somebody should definitely write one about it because it seems quite consequential. I am not sure if the underlying reason would be regularization though, seems more like a case of indirect feature engineering.
- version_five 4y agoI'd argue that both are the same thing, but yes I agree - one way or another it's a way of bringing additional domain knowledge to the problem - and I suppose it entails the downside that if you do it wrong, you can reduce predictive power
- unixpickle 4y agoBefore writing this post, I asked chatgpt for examples of positive transfer from auxiliary losses in the literature. It pointed me to this paper: https://arxiv.org/abs/1705.07115 https://arxiv.org/abs/1705.07115