6 ms·
The normal way to combat this is to have a training set and a validation set, use the validation images in forward pass to measure the performance of the networ
by dheera 5y ago
The normal way to combat this is to have a training set and a validation set, use the validation images in forward pass to measure the performance of the network as it is being trained, but NOT use images from the validation set in updating the training weights.
Basically, evaluate the performance of the network on the validation set, but train it on the training set, and adjust your network structure and hyperparameters accordingly. Networks that "memorize images" will perform poorly.
- enkid 5y agoHow does a large company with million if images make sure there aren't very similar copies in not the training set and validation set? Seems like you could be accidentally over training if there's a lot of copies of very similar images. (For example, there's going to be a lot more pictures of the Taco Bell Chihuahua compared to other Chihuahuas, and possibly some of those pictures have enough similarity to make over fitting possible.
- IfOnlyYouKnew 5y agoThat’s fine… Randomly similar photos will occur in the real world just as much, so the good results for these will carry over.
- dheera 5y agoFirst of all having millions of images is of itself a good way to combat overfitting. But yes, you touch upon a very important point, that is the dataset must be sufficiently diverse. To be fair, humans have this problem as well; if I showed you an octagonal red sign that had the words "GO" inscribed in the middle you may still mistake it for a stop sign at first.