This is very interesting, but the real question is: how do you test which approach is better? For example, in the text detection case there are almost unlimite

This is very interesting, but the real question is: how do you test which approach is better? For example, in the text detection case there are almost unlimited combinations of transforms that you can put together. Usually you use some hybrid of gut feeling and results to decide, but I bet Netflix has enough data to make that call in a more principled way. Would be awesome to hear about that. How do you create a labeled dataset? How exactly do you measure which approach is better? Is there a perceptual element to it, or is it all quantitative? Edit: here's the related money quote from the retargeting paper linked in the other comment: "In terms of objective measures for retargeting, our results show that we are still a long way from imitating human perception. There is a relatively large discrepancy between such measures and the subjective data we collected, and in fact, the most preferred algorithm by human viewers, SV, received low ranking by almost all automatic distance measures."