Great post. I was part of a team that built a commercial image comparison engine, and worked on a lot of this stuff directly. Feeling all nostalgic now. Some o

Great post. I was part of a team that built a commercial image comparison engine, and worked on a lot of this stuff directly. Feeling all nostalgic now. Some of the issues that anyone attempting similar will bump into are: (1) If you're looking at shape features, a heavy dependency on product positioning. We found you need to do classification and use secondary features to get around this. (2) Avoiding matching on background colours. For flat-background images you can sample the corners. For gradient backgrounds you can need some kind of edge detection. (3) Shadow removal. A lot of the colours extracted from images are shadows of the actual perceived colour. You can generally pick the perceived colour by clustering the relevant pixels by hue/saturation and picking the middle of the lightness curve bit it varies by material. (4) Data. Fashion retail products are numerous and go out of stock very quickly so to get any real value out of this specific example you need a nice efficient data pipeline and capacity to handle constant change (5) Query performance / index size. We used an inverted index based on Lucene to query the data. To get decent relevance you need to include more visual words than you'd typically use in a text query. In our setup we found 25-30 visual words per feature was about right. This meant you needed to keep the index size under a million records, which meant we needed a distributed index. MLT queries are useful here, but last time I looked, which was admittedly a long time ago, Lucene had an annoying performance problem that I had to work around (https://issues.apache.org/jira/browse/LUCENE-1690 https://issues.apache.org/jira/browse/LUCENE-1690) (6) Value prop. Make sure you're optimising for the right use case, because as other commenters have pointed out, the relevance of this kind of system needs to be tuned to what you want to accomplish with it and the kind of data you'll be processing. Nobody should let themselves be put off by that list though. There's a lot of value in it, and it's a lot of fun to play with. Happy hunting :)