I've done a similar thing trying to create classifiers for scenes and buildings. It's hard to have a word for 'bottom left corner of a window with bars on it',

I've done a similar thing trying to create classifiers for scenes and buildings. It's hard to have a word for 'bottom left corner of a window with bars on it', for example. And for scenes there's no taxonomy in advance. I think it works better than expected (as good as automatic patch discovery methods), but you end up having to train a lot more classifiers because you'll get a number of redundant classifiers. Something that might be cool is to have a step where the common people try to do the clustering, and an expert corrects them if they do something not useful. Then there'd be a back and forth between experts and crowd people. I haven't really thought that out though.