I think one lesson from the deep learning breakthrough in computer vision a few years ago was that deep nets can actually outperform humans in the engineering o

I think one lesson from the deep learning breakthrough in computer vision a few years ago was that deep nets can actually outperform humans in the engineering of features, and what you are suggesting is basically feature engineering in the output space.