Sure, I think the approach could work successfully in some cases, but I think specifically allowing it to learn transformations of the input color space would b

Sure, I think the approach could work successfully in some cases, but I think specifically allowing it to learn transformations of the input color space would be quite problematic — you are essentially giving it more degrees of surface statistics freedom with which to data mine a better looking overfitting metric during validation steps.