But to just restate the comment you're replying to, clearly theory can prune a lot of branches on the "iterative experiment-based design refinement" method you

But to just restate the comment you're replying to, clearly theory can prune a lot of branches on the "iterative experiment-based design refinement" method you are proposing. Also, I'll mention that you're being too pessimistic about what theory can accomplish. The learning problem is much more constrained than P = NP. The use, for example, of a training set of N examples, drawn iid, and evaluation on samples drawn from the same distribution imposes a lot of structure. I don't know if you're familiar with VC theory, but it's an example of the kind of "surprising" guarantees that can be derived in this setting. Other general examples are weak learning, the bias/variance tradeoff, and (in SVMs) the notion of large margin classifiers. An applicable "theory" of design is what separates engineering from just mucking around.