In more formal terms, you are trying to minimize the expected risk (generalization error). The expected risk is the sum of empirical risk (training set error)

In more formal terms, you are trying to minimize the expected risk (generalization error). The expected risk is the sum of empirical risk (training set error) and the structural risk (model complexity). In many instances, having low empirical risk comes at the cost of having high structural risk, which is overfitting.