I agree: If you have data X and outputs y and must have a linear function of X to minimize square error on y and only y, then yes we have defined our task. I t

I agree: If you have data X and outputs y and must have a linear function of X to minimize square error on y and only y, then yes we have defined our task. I think it's a lucky engineer that has their end-goal defined as such a crisp mathematical task. Defining what is an optimal way to do a fit, for the surrounding application, usually requires thought. Often the reasonableness of the curve does matter, for example if it will be evaluated in locations other than the training data. Also, there may be some choice in how to set up the problem. For example, maybe we're allowed to transform the inputs with some fixed non-linear functions to create extra features. Adding these extra features to the design matrix will improve fit, so an engineer might consider it, but they'll need to be careful about over-fitting if they care about a surrounding application.