3 ms·
Here’s the challenge: could you ever specify an objective function concretely enough to sustain an assault by aggressive optimization? https://thezvi.wordpress.
by ssivark 6y ago
Here’s the challenge: could you ever specify an objective function concretely enough to sustain an assault by aggressive optimization? https://thezvi.wordpress.com/2019/12/31/does-big-business-hate-your-family/ https://thezvi.wordpress.com/2019/12/31/does-big-business-ha...
In all our mathematical experience, the answer turns out to be that we can only ever define an approximate objective function, and we almost always need some kind of “regularization” to constrain our optimization strategy. And in many cases, an effective regularization practically (which results in better “generalization” technically) might be “early stopping” I.e. not optimizing as hard!
- mdiesel 6y agoBut if the definition of "optimisation" is achieving the best solution for the problem, then regularization isn't constraining optimisation, it's part of getting to the best solution and so becomes a part of the better optimiser. I agree, any sufficiently complex problem is beyond what we can define as a concrete objective function and any of the solutions we have. So instead the best we can hope for is that today is closer to being optimal than yesterday, and tomorrow will be better still.
- ssivark 6y agoRegularization might give better solutions, but it’s no more optimizing the stated objective, and often runs counter. In some cases, regularization is explicit and by choice, but it’s also often implicit and not under control.
- mdiesel 6y agoAgain, "stated objective" isn't the original objective but the approximation we've had to use instead. You say that it gets a "better" solution, so the against some indescribable original objective there's a difference.