4 ms·

Optimisation isn't the problem, optimisation is always a good thing. Deciding what the objective is on the other hand, that's the tricky part. Right now we're o

Optimisation isn't the problem, optimisation is always a good thing. Deciding what the objective is on the other hand, that's the tricky part. Right now we're optimising to an incomplete or incorrect objective function. Of course it's not that simple, no doubt someone will be along to mention Goodhart's Law or the inherent problem with essentially quantifying the meaning of life. Still, I don't think people applying logic to their daily lives can be a problem on its own.

ssivark 6y ago
Here’s the challenge: could you ever specify an objective function concretely enough to sustain an assault by aggressive optimization? https://thezvi.wordpress.com/2019/12/31/does-big-business-hate-your-family/ https://thezvi.wordpress.com/2019/12/31/does-big-business-ha... In all our mathematical experience, the answer turns out to be that we can only ever define an approximate objective function, and we almost always need some kind of “regularization” to constrain our optimization strategy. And in many cases, an effective regularization practically (which results in better “generalization” technically) might be “early stopping” I.e. not optimizing as hard!
- mdiesel 6y ago
  But if the definition of "optimisation" is achieving the best solution for the problem, then regularization isn't constraining optimisation, it's part of getting to the best solution and so becomes a part of the better optimiser. I agree, any sufficiently complex problem is beyond what we can define as a concrete objective function and any of the solutions we have. So instead the best we can hope for is that today is closer to being optimal than yesterday, and tomorrow will be better still.
  - ssivark 6y ago
    Regularization might give better solutions, but it’s no more optimizing the stated objective, and often runs counter. In some cases, regularization is explicit and by choice, but it’s also often implicit and not under control.
    - mdiesel 6y ago
      Again, "stated objective" isn't the original objective but the approximation we've had to use instead. You say that it gets a "better" solution, so the against some indescribable original objective there's a difference.

ssivark 6y ago

Here’s the challenge: could you ever specify an objective function concretely enough to sustain an assault by aggressive optimization? https://thezvi.wordpress.com/2019/12/31/does-big-business-hate-your-family/ https://thezvi.wordpress.com/2019/12/31/does-big-business-ha... In all our mathematical experience, the answer turns out to be that we can only ever define an approximate objective function, and we almost always need some kind of “regularization” to constrain our optimization strategy. And in many cases, an effective regularization practically (which results in better “generalization” technically) might be “early stopping” I.e. not optimizing as hard!

mdiesel 6y ago
But if the definition of "optimisation" is achieving the best solution for the problem, then regularization isn't constraining optimisation, it's part of getting to the best solution and so becomes a part of the better optimiser. I agree, any sufficiently complex problem is beyond what we can define as a concrete objective function and any of the solutions we have. So instead the best we can hope for is that today is closer to being optimal than yesterday, and tomorrow will be better still.
- ssivark 6y ago
  Regularization might give better solutions, but it’s no more optimizing the stated objective, and often runs counter. In some cases, regularization is explicit and by choice, but it’s also often implicit and not under control.
  - mdiesel 6y ago
    Again, "stated objective" isn't the original objective but the approximation we've had to use instead. You say that it gets a "better" solution, so the against some indescribable original objective there's a difference.