3 ms·

> Well one area where genetic algorithms are better than basic gradient descent is, naturally, nondifferentiable functions. But that is facile. That's a poor a

> Well one area where genetic algorithms are better than basic gradient descent is, naturally, nondifferentiable functions. But that is facile. That's a poor assertion because even if detivative-free algorithms weren't already a thing, there are a myriad of smoothing techniques that enable nondifferentiable objective functions to be approximated by differentiable functions. > I think the realistic strength is that there are some easy to use libraries that don't require the user to really know anything about optimisation or work too hard and can deal with very difficult optimization scenarios (and sometimes even deal with them well). That's also not a reasonable assertion because there are also plenty software libraries for continuous and discrete optimization that are easy to use. GAs in general are explored in academia because they are a fad that's easily publishable. Other than that this class of methods is in general very computationally expensive and very inefficient, and don't provide any advantage over plain old adaptive sampling, or even dumb regular lattice sampling.

jmmcd 7y ago
> GAs in general are explored in academia because they are a fad that's easily publishable. I would say 90% of GA papers are because of this. > Other than that this class of methods is in general very computationally expensive and very inefficient, and don't provide any advantage over plain old adaptive sampling, or even dumb regular lattice sampling. Definitely false. The vast majority of OR-type papers in good journals include comparison to weak baselines like those, and wouldn't be published (in those venues) if they didn't win.
- geezerjay 7y ago
  > Definitely false. The vast majority of OR-type papers in good journals include comparison to weak baselines like those, and wouldn't be published (in those venues) if they didn't win. That statement is not correct. Although it's customary to accompany papers with benchmarks, these benchmarks focus practically exclusively on popular evolutionary algorithms. Worse, the "no free lunch" theorem grants authors the freedom to cherry-pick which benchmark problems are used.
  - jmmcd 7y ago
    Hmmm, maybe we're reading different papers. Ok, I admit that a lot of papers compare only against other metaheuristics, but only in cases where it is already accepted that those other metaheuristics far out-perform weaker methods. About NFL, I agree that a lot of authors misuse it in that way. But again, in most papers in good journals, what we see is either a comprehensive experiment with a wide enough range of instances, or a real industry problem, not cherry-picking.
    - geezerjay 7y ago
      > Ok, I admit that a lot of papers compare only against other metaheuristics, but only in cases where it is already accepted that those other metaheuristics far out-perform weaker methods. Those methods are invariably other evolutionary methods, and benchmarks are cherry-picked to show only encouraging results under the convenient guise of the "no free lunch" theorem. That' pretty much the norm, such as the repetitive recipe for inventing a metaheuristic algorithm of a) coming up with a clever nature-inspired metaphor with a catchy name, b) put together an algorithm that is arguably inspired on the metaphor, c) come up with a benchmark that arguably portrays the algorithm as being any improvement, even if only on a fortuitous corner case.
      - jmmcd 7y ago
        No, I'm using "weaker methods" in the technical sense, not "performs worse". I agree that many papers of that recipe type exist -- Sorensen and Weyland have skewered them effectively -- they are just froth, to be ignored in a discussion of the true merits of evolutionary computation.
Ragib_Zaman 7y ago
> That's a poor assertion because even if detivative-free algorithms weren't already a thing, there are a myriad of smoothing techniques that enable nondifferentiable objective functions to be approximated by differentiable functions. The function you wish to optimise may be non differentiable because the domain is discrete. E.g. to perform feature selection one can binary encode which features are included and which are excluded, with the fitness function being the accuracy of a model using the included features as inputs and trained for X iterations.

jmmcd 7y ago

> GAs in general are explored in academia because they are a fad that's easily publishable. I would say 90% of GA papers are because of this. > Other than that this class of methods is in general very computationally expensive and very inefficient, and don't provide any advantage over plain old adaptive sampling, or even dumb regular lattice sampling. Definitely false. The vast majority of OR-type papers in good journals include comparison to weak baselines like those, and wouldn't be published (in those venues) if they didn't win.

geezerjay 7y ago
> Definitely false. The vast majority of OR-type papers in good journals include comparison to weak baselines like those, and wouldn't be published (in those venues) if they didn't win. That statement is not correct. Although it's customary to accompany papers with benchmarks, these benchmarks focus practically exclusively on popular evolutionary algorithms. Worse, the "no free lunch" theorem grants authors the freedom to cherry-pick which benchmark problems are used.
- jmmcd 7y ago
  Hmmm, maybe we're reading different papers. Ok, I admit that a lot of papers compare only against other metaheuristics, but only in cases where it is already accepted that those other metaheuristics far out-perform weaker methods. About NFL, I agree that a lot of authors misuse it in that way. But again, in most papers in good journals, what we see is either a comprehensive experiment with a wide enough range of instances, or a real industry problem, not cherry-picking.
  - geezerjay 7y ago
    > Ok, I admit that a lot of papers compare only against other metaheuristics, but only in cases where it is already accepted that those other metaheuristics far out-perform weaker methods. Those methods are invariably other evolutionary methods, and benchmarks are cherry-picked to show only encouraging results under the convenient guise of the "no free lunch" theorem. That' pretty much the norm, such as the repetitive recipe for inventing a metaheuristic algorithm of a) coming up with a clever nature-inspired metaphor with a catchy name, b) put together an algorithm that is arguably inspired on the metaphor, c) come up with a benchmark that arguably portrays the algorithm as being any improvement, even if only on a fortuitous corner case.
    - jmmcd 7y ago
      No, I'm using "weaker methods" in the technical sense, not "performs worse". I agree that many papers of that recipe type exist -- Sorensen and Weyland have skewered them effectively -- they are just froth, to be ignored in a discussion of the true merits of evolutionary computation.

Ragib_Zaman 7y ago

> That's a poor assertion because even if detivative-free algorithms weren't already a thing, there are a myriad of smoothing techniques that enable nondifferentiable objective functions to be approximated by differentiable functions. The function you wish to optimise may be non differentiable because the domain is discrete. E.g. to perform feature selection one can binary encode which features are included and which are excluded, with the fitness function being the accuracy of a model using the included features as inputs and trained for X iterations.