4 ms·

To disagree on a side issue, it's been a long time since NIPS dealt much with classic ANNs (which never had much to do with human brains, in any case). Most of

by another 16y ago

To disagree on a side issue, it's been a long time since NIPS dealt much with classic ANNs (which never had much to do with human brains, in any case). Most of the action there, as in AI and ML at large these days---eg, the focus also at ICML and at other venues---is in statistical methods. (On the other hand, neuroscience and explicitly biological neural modeling are exciting areas, reasonably well-represented at NIPS. Those topics, however, are almost entirely different from neural networks of the multilayer perceptron / [Hyper]NEAT varieties.) But your criticism of the article seems accurate. ACO isn't new, and there's little evidence that it will solve any of the major outstanding problems in AI. Less generously, however, I'd suggest that much of the research related to the family of population-based stochastic search methods, ACO, PSO, and HyperNEAT included, is prone to the same risk: the lack of a field-wide theoretical foundation, coupled with the absence of a field-wide standard methodology and benchmark set for empirical comparison (as opposed to, say, the situation in supervised learning), makes it temptingly easy for a particular researcher to believe too strongly in the capabilities of that researcher's pet algorithm. This situation seems to have balkanized the field (page through a recent GECCO proceedings, for example), and holds back wider progress. That's not to say that HyperNEAT can't do great things. It's a fun approach, and Stanley et al are running far with it. But your boosterism of it, and the boosterism of ACO that you're objecting to, seem closely related. (For contrast, I'd suggest, eg, the natural gradient work at IDSIA. It's unlikely to be the ultimate method, but may be a good model for solid research in this area.)

tansey 16y ago
Point taken on NIPS. I agree its focus has shifted in recent years. Venues like GECCO/WCCI/CEC tend to be better if you're looking for nature-inspired algorithms. The theoretical foundation debate is ancient and probably not worth getting into. However, there are many benchmarks for each class of algorithms. For instance, the double pole balancing task for neural network controllers is considered a standard metric. More general benchmarks, like keepaway soccer, are also available. Also, in fairness, the whole AI field is balkanized. Look at statistical relational learning algorithms for example. The closer you look, the more fractures you'll find in any community.
- another 16y ago
  Absolutely---it's a matter of degree, and judgments thereof are bound to be subjective. That said, I'd still claim that the EC community, broadly taken, appears particularly prone to tribalism. I'm far from a zealot on the "theoretical foundation" question, and it's possible to point to fields that have been, perhaps, even harmed by their possession of such a foundation (eg, reinforcement learning). That said, you need at least one or the other---theory or methodology. You mention DPB, and it's a good example: a lot of value was gained from one line of research centered on that single benchmark, despite its flaws. Pull together a curated suite of such problems, release a common implementation, get it widely used, and the field would benefit immensely. Some equivalent of Caruana's grand comparison, from the supervised ML world, could also do much good. In any case, it's an interesting topic, but we could go back and forth on it forever. GAs and their progeny (including ACO and NEAT) are fun to consider and lend themselves to accessible explanations, so they get a fair amount of press. Whether or not that's justified is something we'll likely disagree on---but I wanted to push back a bit against the implication, intended or not, that they're where the action is in AI today.
SeanLuke 16y ago
> This situation seems to have balkanized the field (page through a recent GECCO proceedings, for example), and holds back wider progress. Stochastic optimization has always been balkanized, but not because of a lack of methodology or benchmarks, but rather because the field was simultaneously invented in several different locations (Evolutionary Programming in San Diego and at NSF, Evolution Strategies in Germany, the Genetic Algorithm in Michigan). That's what happens in such situations: you get competition and differentiation when there really isn't much. Due to various political reasons, GECCO and CEC broke apart and GECCO itself decided to divide itself up by topic. In the meantime, alternative methods (ACO, PSO, various single-state "metaheuristics", etc.) have danced about at the periphery. I think your criticism of benchmarks is much too harsh. First off, it's not really true. A number of areas in evolutionary computation have very well established benchmarks: certainly this is the case for genetic programming; and for vector representations (Rastrigin? Rosenbrock? Schwefel? The De Jong test suite? Griewangk? Etc.). Also multiobjective optimization has established a fairly common set of benchmarks. Second, much of the remainder of the field consists of different kinds of solution representations (ACO; various graph representations such as NEAT/HyperNEAT; list representations; etc.) and in such situations unifying benchmarks make no sense. It's like insisting that the Iris data set be used for problems in text mining. I think there's quite a large degree of theoretical work among these techniques. Indeed there are entire theoretical conferences. But if you're looking for field-wide theoretical foundation, you're barking up the wrong tree. The problem is that most "interesting" solution representations result in dynamics which are essentially impossible to base a theoretical foundation on. In a real sense, various other fields have theoretical foundations because their problems are well formed. Stochastic optimization by its nature is tackling nastier problems for which there is no well-formed solution concept. The problems are ugly and hairy. This shouldn't reflect on the researchers brave enough to tackle them.
- another 16y ago
  Thanks, enjoyed your response. Your "tackling nastier problems" point is well-taken, and one reason I have a great deal of sympathy for work in these research areas. I think that you're painting an overly-rosy picture of the benchmark situation, but you're right in that I was likely too critical. The attention paid to synthetic fitness landscapes, for example, is also a positive sign that that many researchers do care about issues of comparison and analysis. (That said, I obviously believe that the field can and must do much better---and, since it relies so heavily on empirical evidence, its need for consistent methodology is somewhat higher.)

by another 16y ago

tansey 16y ago

Point taken on NIPS. I agree its focus has shifted in recent years. Venues like GECCO/WCCI/CEC tend to be better if you're looking for nature-inspired algorithms. The theoretical foundation debate is ancient and probably not worth getting into. However, there are many benchmarks for each class of algorithms. For instance, the double pole balancing task for neural network controllers is considered a standard metric. More general benchmarks, like keepaway soccer, are also available. Also, in fairness, the whole AI field is balkanized. Look at statistical relational learning algorithms for example. The closer you look, the more fractures you'll find in any community.

another 16y ago
Absolutely---it's a matter of degree, and judgments thereof are bound to be subjective. That said, I'd still claim that the EC community, broadly taken, appears particularly prone to tribalism. I'm far from a zealot on the "theoretical foundation" question, and it's possible to point to fields that have been, perhaps, even harmed by their possession of such a foundation (eg, reinforcement learning). That said, you need at least one or the other---theory or methodology. You mention DPB, and it's a good example: a lot of value was gained from one line of research centered on that single benchmark, despite its flaws. Pull together a curated suite of such problems, release a common implementation, get it widely used, and the field would benefit immensely. Some equivalent of Caruana's grand comparison, from the supervised ML world, could also do much good. In any case, it's an interesting topic, but we could go back and forth on it forever. GAs and their progeny (including ACO and NEAT) are fun to consider and lend themselves to accessible explanations, so they get a fair amount of press. Whether or not that's justified is something we'll likely disagree on---but I wanted to push back a bit against the implication, intended or not, that they're where the action is in AI today.

SeanLuke 16y ago

> This situation seems to have balkanized the field (page through a recent GECCO proceedings, for example), and holds back wider progress. Stochastic optimization has always been balkanized, but not because of a lack of methodology or benchmarks, but rather because the field was simultaneously invented in several different locations (Evolutionary Programming in San Diego and at NSF, Evolution Strategies in Germany, the Genetic Algorithm in Michigan). That's what happens in such situations: you get competition and differentiation when there really isn't much. Due to various political reasons, GECCO and CEC broke apart and GECCO itself decided to divide itself up by topic. In the meantime, alternative methods (ACO, PSO, various single-state "metaheuristics", etc.) have danced about at the periphery. I think your criticism of benchmarks is much too harsh. First off, it's not really true. A number of areas in evolutionary computation have very well established benchmarks: certainly this is the case for genetic programming; and for vector representations (Rastrigin? Rosenbrock? Schwefel? The De Jong test suite? Griewangk? Etc.). Also multiobjective optimization has established a fairly common set of benchmarks. Second, much of the remainder of the field consists of different kinds of solution representations (ACO; various graph representations such as NEAT/HyperNEAT; list representations; etc.) and in such situations unifying benchmarks make no sense. It's like insisting that the Iris data set be used for problems in text mining. I think there's quite a large degree of theoretical work among these techniques. Indeed there are entire theoretical conferences. But if you're looking for field-wide theoretical foundation, you're barking up the wrong tree. The problem is that most "interesting" solution representations result in dynamics which are essentially impossible to base a theoretical foundation on. In a real sense, various other fields have theoretical foundations because their problems are well formed. Stochastic optimization by its nature is tackling nastier problems for which there is no well-formed solution concept. The problems are ugly and hairy. This shouldn't reflect on the researchers brave enough to tackle them.

another 16y ago
Thanks, enjoyed your response. Your "tackling nastier problems" point is well-taken, and one reason I have a great deal of sympathy for work in these research areas. I think that you're painting an overly-rosy picture of the benchmark situation, but you're right in that I was likely too critical. The attention paid to synthetic fitness landscapes, for example, is also a positive sign that that many researchers do care about issues of comparison and analysis. (That said, I obviously believe that the field can and must do much better---and, since it relies so heavily on empirical evidence, its need for consistent methodology is somewhat higher.)