4 ms·

It's interesting to see to what extent matrix multiplication and backpropagation are dominating the AI space these days. I wouldn't be surprised if other appro

by mg 4y ago

It's interesting to see to what extent matrix multiplication and backpropagation are dominating the AI space these days. I wouldn't be surprised if other approaches like genetic programming will make a comeback one day. If there are any papers out there arguing for/against neural networks to stay in the king's seat forever, I would love to see them.

nielsinho 4y ago
You can take a look here: https://youtu.be/27PYlj-qNb0 https://youtu.be/27PYlj-qNb0
- mg 4y ago
  That is a genetic algorithm at work. With "genetic programming" I was referring to applying genetic algorithms to code. So that the result of the evolution is a program that performs well doing a task like driving a car. Nice research on this has been done by John R. Koza. But nothing new came out in this area for quite a while.
JustFinishedBSG 4y ago
> I wouldn't be surprised if other approaches like genetic programming will make a comeback one day. Why would they? If you assume that your objective is kinda smooth genetic algorithms and related are guaranteed to suck. And most "real" world things of interest can be assumed to be pretty smooth
- mg 4y ago
  There are two parts to this. One is the representation: Neural Net vs Program. The other is the search approach: Back propagation vs crossover and mutation. Your smoothness argument applies to the search approach. I would say "smooth" is not enough to describe the search space of real world solutions. I would rather call it fractal like. There are many smooth areas in a visualization of the Mandelbrot set. But the most interesting things happen in areas that are not smooth but follow some delicate logic. You find tiny local minima there that are hard to reach via gradient descent. Genetic algorithms have a lot going for them in this type of search space. To pick one factor: GA's distribute the search power so that areas of the search space that are more promising get more resources. Similar to the optimal approach to deal with so called "multi-armed bandit" problems, GA's divide the search space in an optimal way, computation wise.
- deleted 4y ago
  [deleted]
mistrial9 4y ago
I think you have to start to narrow your question before you get results. That is, for the use-case of fast moving sensor inputs (like in a car) or reviewing a million videos to find identifiable patterns, cNN performs, but other specific queries in domain knowledge or patterns of robotic movement, decision trees and random forest immediately differentiate themselves. So papers on specific topics that are not the specialty of cNN might demonstrate effectiveness, and leave it up to the reader/critic to compare widely against other uses overall.
dekhn 4y ago
NNs only ended up working (at the level of human performance) because of 60+years of research punctuated by periods of very little interest (including times when GP/GA were being explored widely in the literature). And a bunch of that research was in entirely different areas, like GPUs, which went through many cycles of refinement for gaming and HPC before the DL people recognized its utility. To achieve parity, GPs and GAs would need demonstrations on the level of what was seen about 10 years ago in NNs, then enough people would get involved to start doing more tech dev to make the hardware do GAs really, really, really fast.

by mg 4y ago

nielsinho 4y ago

You can take a look here: https://youtu.be/27PYlj-qNb0 https://youtu.be/27PYlj-qNb0

mg 4y ago
That is a genetic algorithm at work. With "genetic programming" I was referring to applying genetic algorithms to code. So that the result of the evolution is a program that performs well doing a task like driving a car. Nice research on this has been done by John R. Koza. But nothing new came out in this area for quite a while.

JustFinishedBSG 4y ago

> I wouldn't be surprised if other approaches like genetic programming will make a comeback one day. Why would they? If you assume that your objective is kinda smooth genetic algorithms and related are guaranteed to suck. And most "real" world things of interest can be assumed to be pretty smooth

mg 4y ago
There are two parts to this. One is the representation: Neural Net vs Program. The other is the search approach: Back propagation vs crossover and mutation. Your smoothness argument applies to the search approach. I would say "smooth" is not enough to describe the search space of real world solutions. I would rather call it fractal like. There are many smooth areas in a visualization of the Mandelbrot set. But the most interesting things happen in areas that are not smooth but follow some delicate logic. You find tiny local minima there that are hard to reach via gradient descent. Genetic algorithms have a lot going for them in this type of search space. To pick one factor: GA's distribute the search power so that areas of the search space that are more promising get more resources. Similar to the optimal approach to deal with so called "multi-armed bandit" problems, GA's divide the search space in an optimal way, computation wise.
deleted 4y ago
[deleted]

mistrial9 4y ago

I think you have to start to narrow your question before you get results. That is, for the use-case of fast moving sensor inputs (like in a car) or reviewing a million videos to find identifiable patterns, cNN performs, but other specific queries in domain knowledge or patterns of robotic movement, decision trees and random forest immediately differentiate themselves. So papers on specific topics that are not the specialty of cNN might demonstrate effectiveness, and leave it up to the reader/critic to compare widely against other uses overall.

dekhn 4y ago

NNs only ended up working (at the level of human performance) because of 60+years of research punctuated by periods of very little interest (including times when GP/GA were being explored widely in the literature). And a bunch of that research was in entirely different areas, like GPUs, which went through many cycles of refinement for gaming and HPC before the DL people recognized its utility. To achieve parity, GPs and GAs would need demonstrations on the level of what was seen about 10 years ago in NNs, then enough people would get involved to start doing more tech dev to make the hardware do GAs really, really, really fast.