5 ms·

John Koza Has Built an Invention Machine

20 points by bootload 17y ago

rajat 17y ago
I was thinking about what type of intellectual endeavor might be "safe" from this Invention Machine, and obviously anything that has an intractable evaluation function is "safe". Take, for example, user interface design. Can we build an evaluation function that will give an answer as to whether one design will be more preferred than another? Is something that we considered ground-breaking when it first came out (ie. the iPhone), obvious from "first principles" of design? The evaluation function is intractable because you couldn't possibly put the generated designs in front of a number of people to evaluate it because of the time it will take to iterate. But then, could you evolve the evaluation function? Take a random sample of things that are considered to have fantastic user interfaces, break them down and evolve an evaluation function that is capable of designing at that level. Then test it against another set of randomly selected samples. And so on. Finally, test the designs on humans.
- Tichy 17y ago
  "you couldn't possibly put the generated designs in front of a number of people to evaluate it because of the time it will take to iterate" It's common practice these days, google A/B Testing. Or Mixpanel, I guess.
  - rajat 17y ago
    A/B testing wouldn't be nearly fast enough iteration for something like this, I think. I don't know how many iterations it takes for the algorithm to come up with something useful, but even if it is anywhere in the 100s, A/B testing won't work or it'll take years... How quickly, in terms of iterations, does this machine usually take to come up with something? I mean, it could get lucky pretty quickly, but what is a reasonable number to expect? I don't know.
- ahi 17y ago
  I think the intractable evaluation problem is salient for many other fields as well. A machinist might know a particular design can't be built, but may not be able to quantify why. I think it's certainly possible that the universe ends before sufficiently complex problems can be processed by a GA or even adequately defined. On the other hand, displayed UI could probably be tackled by GAs even if the evaluation function is never codified. Stick a couple hundred no skill workers in front of a screen and you could get a couple hundred thousand evaluations an hour; in effect industrializing design.
swernli 17y ago
An old article, but interesting for the same familiar reasons. I've done some work in genetic programming, most of which was done for a graduate project that I really enjoyed. The part of this article I resonated with the most was the end where they discuss the issue of finding problems to apply this method to. I have had some designs for evolutionary algorithms floating around, and even started parts of a framework, but keep losing motivation for it for the same reason: if I get the GP system up and running, what do I run it on? What do I have to evolve? I hate that my creativity stagnates without a well defined problem...
- marshallp 17y ago
  all problems where you would have applied machine learning or statistics. stock picking computer vision speech recognition discerning mathematical models of systems, effectively all of science, using symbolic regression
  - mattrepl 17y ago
    Problems for which you have solid models and tools (statistics) to solve them are the exact sort you wouldn't use GP or any sort of evolutionary computing (EC). EC is best used when the only other option is picking random solutions from the entire solution space. At least that's what I've gathered.
    - marshallp 17y ago
      Genetic programming is a vastly underapplied solution in modeling. You automatically get a rough mathematical model, analyze and understand and make progress with it. The scientific community hasn't really embraced what vast computational power can do. Some are catching on though - look for symbolic regression or automated scientist on google. (There's a fully automated 'robot' that does biology experiments built by a university prof in england though i'm not sure whether he usss genetic algorithms or genetic programming.)
mark_l_watson 17y ago
I had lunch with John Koza many years ago and bent his ear a bit with an idea for speeding up the creation of pre-trained recurrent neural networks (ended up being the last example in my 2nd C++ book). He explained why my idea was both interesting and not practical. Years later I also took a tutorial from him at AAAI which was lots of fun.

20 points by bootload 17y ago

rajat 17y ago

I was thinking about what type of intellectual endeavor might be "safe" from this Invention Machine, and obviously anything that has an intractable evaluation function is "safe". Take, for example, user interface design. Can we build an evaluation function that will give an answer as to whether one design will be more preferred than another? Is something that we considered ground-breaking when it first came out (ie. the iPhone), obvious from "first principles" of design? The evaluation function is intractable because you couldn't possibly put the generated designs in front of a number of people to evaluate it because of the time it will take to iterate. But then, could you evolve the evaluation function? Take a random sample of things that are considered to have fantastic user interfaces, break them down and evolve an evaluation function that is capable of designing at that level. Then test it against another set of randomly selected samples. And so on. Finally, test the designs on humans.

Tichy 17y ago
"you couldn't possibly put the generated designs in front of a number of people to evaluate it because of the time it will take to iterate" It's common practice these days, google A/B Testing. Or Mixpanel, I guess.
- rajat 17y ago
  A/B testing wouldn't be nearly fast enough iteration for something like this, I think. I don't know how many iterations it takes for the algorithm to come up with something useful, but even if it is anywhere in the 100s, A/B testing won't work or it'll take years... How quickly, in terms of iterations, does this machine usually take to come up with something? I mean, it could get lucky pretty quickly, but what is a reasonable number to expect? I don't know.
ahi 17y ago
I think the intractable evaluation problem is salient for many other fields as well. A machinist might know a particular design can't be built, but may not be able to quantify why. I think it's certainly possible that the universe ends before sufficiently complex problems can be processed by a GA or even adequately defined. On the other hand, displayed UI could probably be tackled by GAs even if the evaluation function is never codified. Stick a couple hundred no skill workers in front of a screen and you could get a couple hundred thousand evaluations an hour; in effect industrializing design.

swernli 17y ago

An old article, but interesting for the same familiar reasons. I've done some work in genetic programming, most of which was done for a graduate project that I really enjoyed. The part of this article I resonated with the most was the end where they discuss the issue of finding problems to apply this method to. I have had some designs for evolutionary algorithms floating around, and even started parts of a framework, but keep losing motivation for it for the same reason: if I get the GP system up and running, what do I run it on? What do I have to evolve? I hate that my creativity stagnates without a well defined problem...

marshallp 17y ago
all problems where you would have applied machine learning or statistics. stock picking computer vision speech recognition discerning mathematical models of systems, effectively all of science, using symbolic regression
- mattrepl 17y ago
  Problems for which you have solid models and tools (statistics) to solve them are the exact sort you wouldn't use GP or any sort of evolutionary computing (EC). EC is best used when the only other option is picking random solutions from the entire solution space. At least that's what I've gathered.
  - marshallp 17y ago
    Genetic programming is a vastly underapplied solution in modeling. You automatically get a rough mathematical model, analyze and understand and make progress with it. The scientific community hasn't really embraced what vast computational power can do. Some are catching on though - look for symbolic regression or automated scientist on google. (There's a fully automated 'robot' that does biology experiments built by a university prof in england though i'm not sure whether he usss genetic algorithms or genetic programming.)

mark_l_watson 17y ago

I had lunch with John Koza many years ago and bent his ear a bit with an idea for speeding up the creation of pre-trained recurrent neural networks (ended up being the last example in my 2nd C++ book). He explained why my idea was both interesting and not practical. Years later I also took a tutorial from him at AAAI which was lots of fun.