14 ms·

Best of all you don't need to understand your problem domain?! This sounds like a recipe for disaster to me - how do you tell if your model is giving you nonsen

by hadley 17y ago

Best of all you don't need to understand your problem domain?! This sounds like a recipe for disaster to me - how do you tell if your model is giving you nonsense or not?

cmorrisrsg 17y ago
We had a fairly conservative cross-validation function that tested the results of the genetic optimization for a first pass. Following that, Omnivore was put into "observation" mode, so we could verify that the results it generated were predictive going forward, and not just biased towards the test set. Over the course of the past few months, the model has held up quite successfully. Since accusing a customer of being a spammer without non-heuristic proof can be an emotion-ridden experience for everyone involved, we were conservative and optimized for accuracy over efficiency pretty much every step of the process.
- hadley 17y ago
  I don't doubt the quality of your final model, I just think you might have found it easier to get there with a better method.
gnosis 17y ago
It's quite common to easily know when the answers you're getting are good without understanding the problem domain at all (or at least without needing to use that understanding to create a model). For example, you could know that some sounds emitted from a computer speaker sound good or bad, without knowing the first thing about music theory. And evolutionary algorithms (EA) can be used to evolve these sounds to sound better and better, without having been given any sort of explicit model of what "good" or "bad" music is. The solutions evolve merely on the principles of natural selection, by randomly combining solutions and testing them for how "good" or "bad" they are. This has been done. Just search google for: evolving music genetic The above problem is actually a little difficult for EA's to solve, as you need humans to manually evaluate each solution. This slows the EA down a lot, and it takes a lot longer for optimum solutions to evolve. Fortunately, the solutions to many other problems can be evaluated algorithmically, by the computer itself. As a very general example, a solution that's trying to minimize some function can be evaluated via a mathematical comparison that a computer can handle just fine on its own. For some more specific examples, see "36 Human-Competitive Results Produced by Genetic Programming": http://www.genetic-programming.com/humancompetitive.html http://www.genetic-programming.com/humancompetitive.html I'm sure you could find thousands more examples by searching for the results produced by other "model-free methods" like other EA's (genetic algorithms, for example), neural networks, ant and swarm optimization, etc.

by hadley 17y ago

Best of all you don't need to understand your problem domain?! This sounds like a recipe for disaster to me - how do you tell if your model is giving you nonsense or not?

cmorrisrsg 17y ago

We had a fairly conservative cross-validation function that tested the results of the genetic optimization for a first pass. Following that, Omnivore was put into "observation" mode, so we could verify that the results it generated were predictive going forward, and not just biased towards the test set. Over the course of the past few months, the model has held up quite successfully. Since accusing a customer of being a spammer without non-heuristic proof can be an emotion-ridden experience for everyone involved, we were conservative and optimized for accuracy over efficiency pretty much every step of the process.

hadley 17y ago
I don't doubt the quality of your final model, I just think you might have found it easier to get there with a better method.

gnosis 17y ago

It's quite common to easily know when the answers you're getting are good without understanding the problem domain at all (or at least without needing to use that understanding to create a model). For example, you could know that some sounds emitted from a computer speaker sound good or bad, without knowing the first thing about music theory. And evolutionary algorithms (EA) can be used to evolve these sounds to sound better and better, without having been given any sort of explicit model of what "good" or "bad" music is. The solutions evolve merely on the principles of natural selection, by randomly combining solutions and testing them for how "good" or "bad" they are. This has been done. Just search google for: evolving music genetic The above problem is actually a little difficult for EA's to solve, as you need humans to manually evaluate each solution. This slows the EA down a lot, and it takes a lot longer for optimum solutions to evolve. Fortunately, the solutions to many other problems can be evaluated algorithmically, by the computer itself. As a very general example, a solution that's trying to minimize some function can be evaluated via a mathematical comparison that a computer can handle just fine on its own. For some more specific examples, see "36 Human-Competitive Results Produced by Genetic Programming": http://www.genetic-programming.com/humancompetitive.html http://www.genetic-programming.com/humancompetitive.html I'm sure you could find thousands more examples by searching for the results produced by other "model-free methods" like other EA's (genetic algorithms, for example), neural networks, ant and swarm optimization, etc.